LiCVPR Update

[blank-ed]

Finally figured out how to do this lmao. It took some time. So stuff to do:

• replace tf_bodypix with mp selfie segmentation
• use bgmask instead of mask_bg.
• create a new subclass for processor that uses the bg info.
• add the NLMS filter

[SamProell]

Finally figured out how to do this lmao. It took some time. So stuff to do:

* replace `tf_bodypix` with `mp selfie segmentation`

* use `bgmask` instead of `mask_bg`.

* create a new subclass for processor that uses the bg info.

* add the NLMS filter

great that it worked now. I agree with these steps. Do you want me to take over something?

[blank-ed]

@SamProell thank you, but I think I am done with LiCVPR. Could you please check it?

So a few things:

• So first things first, I replaced tf_bodypix with mp selfie segmentation, used bgmask instead of mask_bg and added the NLMS filter.
• I want to know what the point of cv2.rectangle(mask, (0, 0), (w, 5), 255, -1) is in get_default_bgmask function in region_of_interest.py? Why did you make the 5 pixels at the top white? I am just curious
• I feel like its redundant to add a new subclass to Processor, therefore I added a condition for background in all the rPPG algorithms while setting True for LiCVPR and False for CHROM and POS. Do you think this is be a better way to go?
• Do you think I took the right steps implementing the NLMS filter?

Let me know what you think about these points. Thank you!

[SamProell]

@blank-ed thank you for the update.

• about selfie segmentation: I believe you are performing more steps than actually needed. The MediaPipe segmenter can run directly on the frame, no need to flip or convert colors. Also, there is no need to do the thresholding through opencv again. We can get the mask already (as a single-channel image) with bgmask = (seg_results.segmentation_mask < 0.5).astype(np.uint8)

  We can draw the detected background if the user wants to (--draw-facemark) by adding:

  if self.draw_landmarks:
    cv2.bitwise_and(frame, np.zeros_like(frame), frame, mask=bgmask)

  Also I would probably create a new RoiDetector class with this functionality. But this is something I can do later..

• about get_default_bgmask: without proper background segmentation implemented I chose the top 5 rows of the image as the background region, as this is a part of the image that does likely not include the face. It is not really needed anymore now.

• I forgot that there is already a subclass for LiCVPR, this is of course the place to put all the functionality, no need for a new subclass. However, I would prefer a slightly different implementation for the background rgb calculation. I would have have spatial_pooling compute the RGB either for the ROI or the background, but not both (can always be done in two calls). So like this:

  # in processor.py
  def spatial_pooling(self, roi, background=False, append_rgb=False):
  r, g, b = roi.get_mean_rgb(background)
  
  if append_rgb:
      self._rs.append(r)
      self._gs.append(g)
      self._bs.append(b)
  
  return r, g, b
  
  # in region_of_interest.py
  def get_mean_rgb(self, background=False):
  mask = self._mask
  if background:
      if self._bgmask is None:
          raise ValueError("Background mask is not specified")
      mask = self._bgmask
  r, g, b, _ = cv2.mean(self.rawimg, mask)
  return r, g, b

  with this, there are no changes required to the existing processors (chrom, pos, ...)

• NLMS filter: this will need some more digging. In your implementation, g, bg_g and h are single scalars. Therefore, matrix.getH essentially returns h itself and has no effect in your code. In the paper the color values and h are vectors (with multiple values for past frames). Although in the paper, it appers they are using all input frames for the calculation, which is impossible in a live setting. I guess we would need to do the computation for the past N (maybe 30 or 60 frames)... does this help?

[blank-ed]

@SamProell thanks for the tips. I will look into them and update the script. Sorry, I have been off for a couple weeks. I caught some virus and was bedridden. Will update you once I have updated as per your instructions. Thank you