Closed blank-ed closed 9 months ago
Finally figured out how to do this lmao. It took some time. So stuff to do:
* replace `tf_bodypix` with `mp selfie segmentation` * use `bgmask` instead of `mask_bg`. * create a new subclass for processor that uses the bg info. * add the NLMS filter
great that it worked now. I agree with these steps. Do you want me to take over something?
@SamProell thank you, but I think I am done with LiCVPR. Could you please check it?
So a few things:
tf_bodypix
with mp selfie segmentation
, used bgmask
instead of mask_bg
and added the NLMS filter. cv2.rectangle(mask, (0, 0), (w, 5), 255, -1)
is in get_default_bgmask
function in region_of_interest.py
? Why did you make the 5 pixels at the top white? I am just curiousProcessor
, therefore I added a condition for background
in all the rPPG algorithms while setting True
for LiCVPR
and False
for CHROM
and POS
. Do you think this is be a better way to go?Let me know what you think about these points. Thank you!
@blank-ed thank you for the update.
about selfie segmentation: I believe you are performing more steps than actually needed. The MediaPipe segmenter can run directly on the frame, no need to flip or convert colors. Also, there is no need to do the thresholding through opencv again. We can get the mask already (as a single-channel image) with bgmask = (seg_results.segmentation_mask < 0.5).astype(np.uint8)
We can draw the detected background if the user wants to (--draw-facemark
) by adding:
if self.draw_landmarks:
cv2.bitwise_and(frame, np.zeros_like(frame), frame, mask=bgmask)
Also I would probably create a new RoiDetector class with this functionality. But this is something I can do later..
get_default_bgmask
: without proper background segmentation implemented I chose the top 5 rows of the image as the background region, as this is a part of the image that does likely not include the face. It is not really needed anymore now.I forgot that there is already a subclass for LiCVPR, this is of course the place to put all the functionality, no need for a new subclass. However, I would prefer a slightly different implementation for the background rgb calculation.
I would have have spatial_pooling
compute the RGB either for the ROI or the background, but not both (can always be done in two calls). So like this:
# in processor.py
def spatial_pooling(self, roi, background=False, append_rgb=False):
r, g, b = roi.get_mean_rgb(background)
if append_rgb:
self._rs.append(r)
self._gs.append(g)
self._bs.append(b)
return r, g, b
# in region_of_interest.py
def get_mean_rgb(self, background=False):
mask = self._mask
if background:
if self._bgmask is None:
raise ValueError("Background mask is not specified")
mask = self._bgmask
r, g, b, _ = cv2.mean(self.rawimg, mask)
return r, g, b
with this, there are no changes required to the existing processors (chrom, pos, ...)
@SamProell thanks for the tips. I will look into them and update the script. Sorry, I have been off for a couple weeks. I caught some virus and was bedridden. Will update you once I have updated as per your instructions. Thank you
Finally figured out how to do this lmao. It took some time. So stuff to do:
tf_bodypix
withmp selfie segmentation
bgmask
instead ofmask_bg
.