Testing with new dataset

[umutbrkee]

Hello;

as a part of my research, I was planning to test neural methods with different datasets, even creating my own if necessary. I have a dataset that includes ECG device data, so I've been working on adapting it. I thought I could process my dataset in a similar format to UBFC's. Here's the format I used:

---

                 NotUBFC/
                 |   |-- subject1/
                 |       |-- vid.avi                  (name and position is changed)
                 |       |-- ground_truth.txt  (converted from .csv file)
                 |   |-- subject2/
                 |       |-- vid.avi                   (name and position is changed)
                 |       |-- ground_truth.txt  (converted from .csv file)
                 |...
                 |   |-- subjectn/
                 |       |-- vid.avi                   (name and position is changed)
                 |       |-- ground_truth.txt  (converted from .csv file)
            -----------------

Then, since the ground_truth.txt file in UBFC contains signals from ECG devices. I took the ECG signals out of .csv file to convert a ground_truth.txt file. Screenshots are as follows:

Base format of the .csv file in my dataset

only ecg signals as .txt file similar format as ubfc_rPPG

However, I'm encountering this error at the end, and I'm having trouble finding the cause. If you notice anything, I would be grateful for your help.

This is the dataset im using currently (be advised i changed the format of the dataset): https://cmp.felk.cvut.cz/~spetlrad/ecg-fitness/

[yahskapar]

Assuming you're re-using the UBFC-rPPG dataloader directly, can you try printing out frames and bvps (either directly print the data, or maybe just the shape of the data with np.shape) on or after this line?

My feeling on first glance is maybe you have to modify the read_wave() function slightly based on your new .txt files, but a good initial check is the above print statements. Also, as always, double-check the folder paths you supplied in your config file.

[umutbrkee]

Thanks for the tip, I thought the same thing and before opening the issue, I tried printing the data_dir to see if it was reading the data correctly.

As you can see in the picture, there was no problem with data_dir. I tried to print the frames to see if it processed the video correctly, but I could not get any output about the frames anywhere.

Along with not being able to print the frames, I realized that I couldn't print anything there in the code and I couldn't find the reason.

[yahskapar]

That sounds pretty strange, and probably very likely caused by the process itself silently dying for some reason. Two possible things to investigate:

1) In #219 I believe you already adjusted the multi_process_quota from 8 to 1 - can you confirm this problem persists when you use a multi_process_quota of 1? If you've been using something other than 1 just because it worked with another dataset, try using 1 again with this new dataset. There are multiple factors that can affect the optimal multi_process_quota setting from system to system, and some of those factors could be dataset-dependent (e.g., how resource usage is affected by the length or effective size of the videos).

2) Though the ECG-Fitness dataset videos appear to be only a minute long (the same as PURE), they are also a much higher resolution at 1920x1080. On top of the videos being uncompressed, this is likely why I'd expect reading the videos and processing them to be much more computationally expensive in contrast to 640x480 videos in the UBFC-rPPG dataset, for example. Try adding a print statement just below this commented line in your corresponding read_video() function. Then, add another print statement just before we return from the function.

Can you reach the first print statement but not the second? If so, I think this is indicative of an issue with reading these particular types of videos on your machine with the resources (e.g., memory) you have. On top of seeing if there's some way you could get this to work on your machine with at least multi_process_quota set to 1 and memory freed up from elsewhere on your machine (a way to check this generally on Linux is using free -mh), you could try changing the video capture stream to use hardware acceleration in the below line of code:

https://github.com/ubicomplab/rPPG-Toolbox/blob/d992de556fbbd6c865e38d475e25ec4eccebb55e/dataset/data_loader/UBFCrPPGLoader.py#L98

Change this to be:

VidObj = cv2.VideoCapture(video_file, cv2.CAP_FFMPEG)

Depending on how OpenCV was installed in your environment, you may need to download a version that is compiled with FFMPEG (e.g., the build here).

There's also other, potentially more efficient ways to read videos (e.g., vidgear), but that's probably more tricky to get working in the context of this toolbox and how we currently do multi-processing.

[umutbrkee]

1. can you confirm this problem persists when you use a multi_process_quota of 1?

First of all, yes, I can confirm that multi_process_quota was at 1 from the beginning.

2. Try adding a print statement just below this commented line in your corresponding read_video() function. Then, add another print statement just before we return from the function.

In my attempts today, I could not reach the 2nd print statement. Following your suggestion, I increased the RAM of Linux from 7 to 10 as much as WSL allows. When I checked again Finally I was able to reach the 2nd print statement , this was the result I got:

can you try printing out frames and bvps (either directly print the data, or maybe just the shape of the data with np.shape) on or after this line?

However, I still cannot access this print statement.

Change this to be:

VidObj = cv2.VideoCapture(video_file, cv2.CAP_FFMPEG)

EDIT: I also did this somewhere in the process where i got the 2. print statement

If you have a more efficient recommendation to use toolbox instead of WSL, I can consider it. I may also be having problems since they share 16GB of RAM and other components with Windows.

[yahskapar]

I think we've narrowed this down to a resource usage issue at this point, especially given the high-resolution videos in ECG-Fitness and considering, at least with multi_process_quota set to 1, you've been able to successfully pre-process and use other datasets such as UBFC-rPPG.

Few more suggestions:

1) Try reducing the memory usage before or while loading the videos. You can try resizing all of the ECG-Fitness videos to 640x480 using OpenCV and the cv2.INTER_AREA setting in particular and saving them before running this toolbox. Alternatively, you can try something like below in the read_video() function's while loop:

    target_resolution = (640, 480)
    while True:
        success, frame = VidObj.read()
        if not success:
            break

        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        frame = cv2.resize(frame, target_resolution, interpolation = cv2.INTER_AREA)
        frames.append(frame)

Basically, this would resize the frames prior to storing them, and should have some impact in reducing the memory usage. You can try to also monitor the memory usage using something like top while the toolbox runs.

2) Try working with the toolbox on Windows. Based on past issues in this toolbox, I've heard of folks using this toolbox with Windows, and it's possible you'll be able to better utilize your RAM resources versus trying to make things work with WSL. I think you should explore suggestion 1) above first though, as it may be suitable for how you might want to work with the ECG-Fitness dataset.

All the best.

[yahskapar]

I'm going to go ahead and close this issue due to the lack of follow-up, but please feel free to re-open or create a new issue if you continue having problems @umutbrkee.