Generate file issues

[q958287831]

May I ask if the final generated file is an npy file? How can I convert it into a video file?

[yahskapar]

Hi @q958287831,

Yes, the final, motion-augmented videos are all .npy files for the UBFC-rPPG, PURE, and UBFC-PHYS dataset inputs. The outputs are the same as the input .mat files for the SCAMPS dataset. The way this motion augmentation pipeline works is that, given an input dataset in its downloaded format such as UBFC-rPPG or PURE for example, the output is those same dataset folders with the video file or frames folder replaced with an .npy file. This should make it quite straightforward to use the output of this motion augmentation pipeline with the rPPG-Toolbox after reading this section. In case there is still any confusion on this aspect of your question, let me know and I can try to guide you on how to test or generally run experiments using the default outputs.

I recommend sticking to the .npy files to avoid any potential degradation of the video by applying further video compression. If you still want to convert the .npy file into a video file, you can modify any of the np.save() lines in this code bock:

https://github.com/Roni-Lab/MA-rPPG-Video-Toolbox/blob/8839a8e08aafc417a2e4593984fc815894fd479b/augment_videos.py#L275-L292

To instead use something like a video writing example here, or alternatively using OpenCV or the VideoGear libraries that are already imported in augment_videos.py. Here's some very minimal OpenCV code that I haven't tested myself:

filename = source_video_name + '_' + driving_video_name + '.avi'
output_video_path = os.path.join(augmented_path, source_video_name, filename)
fourcc = cv2.VideoWriter_fourcc(*'XVID')  # Use XVID codec for minimal compression (not necessarily raw?)
out = cv2.VideoWriter(output_video_path, fourcc, 30, (640, 480))  # Adjust the frame size and FPS (30 for UBFC-rPPG and PURE, 35 for UBFC-PHYS)

for frame in final_preds:
    out.write(frame)

out.release()

A while back I wrote the augmented videos out in an uncompressed 8-bit RGB format (akin to the .avi videos UBFC-rPPG originally contains), but I can't find the code I used to ensure that was happening reliably. If you care about minimizing the effect of compression on the video and subsequently the underlying rPPG signal, I again recommend just using the .npy files or being careful with how you write the videos out into a video container (like .avi).

[q958287831]

@yahskapar

Thank you very much. Mainly, I would like to see how the synthesized video works, just like the gif image you displayed on the homepage. I will try to use the code you shared to generate the video. Thank you again!

[q958287831]

@yahskapar

Hello, I attempted to use the code you provided to save the file as a video format, but the resulting video is in a mosaic shape and does not contain any faces. I don't know if there was an issue with saving, or if there was an issue with testing?

笔记

[yahskapar]

Hi @q958287831,

Sorry, I didn't test the video saving code I provided in my reply above, so very likely something wrong with saving. You can try looking up ways to write a video from numpy frames if all you want to do is visualize the result, that should be quite straightforward after a bit of digging. I dug up some code of mine that I have used in the past that may be of use to you:

import os, glob
import cv2
import numpy as np
from scipy import io as scio

# Functions for reading rPPG media of interest and saving frames
def read_mat(mat_file):
        try:
            mat = scio.loadmat(mat_file)
        except:
            for _ in range(20):
                print(mat_file)
        frames = np.array(mat['video'])
        return frames

def read_npy_video(self, video_file):
    """Reads a video file in the numpy format (.npy), returns frames(T,H,W,3)"""
    frames = np.load(video_file[0])
    if np.issubdtype(frames.dtype, np.integer) and np.min(frames) >= 0 and np.max(frames) <= 255:
        processed_frames = [frame.astype(np.uint8)[..., :3] for frame in frames]
    elif np.issubdtype(frames.dtype, np.floating) and np.min(frames) >= 0.0 and np.max(frames) <= 1.0:
        processed_frames = [(np.round(frame * 255)).astype(np.uint8)[..., :3] for frame in frames]
    else:
        print("Failed!")
    return np.asarray(processed_frames)

def save_video_frames(frames, video_name, save_path):
    """Saves video frames as an mp4 video file in the save path"""
    os.makedirs(save_path, exist_ok=True)
    height, width, _ = frames[0].shape
    video_name = video_name + ".mp4"
    video_file = os.path.join(save_path, video_name)
    fourcc = cv2.VideoWriter_fourcc(*"mp4v")
    out = cv2.VideoWriter(video_file, fourcc, 30.0, (width, height))
    for frame in frames:
        frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
        out.write(frame)
    out.release()
    print(f"Video saved: {video_file}")

You can re-use the functions from the above code, more specifically load the video using read_npy_video() after supplying it a filepath as a list, or you can strip the line with video_file[0] to be just video_file if you directly supply the filepath. This should generate a MP4 video.

import numpy as np
import imageio

npy_path = 'your/path/to/npy/file/here'

aug_frames = np.load(npy_path)
print(f'Augmented Length:{np.shape(aug_frames)}')

# Define the start and end frames for the GIFs
start_frame = 300
end_frame = 450

aug_selected = aug_frames[start_frame:end_frame]

imageio.mimsave('your-gif-name-here.gif', aug_selected, fps=30)

I will add some tested scripts to the utils folder of this repo later when I have time, as for now I'm too occupied with other projects. Hopefully the above code is somewhat helpful for now.

[q958287831]

Thank you very much! You are an outstanding and responsible expert! But how is the GIF generated for the Github project homepage?

------------------ 原始邮件 ------------------ 发件人: "Akshay @.>; 发送时间: 2023年9月11日(星期一) 下午4:30 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [Roni-Lab/MA-rPPG-Video-Toolbox] Generate file issues (Issue #9)

嗨，

抱歉，我没有测试我在上面的回复中提供的视频保存代码，所以保存很可能有问题。如果您只想可视化结果，您可以尝试查找从 numpy 帧编写视频的方法，经过一些挖掘后应该非常简单。我挖出了我过去使用过的两个可能对你有用的脚本： import os, glob import cv2 import numpy as np from scipy import io as scio # Functions for reading rPPG media of interest and saving frames def read_mat(mat_file): try: mat = scio.loadmat(matfile) except: for in range(20): print(mat_file) frames = np.array(mat['video']) return frames def read_npy_video(self, video_file): """Reads a video file in the numpy format (.npy), returns frames(T,H,W,3)""" frames = np.load(video_file[0]) if np.issubdtype(frames.dtype, np.integer) and np.min(frames) >= 0 and np.max(frames) <= 255: processed_frames = [frame.astype(np.uint8)[..., :3] for frame in frames] elif np.issubdtype(frames.dtype, np.floating) and np.min(frames) >= 0.0 and np.max(frames) <= 1.0: processed_frames = [(np.round(frame 255)).astype(np.uint8)[..., :3] for frame in frames] else: print("Failed!") return np.asarray(processed_frames) def save_video_frames(frames, video_name, save_path): """Saves video frames as an mp4 video file in the save path""" os.makedirs(save_path, existok=True) height, width, = frames[0].shape video_name = video_name + ".mp4" video_file = os.path.join(save_path, video_name) fourcc = cv2.VideoWriter_fourcc("mp4v") out = cv2.VideoWriter(video_file, fourcc, 30.0, (width, height)) for frame in frames: frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) out.write(frame) out.release() print(f"Video saved: {video_file}")
您可以重用上述代码中的函数，更具体地说，在提供文件路径作为列表后加载视频，或者如果您直接提供文件路径，则可以将行剥离为仅。这应该会生成一个 MP4 视频。read_npy_video()video_file[0]video_file import numpy as np import cv2 import os, glob import scipy.io import math import matplotlib.pyplot as plt import imageio npy_path = 'your/path/to/npy/file/here' aug_frames = np.load(npy_path) print(f'Augmented Length:{np.shape(aug_frames)}') # Define the start and end frames for the GIFs start_frame = 300 end_frame = 450 aug_selected = aug_frames[start_frame:end_frame] imageio.mimsave('your-gif-name-here.gif', aug_selected, fps=30)
稍后有时间时，我会在此存储库的文件夹中添加一些经过测试的脚本，因为现在我太忙于其他项目了。希望上面的代码现在有所帮助。utils

- 直接回复此电子邮件，在 GitHub 上查看或取消订阅。 您收到此消息是因为您被提及。Message ID: @.***>

[yahskapar]

If you mean how I put that figure together, I actually put it together using Google Slides here. The GIFs for the video were generated using a very similar script as what I shared in my previous reply, and the GIFs for the signal were generated using another script which I can't find right now, but I can dig up later when I have time.

[q958287831]

Okay, thank you very much. Your help is crucial to me!

------------------ 原始邮件 ------------------ 发件人: "Akshay @.>; 发送时间: 2023年9月11日(星期一) 下午4:42 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [Roni-Lab/MA-rPPG-Video-Toolbox] Generate file issues (Issue #9)

如果你的意思是我是如何把这个数字放在一起的，我实际上是在这里使用谷歌幻灯片把它放在一起的。视频的 GIF 是使用与我之前回复中分享的非常相似的脚本生成的，信号的 GIF 是使用我现在找不到的另一个脚本生成的，但我可以在以后有时间时挖掘。

- 直接回复此电子邮件，在 GitHub 上查看或取消订阅。 您收到此消息是因为您被提及。Message ID: @.***>

[yahskapar]

Hi @q958287831,

I ened up finding the signal .gif generation on my laptop. Here's the code below:

import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
import numpy as np
import imageio

# result_dict = np.load('MAUBFC_MAUBFC_qual.npy', allow_pickle=True).item()
# print(result_dict['subject8'].keys())
# ppg_data = result_dict['subject8']['label_ppg']
# ppg_data = (ppg_data - np.min(ppg_data)) / (np.max(ppg_data) - np.min(ppg_data)) # normalize the data
# print(ppg_data.shape)

# Generate some example data
ppg_data = np.random.rand(1000, 10)
print(ppg_data.shape)

# Create a figure and axis with figsize argument
fig, ax = plt.subplots(figsize=(10, 6))

# Create a line object for the PPG data
line, = ax.plot([], [], color='red')

# Set the x-axis and y-axis limits
ax.set_xlim(0, 100)
ax.set_ylim(np.min(ppg_data), np.max(ppg_data))

# Remove the spines of the axis
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
# ax.spines['bottom'].set_visible(False)
# ax.spines['left'].set_visible(False)

# Hide the tick marks and labels of the axis
ax.tick_params(axis='both', which='both', length=0, width=0, labelbottom=False, labelleft=False)

# Create a function to update the plot for each frame
def update(frame):
    # Calculate the x-axis limits for the current frame
    xlim = (max(0, frame - 100), frame)

    # Update the x-axis limits of the plot
    ax.set_xlim(xlim)

    # Update the data of the line object
    line.set_data(np.arange(xlim[0], xlim[1]), ppg_data[xlim[0]:xlim[1]])

# Create an animation object
animation = FuncAnimation(fig, update, frames=range(150), interval=20)

# Save the animation as a gif file
with imageio.get_writer('s8_label_ppg_animation.gif', mode='I', fps=30) as writer:
    for i in range(150):
        # Update the plot for the current frame
        animation._func(i)

        # Draw the plot
        fig.canvas.draw()

        # Convert the plot to an image
        image = np.frombuffer(fig.canvas.tostring_rgb(), dtype='uint8')
        image = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))

        # Append the image to the GIF file
        writer.append_data(image)

fig.tight_layout()

# Close the figure
plt.close(fig)

I recommend getting familiar with the rPPG-Toolbox and its evaluation pipeline if you want to re-use the commented part of the code that leverages the label_ppg data. If you don't want to do that, you're welcome to use the rest of the code as you see fit by generating some synthetic signal using numpy or just loading your own signal to make into a .gif.

Let me know if that answers your question and we can go ahead and close this issue, if you have further questions you can always make an ew issue.

[q958287831]

Thank you very much. Your answer perfectly solved my problem. I have successfully generated the video file, which contains severe compression, but the synthesis effect has been verified. Thank you very much again!

------------------ 原始邮件 ------------------ 发件人: "Akshay @.>; 发送时间: 2023年9月12日(星期二) 上午10:11 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [Roni-Lab/MA-rPPG-Video-Toolbox] Generate file issues (Issue #9)

嗨，

我在我的笔记本电脑上找到了.gif生成的信号。下面是代码： import matplotlib.pyplot as plt from matplotlib.animation import FuncAnimation import numpy as np import imageio # result_dict = np.load('MAUBFC_MAUBFC_qual.npy', allow_pickle=True).item() # print(result_dict['subject8'].keys()) # ppg_data = result_dict['subject8']['label_ppg'] # ppg_data = (ppg_data - np.min(ppg_data)) / (np.max(ppg_data) - np.min(ppg_data)) # normalize the data # print(ppg_data.shape) # Generate some example data ppg_data = np.random.rand(1000, 10) print(ppg_data.shape) # Create a figure and axis with figsize argument fig, ax = plt.subplots(figsize=(10, 6)) # Create a line object for the PPG data line, = ax.plot([], [], color='red') # Set the x-axis and y-axis limits ax.set_xlim(0, 100) ax.set_ylim(np.min(ppg_data), np.max(ppg_data)) # Remove the spines of the axis ax.spines['right'].set_visible(False) ax.spines['top'].set_visible(False) # ax.spines['bottom'].set_visible(False) # ax.spines['left'].set_visible(False) # Hide the tick marks and labels of the axis ax.tick_params(axis='both', which='both', length=0, width=0, labelbottom=False, labelleft=False) # Create a function to update the plot for each frame def update(frame): # Calculate the x-axis limits for the current frame xlim = (max(0, frame - 100), frame) # Update the x-axis limits of the plot ax.set_xlim(xlim) # Update the data of the line object line.set_data(np.arange(xlim[0], xlim[1]), ppg_data[xlim[0]:xlim[1]]) # Create an animation object animation = FuncAnimation(fig, update, frames=range(150), interval=20) # Save the animation as a gif file with imageio.get_writer('s8_label_ppg_animation.gif', mode='I', fps=30) as writer: for i in range(150): # Update the plot for the current frame animation._func(i) # Draw the plot fig.canvas.draw() # Convert the plot to an image image = np.frombuffer(fig.canvas.tostring_rgb(), dtype='uint8') image = image.reshape(fig.canvas.get_width_height()[::-1] + (3,)) # Append the image to the GIF file writer.append_data(image) fig.tight_layout() # Close the figure plt.close(fig)
如果您想重用利用数据的代码的注释部分，我建议您熟悉 rPPG-Toolbox 及其评估管道。如果您不想这样做，欢迎您使用 numpy 生成一些合成信号或只是加载您自己的信号以制作成.gif，从而根据需要使用其余代码。label_ppg

让我知道这是否回答了您的问题，我们可以继续关闭此问题，如果您有其他问题，您可以随时提出 ew 问题。

- 直接回复此电子邮件，在 GitHub 上查看或取消订阅。 您收到此消息是因为您被提及。Message ID: @.***>

[yahskapar]

Great! As I mentioned, for actually training with the motion-augmented data or testing on motion-augmented data, I recommend leaving the outputs as .npy files to avoid any unwanted compression that could occur from writing out to certain video containers. I should also reiterate that the rPPG-Toolbox supports the usage of such files when using supported datasets (e.g., UBFC-rPPG, PURE). More details here.

I'll go ahead and close this issue as the problem has been resolved.