pzelasko
commented
5 months ago Hi @kerolos, thanks for opening this discussion! I can help you get your video recipe set up.
Is there a plan to add a recipe that supports video data in Lhotse?
There is one AV recipe currently for GRID AV corpus: https://github.com/lhotse-speech/lhotse/blob/master/lhotse/recipes/grid.py In general, lhotse recipes download and prepare the manifests for datasets, but actual training is out of lhotse's scope. You may want to set up a separate repository with your experiment's code that imports lhotse.
How can I start using customized features, for example, using MediaPipe Pose Estimation tools?
Once you create a recording, you can load the video, process it with some module, and save + attach as a custom field to the cut. For example:
video_recording = Recording.from_file("/path/to/-fZc293MpJk_0-1-rgb_front.mp4") # lhotse will auto-construct video recording manifest
video_cut = video_recording.to_cut()
video_frames = video_cut.load_video() # video frames is a uint8 np.array with shape (T, C, H, W) [or some other permutation, I don't remember off the top of my head]
video_features = compute_some_features(video_frames) # video_features is np.array with arbitrary shape
# Option 1 -> save to some storage directly
# temporal_dim indicates which dimension in video_features shape corresponds to time; set accordingly.
with NumpyHdf5Writer("video_features.h5") as writer:
video_cut.video_features = writer.store_array(video_cut.id, video_features, frame_shift=video_recording.video.fps, temporal_dim=0)
# Option 2 -> holds data in memory, write to some storage later (useful if you're going to use Lhotse Shar format):
video_cut = video_cut.attach_tensor("video_features", video_features, frame_shift=video_recording.video.fps, temporal_dim=0)
If you save the final video_cut, you can then later load video_features with cut.load_video_features() and access the manifest via cut.video_features (special field and method are auto-added for custom fields registered via attach_tensor). You can compute many different features and attach all of them under different names.
What format should be used to save the extracted features (i.has_features ) and saved as features and load them later for training ?
I would use one of numpy format writers in lhotse (e.g. NumpyHdf5Writer in the example above). Don't use lilcom unless you are sure it makes sense (it is a lossy format optimized for log-domain features). You may also want to explore lhotse shar format which I think should work with video recordings (and definitely works with video features extracted as above). It is better optimized for I/O which might help you process large video data in training.
That said, video features would likely require better compression for very large datasets, which is something we can explore later.
i have also Frames per second, it is not always fixed it in between 24 fps to 50 fps , how can i deal with that ?
You can access the fps via recording.video.fps or cut.video.fps. If you want to resample the video, you have two options: 1) either load the whole thing / cut of a given duration, and downsample/resample then in python; 2) leverage torchaudio ffmpeg bindings to resample the video (you might need to check out their tutorials to learn how to pass specific ffmpeg transform commands and find a way to expose/add it in AudioSource API. For reference, this code loads the video)
Final comment, Recording manifest doesn't support custom fields, so you'd be better off moving feature_path key to supervision as {..., "custom": {"feature_path": ...}}
kerolos
Extend Lhotse to support video features for tasks such as sign language recognition (e.g., How2Sign) and human activity recognition. This enhancement will be useful for the Icefall platform.
Details
With the recent support for video in PR #1151, I am interested in developing a new recipe to handle video data and extract features using tools like MediaPipe.
Objectives
Recipe Addition:
lhotse/recipesdirectory.Feature Extraction:
Implementation Steps
Create Manifest Files:
Recordings manifest (
recordings.jsonl):Supervisions manifest (
supervisions.jsonl):Feature Extraction Script:
Create a script
compute_features_sign_language.py:Questions
I would appreciate any guidance or support on implementing this feature and utilizing it within the Icefall platform @pzelasko .
Thank you!