search

Lightning-Universe / lightning-flash

Your PyTorch AI Factory - Flash enables you to easily configure and run complex AI recipes for over 15 tasks across 7 data domains
https://lightning-flash.readthedocs.io
Apache License 2.0
1.74k stars 213 forks source link

`VideoClassificationDataset` does not support frame video folders #1491

Open Skylion007 opened 1 year ago

Skylion007 commented 1 year ago

I realized some of the helper code around the pytorch video does not allow the videos to be frame_video objects, which is disappointing and is supported by the underlying pytorch video.

https://pytorchvideo.readthedocs.io/en/latest/_modules/pytorchvideo/data/labeled_video_dataset.html#labeled_video_dataset

Discussed in https://github.com/Lightning-AI/lightning-flash/discussions/1490

Originally posted by **Skylion007** November 26, 2022 Decoding videos from AVI / MP4 is eating up the CPU on my GPU server, is there a way to have the VideoClassificationData folder accept a folder of frames and maybe some metadata as the video instead? Or recommendation for a video codec that would be faster to decode?
Skylion007 commented 1 year ago

So it looks like it does work if specify the folders manually from_files by globbing those dirs. However, the more intuitive from_folders classmethod appears to filter out all the folders that only contain JPEGs. Loading small samples from these frame_videos is also less efficient than I would have expected, which is a tad disappointing. It can even be slower than loading AVI encoded videos (but faster than H264 ones it seems).

Borda commented 1 year ago

do you have a specific suggestion on how to improve the user/developer experience?