search

iejMac / clip-video-encode

Easily compute clip embeddings from video frames
MIT License
136 stars 19 forks source link

bug fix: each webdataset loses the last 4% of the data #54

Closed iejMac closed 2 years ago

iejMac commented 2 years ago

when using no distribution 4800/5000 videos get written when using distribution into 8 workers 4975/5000 videos get written

iejMac commented 2 years ago

it's the last "chunk", so we're not checking for the last chunk correctly here https://github.com/iejMac/clip-video-encode/blob/fffb5043c309890b619132a75808a365ef87121e/clip_video_encode/clip_video_encode.py#L121

iejMac commented 2 years ago

the problem is that some videos don't load and then you won't get to n_vids