search

iejMac / clip-video-encode

Easily compute clip embeddings from video frames
MIT License
136 stars 19 forks source link

set up resuming for failed runs #57

Open iejMac opened 2 years ago

iejMac commented 2 years ago

something like rom suggested where you generate input shard first then you just test if the output shard for that input shard already exists

iejMac commented 2 years ago
iejMac commented 1 year ago

make it like in video2dataset using input_sharder and all

iejMac commented 1 year ago

or we can hack it for now: when you're done writing a shard, write a key_stat.json file as a signal for that shard being done

iejMac commented 1 year ago

based on found .json files in dataset folder update this https://github.com/iejMac/clip-video-encode/blob/44f2214032c9219663899cd4e3ef69724af6f619/clip_video_encode/clip_video_encode.py#L113