iejMac / clip-video-encode

Easily compute clip embeddings from video frames
MIT License
132 stars 19 forks source link

clip-video-encode: implement resuming failed runs #62

Open iejMac opened 1 year ago

iejMac commented 1 year ago

this is VERY hacky for now. It's going to require more work to make this be clean and work well. We need to implement input sharding because currently all videos are fed as a constant stream into the SharedQueue based on which finish reading first so it's hard to determine which videos are in which shard. I temporarily solved this by iterating over 'fake shards' by just indexing the full arrays