search

luketerry0 / distributed_dino_embedding

0 stars 0 forks source link

Process images in batches #1

Open luketerry0 opened 1 month ago

luketerry0 commented 1 month ago

Process images in batches, being careful to record which embeddings correspond to which filenames

luketerry0 commented 1 month ago

Refactor to use torchrun and distribute over multiple nodes

JayRothenberger commented 1 month ago

Don't forget that images will have to be sent .to(device) to be used with a model on the gpu

JayRothenberger commented 1 month ago

.append(batch) to a list in each rank and then stack those batches together when you save them out at the end. Make sure to also save the list of file names in the same order.