save embeddings during generation to avoid potential time limit issues on slurm

hassonlab / 247-pickling

Contains code to create pickles from raw/processed data

1 stars 10 forks source link

Closed hvgazula closed 1 month ago

hvgazula commented 1 year ago

Write to disk after every k batches. @zkokaja and @VeritasJoker, Any other ideas?

hvgazula commented 1 month ago

not an issue anymore with new and powerful GPUs