Cluster execution of scripts

YanCote / IFT6268-simclr

Project for IFT6268

0 stars 0 forks source link

Cluster execution of scripts #19

Open YanCote opened 3 years ago

YanCote commented 3 years ago

Execute one Task on a Cluster (CC / MILA)

Execute a Module.py on a chosen ressources Does the tensorflow code support running on MultiGPUs

Find where to save the Datasets

YanCote commented 3 years ago

Task are killed randomly on CC Graham cluster, just opened a ticker to get support, no way to understand what's going on

marued commented 3 years ago

This was finally fixed. Lesson learned: tf.Dataset.Generator is not compatible with TF v1. We had to replace the generator with other map operations to load images and do the other data processing beforehand.

YanCote commented 3 years ago

So we are able to run TF1 script on GPU on CC's graham. We still need to validate running with Multiple GPU and the optimal place to store our dataset