Audiracmichelle/issue64

[audiracmichelle]

64

[ekarais]

Thanks @audiracmichelle! A few things:

• In my new branch, I am moving the datasets/ folder into scratch, as the folder will grow to 100G+ once I relaunch all the experiments (lots of unique .jsonl files will be created). There, I introduced a new environment variable called STORAGE_PATH that the user sets in their .env file. Then, the scripts create STORAGE_PATH/datasets/ and STORAGE_PATH/checkpoints/ when they are needed.
• We started tracking datasets/ as a quick hack for Google Colab, my plan was to stop tracking this folder entirely. Is there a good reason from the spatial data's side to keep tracking this?

[audiracmichelle]

that sounds good, although Mauricio and I were discussing to keep a dataplaceholder and arrange things so that the InMemoryDataset created the data under the following structure: data/qm9_hash1 /raw /processed data/qm9_hash2 /raw /processed ... data/spatial_cc /raw /processed

[audiracmichelle]

regarding the scratch folder, it is ok to use the inviron but let's just use a symlink to scratch in cannon instead