Dangerous advice: it looks that by default the dataset is at /data/milatmp1

rdevon / cortex

A machine learning library for PyTorch

BSD 3-Clause "New" or "Revised" License

92 stars 25 forks source link

Dangerous advice: it looks that by default the dataset is at /data/milatmp1 #18

Closed dmitriy-serdyuk closed 6 years ago

dmitriy-serdyuk commented 6 years ago

If this is true, Fred is not going to be happy when a lot of people will be hammering NFS.

Given that an experiment is run by a magic script, it is simple to copy data to /Tmp.

rdevon commented 6 years ago

Wait, where is milatmp1 defaulting?

dmitriy-serdyuk commented 6 years ago

In the readme:

torchvision_data_path: /data/milatmp1/hjelmdev/data/

rdevon commented 6 years ago

Ah, right. So one function I need to implement is to designate a local path to which the data is copied before training. Haven't gotten around to it, but I can add it to the todos

nouiz commented 6 years ago

If you have a quick implementation, then good. Otherwise, we are planning to have the local computer cache the networks file system. All new computer we plan to buy will have the hardware for this. Currently, there is a flag to our slurm config to go on computer configured like this.