search

ECP-CANDLE / candle_lib

MIT License
1 stars 9 forks source link

DRAFT: datadir and creation of directories in CDD #18

Closed rajeeja closed 1 year ago

rajeeja commented 1 year ago

For CANDLE_MODEL_TYPE BENCHMARKS we should never create dirs or touch anything in CDD (CANDLE_DATA_DIR). For CANDLE_MODEL_TYPE SINGULARITY we can create these directories and store results.

More work needed to guard these two cases.

See build 225 and 226 here: https://jenkins-gce.cels.anl.gov/job/CANDLE-WF-mlrMBO-NT3/

This is context for adapting things for the IMPROVE project.

Where is datadirs used? Is it only for IMPROVE or was it used for ECP-CANDLE/Benchmarks?

jmohdyusof commented 1 year ago

When was CANDLE_MODEL_TYPE discussed?

rajeeja commented 1 year ago

@jmohdyusof please review these changes, w/o it were Jenkins would break as it was creating new directories CDD/.. and it is a non-writeable area for Jenkins

jmohdyusof commented 1 year ago

I don't see how this solves anything. Just set CANDLE_DATA_DIR to a place Jenkins can write

rajeeja commented 1 year ago

I don't see how this solves anything. Just set CANDLE_DATA_DIR to a place Jenkins can write

CANDLE_MODEL_TYPE "BENCHMARKS" assumes that "Data" directory is for input files and shouldn't have any outputs. For BENCHMARKS aka non SINGULARITY CANDLE_MODEL_TYPE - CDD should not create anything in CDD area.

CDD on Jenkins is a place for all the input files and those files are big, there is a space constraint and so we use a globalscratch to store those files.

jmohdyusof commented 1 year ago

This PR doesn't have any code to process CANDLE_MODEL_TYPE. If get_file is not called, I don't see how this is affecting Jenkins. Why not define a directory for Jenkins separately

rajeeja commented 1 year ago

This PR doesn't have any code to process CANDLE_MODEL_TYPE. If get_file is not called, I don't see how this is affecting Jenkins. Why not define a directory for Jenkins separately

grep PermissionError in https://jenkins-gce.cels.anl.gov/job/CANDLE-P1B1-torch-ckpt/236/console

https://jenkins-gce.cels.anl.gov/job/CANDLE-P1B1-torch-ckpt/237/console works, because it uses the PR branch.

rajeeja commented 1 year ago

This PR doesn't have any code to process CANDLE_MODEL_TYPE. If get_file is not called, I don't see how this is affecting Jenkins. Why not define a directory for Jenkins separately

CDD is used for finding the input files for Benchmarks and I can't change that. The only requirement is not to create directories in CDD area. We'd have to comeup with a flag or way to establish this.

jmohdyusof commented 1 year ago

I don't understand where this path is coming from: PermissionError: [Errno 13] Permission denied: '/nfs/gce/globalscratch/jain/Data/p1b1' The code generates a path: datadir = os.path.abspath(os.path.join(datadir, params["model_name"], "Data")) which is different

In any case, this PR just removes the makedirs so it will break other workflows.

rajeeja commented 1 year ago

I don't understand where this path is coming from: PermissionError: [Errno 13] Permission denied: '/nfs/gce/globalscratch/jain/Data/p1b1'

Yes, we need a solution here. for this problem CANDLE_DATA_DIR is set to /nfs/gce/globalscratch/jain/Data/

p1b1 is appended by _candlelib and it is trying to create the dir it should not.

jmohdyusof commented 1 year ago

What data is in CDD that is needed by the run? We had a branch where if CDD was an absolute path it was not modified, this seems like it would fix this problem?

rajeeja commented 1 year ago

yes, that would be fine. CDD has all the Data (csv files) already downloaded, it is similar to our Benchmarks/Data/Pilot1 directory.