search

JDACS4C-IMPROVE / Singularity

Singularity definitions that can be extended to support execution of community models.
MIT License
3 stars 5 forks source link

Python files for testing #66

Closed rajeeja closed 1 year ago

RylieWeaver commented 1 year ago

Model Curation Testing (python test_model_curation.py --model_name=DeepTTC --model_source_dir=../../DeepTTC/):

I tried with DrugCell, DeepTTC, and GraphDRP. Heavy on that the necessary dependencies are required. Some models don't have everything in a requirements.txt, or it's named different, or it has errors. Plus, some models don't have correct naming conventions for baseline/config files.

CANDLE_DATA_DIR also needed to be defined before I could run. Simply exporting worked.

With some debugging, I was able to get GraphDRP to work. Looks like your process works as long as things are defined correct.

RylieWeaver commented 1 year ago

Model Build Testing (python build_container.py --model_name DeepTTC):

Got this error, essentially saying the def file was not found:

(GraphDRP_py37) weaverr@lambda0:~/Singularity/test$ python build_container.py --model_name DeepTTC
Deployment directory exists: ..//build/
Singularity container does not exist ..//build/DeepTTC.sif
Definition file exists: ..//definitions/DeepTTC.def
Namespace(definitions_file='..//definitions/DeepTTC.def', deployment_dir_file='..//build/DeepTTC.sif', model_name='DeepTTC', options='--fakeroot --disable-cache ', singularity_dir='../')
Current working directory: /lambda_stor/homes/weaverr/Singularity/test
change directory to the singularity directory
Current working directory: /lambda_stor/homes/weaverr/Singularity
Running command: singularity build --fakeroot --disable-cache ..//build/DeepTTC.sif ..//definitions/DeepTTC.def
Standard Output:

Standard Error:
FATAL:   Unable to build from ..//definitions/DeepTTC.def: unable to open file ..//definitions/DeepTTC.def: open ..//definitions/DeepTTC.def: no such file or directory

Build completed, created, exit code: 255

I pushed some changes that seemed to fix this for me, although I'm unsure of their robustness. In the end, I get an error with permissions to build in /build/:

(GraphDRP_py37) weaverr@lambda0:~/Singularity/test$ python build_container.py --model_name DeepTTC
Traceback (most recent call last):
  File "build_container.py", line 56, in <module>
    os.mkdir(deployment_dir)
PermissionError: [Errno 13] Permission denied: '/build/'
RylieWeaver commented 1 year ago

Testing Container (python test_container.py --model_name DeepTTC):

I get this error:

(GraphDRP_py37) weaverr@lambda0:~/Singularity1/test$ python test_container.py --model_name DeepTTC
Singularity container already exists:  ..//images/DeepTTC.sif
Definition file exists: ..//definitions/DeepTTC.def
Namespace(candle_data_dir='/tmp/', definitions_file='..//definitions/DeepTTC.def', deployment_dir_file='..//images/DeepTTC.sif', gpuid='0', model_name='DeepTTC', options='--epochs 1', singularity_dir='../')
Singularity container exists
Running singularity container
Running command: singularity run --nv --bind /tmp/:/candle_data_dir ..//images/DeepTTC.sif train.sh 0 /candle_data_dir --epochs 1
Standard Output:
b"CMD = python /usr/local/DeepTTC/deepttc_baseline_pytorch.py --epochs 1\nusing CUDA_VISIBLE_DEVICES 0\nusing CANDLE_DATA_DIR /candle_data_dir\nusing CANDLE_CONFIG \nrunning command python /usr/local/DeepTTC/deepttc_baseline_pytorch.py --epochs 1\nImporting candle utils for pytorch\nmodel name:  deepttc\nConfiguration file:  /usr/local/DeepTTC/DeepTTC.default\nParams:\n{'batch_size': 512,\n 'benchmark_dir': '/root/git/ModelTesting/benchmark/IMP_data',\n 'benchmark_result_dir': 'results',\n 'cancer_id': 'TCGA_DESC',\n 'ckpt_checksum': False,\n 'ckpt_directory': None,\n 'ckpt_keep_limit': 5,\n 'ckpt_keep_mode': 'linear',\n 'ckpt_restart_mode': 'auto',\n 'ckpt_save_best': True,\n 'ckpt_save_best_metric': 'val_loss',\n 'ckpt_save_interval': 0,\n 'ckpt_save_weights_only': False,\n 'ckpt_skip_epochs': 0,\n 'data_dir': '/candle_data_dir/deepttc/Data',\n 'data_type': <class 'numpy.float32'>,\n 'default_data_url': 'https://ftp.mcs.anl.gov/pub/candle/public/improve/reproducability/DeepTTC/',\n 'dropout': 0.1,\n 'drug_id': 'DRUG_ID',\n 'epochs': 1,\n 'experiment_id': 'EXP000',\n 'generate_input_data': True,\n 'input_dim_drug': 2586,\n 'input_dim_drug_classifier': 128,\n 'input_dim_gene_classifier': 256,\n 'learning_rate': 0.0001,\n 'logfile': None,\n 'mode': 'run',\n 'model_name': 'deepttc',\n 'output_dir': '/candle_data_dir/deepttc/Output/EXP000/RUN000',\n 'predictions_output': 'output/results.tsv',\n 'profiling': False,\n 'rng_seed': 1,\n 'run_id': 'RUN000',\n 'sample_id': 'COSMIC_ID',\n 'save_data': 'True',\n 'shuffle': False,\n 'target_id': 'LN_IC50',\n 'timeout': -1,\n 'train_bool': True,\n 'transformer_attention_probs_dropout': 0.1,\n 'transformer_emb_size_drug': 128,\n 'transformer_hidden_dropout_rate': 0.1,\n 'transformer_intermediate_size_drug': 512,\n 'transformer_n_layer_drug': 8,\n 'transformer_num_attention_heads_drug': 8,\n 'use_lincs': False,\n 'verbose': False,\n 'vocab_dir': 'GDSC_data'}\n"

Standard Error:
b'+ CANDLE_MODEL=deepttc_baseline_pytorch.py\n+ IMPROVE_MODEL_DIR=/usr/local/DeepTTC\n+ CANDLE_MODEL=/usr/local/DeepTTC/deepttc_baseline_pytorch.py\n+ \'[\' 4 -lt 2 \']\'\n+ \'[\' 4 -eq 2 \']\'\n+ \'[\' 4 -eq 3 \']\'\n+ CUDA_VISIBLE_DEVICES=0\n+ shift\n+ CANDLE_DATA_DIR=/candle_data_dir\n+ shift\n+ CONFIG_PATH=/candle_data_dir/--epochs\n+ \'[\' -f /candle_data_dir/--epochs \']\'\n+ CMD=\'python /usr/local/DeepTTC/deepttc_baseline_pytorch.py --epochs 1\'\n+ echo \'CMD = python /usr/local/DeepTTC/deepttc_baseline_pytorch.py --epochs 1\'\n+ echo \'using CUDA_VISIBLE_DEVICES 0\'\n+ echo \'using CANDLE_DATA_DIR /candle_data_dir\'\n+ echo \'using CANDLE_CONFIG \'\n+ echo \'running command python /usr/local/DeepTTC/deepttc_baseline_pytorch.py --epochs 1\'\n+ export CUDA_VISIBLE_DEVICES=0\n+ CUDA_VISIBLE_DEVICES=0\n+ export CANDLE_DATA_DIR=/candle_data_dir\n+ CANDLE_DATA_DIR=/candle_data_dir\n+ python /usr/local/DeepTTC/deepttc_baseline_pytorch.py --epochs 1\n/opt/conda/lib/python3.10/site-packages/candle/parsing_utils.py:743: RuntimeWarning: These keywords used in the configuration file are not defined in CANDLE: [\'default_data_url\', \'predictions_output\']\n  warnings.warn(message, RuntimeWarning)\n/opt/conda/lib/python3.10/site-packages/candle/file_utils.py:217: RuntimeWarning: Path: /candle_data_dir/deepttc/Output/EXP000/RUN000 already exists... overwriting.\n  warnings.warn(message, RuntimeWarning)\nTraceback (most recent call last):\n  File "/usr/local/DeepTTC/deepttc_baseline_pytorch.py", line 279, in <module>\n    main()\n  File "/usr/local/DeepTTC/deepttc_baseline_pytorch.py", line 266, in main\n    gParameters = initialize_parameters()\n  File "/usr/local/DeepTTC/deepttc_baseline_pytorch.py", line 221, in initialize_parameters\n    gParameters = candle.finalize_parameters(common)\n  File "/opt/conda/lib/python3.10/site-packages/candle/parsing_utils.py", line 809, in finalize_parameters\n    with open(final_params, "w") as fp:\nPermissionError: [Errno 13] Permission denied: \'/candle_data_dir/deepttc/Output/EXP000/RUN000/final_params.txt\'\n'
Testing cointainer completed, exit code: 1

This has to do with write permissions to /tmp/, which you made default candle_data_dir. I can write to /tmp/, but I think not to directories in /tmp/ that were created by other (such as /tmp/deepttc). This command worked for me:

python test_container.py --model_name DeepTTC --candle_data_dir /tmp/weaverr
rajeeja commented 1 year ago

@RylieWeaver9 can you test it again?

RylieWeaver commented 1 year ago

@RylieWeaver9 can you test it again?

Same errors at the moment. This was after a git pull of rajeeja/testing

When trying to build: OSError: [Errno 13] Permission denied: '/build/'

When trying to test container: PermissionError: [Errno 13] Permission denied: '/candle_data_dir/deepttc/Output/EXP000/RUN000/final_params.txt'