Closed wilke closed 1 year ago
Command: singularity run --nv --bind
pwd/tmp:/candle_data_dir build/TGSA.sif train.sh 1 /candle_data_dir --epochs 1
Status: Failed
Output:
CMD = python /usr/local/TGSA/candle_train.py --epochs 1
/usr/local/TGSA/train.sh: line 59: ./candle_glue.sh: No such file or directory
using original data placed in /candle_data_dir
/candle_data_dir//Data
/usr/local/TGSA/train.sh: line 77: ./candle_glue.sh: No such file or directory
using original data placed in /candle_data_dir//Data
using CUDA_VISIBLE_DEVICES 1
...
'weight_path': ''}
Training on Pilot1 dataset
Traceback (most recent call last):
File "/usr/local/TGSA/candle_train.py", line 205, in <module>
main()
File "/usr/local/TGSA/candle_train.py", line 202, in main
run(gParams)
File "/usr/local/TGSA/candle_train.py", line 111, in run
os.makedirs(output_root_dir)
File "/opt/conda/lib/python3.7/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/opt/conda/lib/python3.7/os.py", line 223, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/usr/local/TGSA/benchmark_dataset_generator/improve_data_dir'
Command works but not printing IMPROVE_RESULT
Fixed and pull requested to improve branch
Starting over for next release cycle.
Command:
train.sh 1 ./tmp
singularity run --bind
pwd/tests:/candle_data_dir --nv build/TGSA.sif train.sh 4 /candle_data_dir --epochs 1
Status: In progress
Depends on:
33
53
Output:
1: