Closed wilke closed 11 months ago
strange i cant seem to replicate this on my end
Here is the command i tried singularity exec --nv --bind /homes/ac.rgnanaolivu/improve_data_dir/DrugCell/tmp/:$PWD/data /homes/ac.rgnanaolivu/improve_data_dir/Singularity/images/DrugCell:0.0.1-20230914.sif train.sh $CUDA_VISIBLE_DEVICES $CANDLE_DATA_DIR
I was able to reproduce the error and fix. It had to do with CUDA GPU / numpy CPU communication of the test loss tensor. Double check when you can Rohan, I made the change on the 'rylie' branch and pushed to develop.
Thanks Rylie
Did run. build and execute image:
Best performed model (epoch) 0 {'test_loss': 0.0008328101038932801, 'test_pcc': -2.8085122494303505e-07, 'test_MSE': 0.8901301622390747, 'test_r2': -19.574140548706055, 'test_scc': nan, 'IMPROVE RESULT': 0.8901301622390747}
IMPROVE_RESULT MSE: 0.8901301622390747
Commands:
singularity run --nv --bind
pwd/tmp/:/candle_data_dir build/DrugCell.sif train.sh 4 /candle_data_dir --epochs 1
Error:
Output: