search

cunningham-lab / neurocaas_contrib

Repo to help develop on AMIs associated with NeuroCAAS
GNU General Public License v3.0
2 stars 4 forks source link

Conda environment error, '/root/.keras/keras_tensorflow.json': No such file or directory #2

Open ShuonanChen opened 3 years ago

ShuonanChen commented 3 years ago

In one of the script under [Lambda][LambdaConfig][COMMAND], this part gives an error:

source /home/ubuntu/anaconda3/etc/profile.d/conda.sh
conda activate tensorflow2_p36 && python $neurocaasrootdir/bardensr/test_image_new.py --datafile_name $3 --input_pah $INDIR --output_path $OUTDIR --unused_included 'no' --num_unused 3 >> $LOGDIR/bardensr_out.txt

the error file says: 

cp: cannot stat '/root/.keras/keras_tensorflow.json': No such file or directory
cp: cannot stat '/root/.keras/keras_tensorflow.json': No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 76: /root/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/00_activate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 76: /root/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/00_activate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 76: /root/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/00_activate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 76: /root/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/00_activate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 80: /root/anaconda3/envs/tensorflow2_p36/etc/conda/deactivate.d/deactivate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 80: /root/anaconda3/envs/tensorflow2_p36/etc/conda/deactivate.d/deactivate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 80: /root/anaconda3/envs/tensorflow2_p36/etc/conda/deactivate.d/deactivate.sh: No such file or directory
/home/ubuntu/anaconda3/envs/tensorflow2_p36/etc/conda/activate.d/99_install_tf2.sh: line 80: /root/anaconda3/envs/tensorflow2_p36/etc/conda/deactivate.d/deactivate.sh: No such file or directory
WARNING: Skipping tensorflow-gpu as it is not installed.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '/root/.dl_binaries/tensorflow2/cpu_3.6/file_with_whl_file_name_for_pip'

Any suggestions would be much appreciated, thank you!

jacksonloper commented 3 years ago

One solution is to run the commands as the regular user instead of root? In the template.json, instead of

"COMMAND": "foo.sh"

run

"COMMAND": "sudo -u ubuntu foo.sh"

@cellistigs does that make sense? or will this ruin things later?

cellistigs commented 3 years ago

I have personally not tried that, but I think that should work.

@ShuonanChen could you also try inserting the command: source "/home/ubuntu/.dlamirc"

and see if this fixes the issue? This file contains parameters that set up all of the deep learning ami related resources.

ShuonanChen commented 3 years ago

Thank you,

Adding sudo -u ubuntu before the run_main.sh in the COMMAND in the json file worked! So this problem has solved.

I included source "/home/ubuntu/.dlamirc", but it still gives the same error: cp: cannot stat '/root/.keras/keras_tensorflow.json': No such file or directory. Somehow it is going to root directory. I have .keras/keras_tensorflow.json under the home directory:

$ ls ~/.keras/keras_tensorflow.json  -lt
-rw-rw-r-- 1 ubuntu ubuntu 32 Mar  1 14:40 /home/ubuntu/.keras/keras_tensorflow.json

but dont think i have access to the /root/ directory. Also, this error doesn't happen when using pytorch_p36 instead of tensorflow_p36.