search

lmb-freiburg / Unet-Segmentation

The U-Net Segmentation plugin for Fiji (ImageJ)
https://lmb.informatik.uni-freiburg.de/resources/opensource/unet
GNU General Public License v3.0
87 stars 25 forks source link

client loop: send disconnect: broken pipe; SFTP Failure 4 #90

Open ledemay opened 2 years ago

ledemay commented 2 years ago

Hello,

I am a PhD student attempting to train a 2D model to segment Purkinje cells using the u-net Fiji plugin. I am running this on a Mac mini OS with an M1 chip, so I am running the CPU only code (because there is no NIVIDIA graphics card) and am using the Ubuntu 18.04 AMI and g4dn instance type. I am very new to this and am having difficulty getting past two errors: after about 8% into fine-tuning, the terminal displays client_loop: send disconnect: broken pipe; soon after the u-net job manager will send an SFTP Failure 4 and abort the job. I am not having issues connecting to my instance, so I think it might be an issue with my bashrc file.

This is the code I have been using to connect and set up my instance:

Ssh -i /Users/neuroanatomy/Desktop/LD_key.pem ubuntu@ec2-3-16-82-227.us-east-2.compute.amazonaws.com

wget https://lmb.informatik.uni-freiburg.de/resources/opensource/unet/caffe_unet_package_18.04_cpu.tar.gz
tar xfvz caffe_unet_package_18.04_cpu.tar.gz
rm -rf u-net
mv caffe_unet_package_18.04_cpu u-net

echo "export PATH=$PATH:/home/ubuntu/u-net/bin" | cat - ~/.bashrc > tmp
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/u-net/lib:/home/ubuntu/u-net/extlib" | cat - tmp > ~/.bashrc

I have been sourcing the bashrc file in the bash profile: Screen Shot 2022-08-24 at 1 23 05 PM But I think my issue is that the bashrc file may not ensure that no output to standard output is being generated in non-interactive mode. I have been running into this issue when running the instance with the bashrc file as is, so I tried to edit it to address this problem, but am still running into the same issue:

Screen Shot 2022-08-24 at 1 29 47 PM

Any insight into this issue will be greatly appreciated.

Thank you!

ThorstenFalk commented 2 years ago

If your .bashrc generates any output to the shell, the plugin may fail. You can easily check this by running

source ~/.bashrc

in a terminal on the server. If no output is generated, it should be fine, otherwise identify the lines generating the output and comment them out. In any case try

caffe --help

and

caffe-unet --help

Both should output a help message from shell. If this is the case, ensure that caffe is not installed via the ubuntu package manager. The ubuntu-Installation comes with a different caffe library which is incompatible to caffe packages in caffe-unet.