Closed vivek-bala closed 8 years ago
Can you please try again? This looks like a connection timeout... Can you log into archer otherwise? Please try with:
/usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/vivek/.ssh/id_rsa -o ControlMaster=auto -o ControlPath=/tmp/saga_ssh_vivek_%h_%p.vb224.ctrl -o TCPKeepAlive=no -o ServerAliveInterval=10 -o ServerAliveCountMax=20 vb224@login.archer.ac.uk
preferably while the application is hanging and prints those HELLO_%d_SAGA
log messages...
Can you please try again? This looks like a connection timeout...
This is repeatable. I get this everytime now. Will try the command you mentioned now.
Can you log into archer otherwise?
Yes.
I get the same error again: saga.exceptions.NoSuccess: Could not detect shell prompt
On executing the following command,
$ /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/vivek/.ssh/id_rsa -o ControlMaster=auto -o ControlPath=/tmp/saga_ssh_vivek_%h_%p.vb224.ctrl -o TCPKeepAlive=no -o ServerAliveInterval=10 -o ServerAliveCountMax=20 vb224@login.archer.ac.uk
I don't get any output/message. It just hangs. This is from the VM. I was able to log into Archer during this time.
That hang is exactly what also then hangs in SAGA :/ I am not exactly sure what to do about that though - no ssh connectivity is nothing we can fix or work around in the radical stack :/
I don't understand the difference between what saga does (assuming the above command is what saga executes) and a normal ssh into the machine (which was successful). Rather, pondering over why one works and not the other.
Can you remove any /tmp/sagassh* files and try again?
Ok, that worked ! The job is now in the queue.
@marksantcroos What exactly was happening ?
The ssh master channel either died or got stuck. Removing that file prompted ssh to create a new one...
I am not exactly sure how to avoid/detect that situation. It is not the first time this happened, but it is also not very frequent, thus hard to reproduce. We probably won't touch that code path right now, but eventually that deserves some recovering mechanism.
Is this issue still relevant for devel
branch? If not I am closing this ticket.
closed due to lack of response
Verbose log: https://gist.github.com/vivek-bala/7735f8c8045375493f18
It seemed to be fine in the morning. Submitted on to the queue and was pending. I reran it now, it fails with
saga.exceptions.NoSuccess: Could not detect shell prompt
.