Open EricR86 opened 6 years ago
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
This is likely to have to do with configuration issues on the cluster and not anything to do with Segway's programming. Nor is there anything likely we can really do about this by changing Segway.
Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).
I believe there may be a race condition here in the way observation files are managed in minibatch mode. Currently, observation filenames are not instance-specific. It seems possible, for example, that two instances could simultaneously be attempting to delete or open the same observation file for reading/writing.
Section A.10 of the NFS FAQ also mentions ESTALE errors being reported when referring to items that may have been deleted. The tofile
call is a convenience call for an open/write operation and I'd imagine it's possible for the file to have been deleted by another instance and stale by the time a write happens. I'd imagine this possibly could happen on a non-NFS filesystem but it seems far less likely (and far more likely on NFS filesystems).
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
TMPDIR
to a networked file system. This will be documented as a requirement. Since we only run on Linux we could have segway-task spit out a warning if stat --file-system --format=%T "$TMPDIR"
is nfs
. But the user would only see that if they looked at the error files. Maybe it wouldn't be such a horrible idea for the console to print any errors that occur as it marks a job complete. This is all a bit complex though.
Original report (BitBucket issue) by Mickaël Mendez (Bitbucket: Mickael Mendez).
While running Segway 2.0.2 in reverse mode I ran into a
Stale file handle error
. Below are the logs and the command of the job that failed.segway command
Segway output
EM training error
The source of the error seem to come from the job:
emt0.19.1233.train.637ed75e7b0f11e8975fbd311cda90ee
____ PROGRAM ENDED SUCCESSFULLY WITH STATUS___
This jobs has two unexpected behavior:
jobs.tab