Closed yaroslavvb closed 6 years ago
@yaroslavvb Did an old file somehow get run? I had refactored the dataloader so that that line is no longer called https://github.com/diux-dev/cluster/commit/afb8bfbb9293f7d4d1093a5cbb56b9589dc089ee#diff-75be79d6640e3bb96c7683235830b933L307
aha training is baked into pytorch.v7 AMI, and my task.upload(training) fails silently when directory is present so we end up with old version
gonna fix the upload
I'll try to make sure to delete all the training stuff on the next AMI version
false alarm, upload works as expected, the issue was that my refactored version used script version from ~ instead of ~/training
One of my runs crashed with this error, have you seen something like this? @bearpelican