search

spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
492 stars 43 forks source link

spotty download fails because /tmp/spotty is missing #50

Closed vadimkantorov closed 5 years ago

vadimkantorov commented 5 years ago

spotty download relies on /tmp/spotty/instance/scripts/upload_files.sh being present on an instance https://github.com/apls777/spotty/blob/master/spotty/providers/aws/deployment/cf_templates/data/instance.yaml#L175

Somehow in my case it's not there. When I ssh on an instance, I see in /tmp only my uploaded project. It seems that my project directory kind of overwrote the /tmp scripts.

So the command that should upload files from instance to S3 fails, but quietly. There is no error forwarded to the suer.

vadimkantorov commented 5 years ago
  1. I mixed up the docker /tmp and host instance /tmp
  2. The include filters didn't capture my files because I configured projectDir: /workspace/project, but put checkpoints in /workspace/experiments and it's impossible to retrieve files outside the 'projectDir' with the current code of upload_files.sh