Closed turian closed 2 years ago
@turian I didn't see this issue before. Did you try to start the instance again? If it fails with this error all the time, can you please send me your config? Also, can you please check stack errors in CloudFormation?
Here is my config:
project:
name: asdf
syncFilters:
- exclude:
- .idea/*
- .git/*
- '*/__pycache__/*'
- _workdir/*
- .mypy_cache/*
- lightning_logs/*
- logs/*
containers:
- projectDir: /workspace/project
image: turian/heareval
ports:
# TensorBoard
- containerPort: 6006
hostPort: 6006
# Jupyter
- containerPort: 8888
hostPort: 8888
volumeMounts:
- name: workspace
mountPath: /workspace
runtimeParameters: ['--shm-size', '20G']
instances:
- name: i1-joseph
provider: aws
parameters:
region: eu-central-1
instanceType: p3.2xlarge
spotInstance: True
ports: [6006, 8888]
volumes:
- name: workspace
parameters:
size: 500
deletionPolicy: retain
mountDir: /workspace
on the AWS management console in CloudFormation, it says the status is "CREATE_COMPLETE".
I tried it again, now I am getting this error:
2021-10-01 12:16:39,189 P2135 [INFO] failed to register layer: Error processing tar file(exit status 1): write /usr/local/lib/python3.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so: no space left on device
I tried again with these volumes but it didn't work, still the disk space error:
volumes:
- name: workspace
parameters:
size: 500
deletionPolicy: retain
mountDir: /workspace
- name: docker
parameters:
mountDir: /docker
size: 200
@turian Try to increase root volume size: add rootVolumeSize: 100
to the instance parameters.
I haven't used spotty on AWS for a while. Now, I am getting the following error.
How do I resolve this?
and the log says: