No space left on device

paymog commented 1 month ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Atlantis errored out saying there's no space left on the device.

Reproduction Steps

Don't know how to reproduce.

Logs

running "terraform -chdir=ci-cdktf.out/stacks/goldsky-infra-dev init" in "/atlantis-data/repos/goldsky-io/goldsky-infra/1299/default": exit status 1: running "terraform -chdir=ci-cdktf.out/stacks/goldsky-infra-dev init" in "/atlantis-data/repos/goldsky-io/goldsky-infra/1299/default": 

Initializing the backend...

Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/http versions matching "2.1.0"...
- Finding hashicorp/kubernetes versions matching "2.31.0"...
- Finding hashicorp/random versions matching "3.1.3"...
- Finding hashicorp/tls versions matching "4.0.5"...
- Finding hashicorp/aws versions matching "5.61.0"...
- Finding dopplerhq/doppler versions matching "1.1.6"...
- Finding hashicorp/helm versions matching "2.14.1"...
- Installing hashicorp/tls v4.0.5...
- Installed hashicorp/tls v4.0.5 (signed by HashiCorp)
- Installing hashicorp/aws v5.61.0...
- Installing dopplerhq/doppler v1.1.6...
- Installing hashicorp/helm v2.14.1...
- Installing hashicorp/http v2.1.0...
- Installing hashicorp/kubernetes v2.31.0...
- Installing hashicorp/random v3.1.3...
╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/aws v5.61.0: write
│ .terraform/providers/registry.terraform.io/hashicorp/aws/5.61.0/linux_amd64/terraform-provider-aws_v5.61.0_x5:
│ no space left on device
╵

╷
│ Error: Failed to install provider
│ 
│ Error while installing dopplerhq/doppler v1.1.6: mkdir
│ .terraform/providers/registry.terraform.io/dopplerhq: no space left on
│ device
╵

╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/helm v2.14.1: mkdir
│ .terraform/providers/registry.terraform.io/hashicorp/helm: no space left on
│ device
╵

╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/http v2.1.0: mkdir
│ .terraform/providers/registry.terraform.io/hashicorp/http: no space left on
│ device
╵

╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/kubernetes v2.31.0: mkdir
│ .terraform/providers/registry.terraform.io/hashicorp/kubernetes: no space
│ left on device
╵

╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/random v3.1.3: mkdir
│ .terraform/providers/registry.terraform.io/hashicorp/random: no space left
│ on device

Environment details

Using default volume in k8s. Here's the disk info (it may have cleaned itself up)

atlantis-0:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  20.0G     15.4G      4.6G  77% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                    61.5G         0     61.5G   0% /sys/fs/cgroup
/dev/nvme1n1              4.8G      1.7G      3.0G  37% /atlantis-data
tmpfs                   120.2G      4.0K    120.2G   0% /var/github-app
/dev/nvme0n1p1           20.0G     15.4G      4.6G  77% /etc/hosts
/dev/nvme0n1p1           20.0G     15.4G      4.6G  77% /dev/termination-log
/dev/nvme0n1p1           20.0G     15.4G      4.6G  77% /etc/hostname
/dev/nvme0n1p1           20.0G     15.4G      4.6G  77% /etc/resolv.conf
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   120.2G      4.0K    120.2G   0% /home/atlantis/.aws
/dev/nvme0n1p1           20.0G     15.4G      4.6G  77% /etc/atlantis/repos.yaml
tmpfs                   120.2G     12.0K    120.2G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    61.5G         0     61.5G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                    61.5G         0     61.5G   0% /sys/firmware

Additional Context

Running helm chart 5.4.4

jack9410 commented 1 month ago

Getting this error after upgrading helm chart to 5.0.1. Never had this error while using helm chart 4.x.x.

Almenon commented 3 weeks ago

Terraform modules and providers can take up a lot of disk space. You could try adding disk space.

paymog commented 3 weeks ago

I bumped the volume to 20gb and haven't had the issue since

snorlaX-sleeps commented 3 weeks ago

A second solution (outside of increasing the disk space) is to unlock or close PR's - this will delete the workspace and providers from disk and free up space

runatlantis / atlantis