Staging 503 error

[that1guy]

Hi Josh,

Just did my second push of day to see if most recent code would appear on staging. Jenkins indicates one of the containers is out of swap space, thus getting a 503 error on staging.

Demoing to Joe tomorrow (investor) so would like your help resolving this by tomorrow morning. Let me know what the trick is and I'll perform fix in future if this happens again. :)

Thanks! -Brad

[brozeph]

The 503 and the build server swap space are separate issues, but why you are getting a 503 from hts-app is due to the fact that staging-docker-host (not the build server) is out of space.

docker logs hts-app

Running "nodemon:stage" (nodemon) task
Fatal error: ENOSPC, no space left on device

To resolve, I'm optimizing the Dockerfile for every project and attempting to eek out as much additional free space as possible. This should free up a good amount of space, but longer term, I'll need to determine how to allocate a larger volume for the overlay file system.

[brozeph]

Ok - legit - found a way to fix, but this will require a minor blip in availability of staging. Since it's already returning 503 errors, I figure this is acceptable for now...

sudo fdisk /dev/xvda

1. select p to list partitions, verify that xvda1 exists
2. write down the Start column value
3. select d to delete xvda1 (don't worry, nothing has happened yet until the changes are written)
4. select n to create new partition
   1. select p to make the partition primary
   2. enter 1 for partition number
   3. enter the value from the Start column recorded in step 2 to set start of First Sector (in my case, this was 16065 on two machines running Rancher OS)
   4. hit enter to accept the default value for Last Sector
5. select a to set a bootable partition
   1. enter 1 for partition number
6. pray
7. enter w to write the changes (no changes will be made just yet - you will see a message asking to reboot the machine)

Now, reboot the box:

sudo reboot

Once the machine is back up, SSH in and run one final command (this will make the existing xvda1 partition span the entire disk that is allocated for it:

sudo resize2fs /dev/xvda1

To verify:

df -h