This completes the migration of the snapshotEngine to work with DigitalOcean.
Changes
Converted busy waits to waits that sleep 1 minute. This resulted in a 30x-60x reduction in CPU time for idling pods. 60% of a core to 1%-2% of a core for idle pods.
DigitalOcean has a snapshot timeout of 10 minutes so the Snapshot Warmer has been reduced to only creating a snapshot and then waiting for 10 minutes. It maintains 5 snapshots still, but it does not delete "stuck" snapshots anymore.
For some reason localhost does not work from the Tezos container when trying to reach the RPC so this was changed to 127.0.0.1.
Both AWS and DigitalOcean credentials are stored as Kubernetes secrets.
Storage Class has been changed from AWS' ebs-sc to DigitalOceans' do-block-storage.
Snapshot Maker pod no longer waits for a node to be ready as this logic was moved to both the Snapshot Scheduler pod and the Snapshot Warmer pods. Before snapshots could be taken if a node was not synced, now that is not possible.
Sleeps were added to creation and deletion commands as DigitalOcean's API timeouts are stricter than AWS.
Snapshot Maker no longer waits for a snapshot to be finished and just uses the last snapshot as they are near instantaneous on DigitalOcean volumes.
Alternate cloud provider functionality was removed as we are currently supporting web resources in AWS and Kubernetes and artifacts on DigitalOcean.
Artifact metadata is pushed to DigitalOcean Spaces instead of AWS S3.
Jekyll build output was quieted as it fills the logs with non-useful information when troubleshooting. When Jekyll fails another message before or after is more helpful anyways.
Cloud provider credential switching logic was changed to default to Digital Ocean and aws if provided.
This completes the migration of the snapshotEngine to work with DigitalOcean.
Changes
localhost
does not work from the Tezos container when trying to reach the RPC so this was changed to127.0.0.1
.ebs-sc
to DigitalOceans'do-block-storage
.aws
if provided.This branch is currently deployed and running via https://github.com/oxheadalpha/xtz-shots-infra