devops: snapshot service

troian commented 1 month ago

Is your feature request related to a problem? Please describe.

Akash Network needs to have backup source of snapshot which is:

ui

authorization by Github org
show list of supported networks
- show list of snapshots
- schedule snapshot on specific height
add new network

snapshots:

snapshots for both mainnet and sandbox on following schedule:
- every hour (keep only latest)
- every 24h (keep for last 7 days)
- schedule snapshot at specific height (for example prior network upgrade)

storage Minio seems to be the best option, but we're open to consider others

Describe the solution you'd like

n/a

Describe alternatives you've considered

n/a

Search

[X] I did search for other open and closed issues before opening this

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Additional context

No response

Zblocker64 commented 1 month ago

I am actively working on this

Zblocker64 commented 3 weeks ago

Just giving an update on my progress so far...

I have created scripts that prune and create snapshots every hour and uploads them to a minio file server. I have also created scripts that look for an upgrade and will auto create snapshots before and after an upgrade. I am working on creating the production minio server for high availability and fast transfer speed. I am looking to be done mid July with a fully functional snapshot service.

Zblocker64 commented 6 days ago

Here is the main script that makes the snapshot. Still in development and not considered to be final.

# Variables
MINIO_ENDPOINT="http://download.akashprovid.com"
MINIO_ACCESS_KEY="keygohere"
MINIO_SECRET_KEY="secretgohere"
MINIO_BUCKET="akashnet-2"
MINIO_PATH="/"
MINIO_ALIAS="myminio"
CHAIN_ID="akashnet-2"
SNAP_PATH="$HOME/snapshots/akash"
LOG_PATH="$HOME/snapshots/akash/akash_log.txt"
DATA_PATH="$HOME/.akash"
SERVICE_NAME="akash-node"
RPC_ADDRESS="http://localhost:26657"
SNAP_NAME="${CHAIN_ID}_latest.tar.lz4"

# Sets the Minio binary path and renames the previous hours snapshot
export PATH=$PATH:$HOME/minio-binaries/
mc alias set ${MINIO_ALIAS} ${MINIO_ENDPOINT} ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}
mc mv ${MINIO_ALIAS}/${MINIO_BUCKET}/${SNAP_NAME} ${MINIO_ALIAS}/${MINIO_BUCKET}/${NEW_SNAP_NAME}

log_this() {
    local logging="$@"
    printf "|$(now_date)| $logging\n" | tee -a ${LOG_PATH}
}

# Gets latest block height, not currently using but could use this to name snapshot based off of block height.
LAST_BLOCK_HEIGHT=$(curl -s ${RPC_ADDRESS}/status | jq -r .result.sync_info.latest_block_height)
log_this "LAST_BLOCK_HEIGHT ${LAST_BLOCK_HEIGHT}"

# Stop the Node
log_this "Stopping ${SERVICE_NAME}"
systemctl stop ${SERVICE_NAME}; echo $? >> ${LOG_PATH}

# Compresses the data folder, this is the snapshot file
log_this "Creating new snapshot"
time tar -cf - -C ${DATA_PATH} data 2>>${LOG_PATH} | lz4 -c -9 > ${SNAP_PATH}/${SNAP_NAME} 2>>${LOG_PATH}

# Starts the node
log_this "Starting ${SERVICE_NAME}"
systemctl start ${SERVICE_NAME}; echo $? >> ${LOG_PATH}

# Moves the compressed snapshot location
log_this "Moving new snapshot to ${SNAP_PATH}"
du -hs ${SNAP_PATH} | tee -a ${LOG_PATH}

# Uploads the snapshot to the file server
log_this "Uploading snapshot to Minio"
mc cp ${SNAP_PATH}/${SNAP_NAME} ${MINIO_ALIAS}/${MINIO_BUCKET}/${MINIO_PATH}/ &>>${LOG_PATH}

# Delets the local compressed snapshot
log_this "Removing old snapshot(s):"
cd ${SNAP_PATH}
rm -fv ${SNAP_NAME}

log_this "Done\n---------------------------\n"

akash-network / support