Non-MCC Collections on Kubes Cron

[kespinola]

Goal

Non MCC collections are recomputed every hour.

Requirements

• Add non-mcc collection sql generator script into config map @imabdulbasit
• Run a container with psql installed and access to writer db using kube cron
• Manage the cron as a argo app so have UI to explore the status of the cron

Environments

• [x] Staging
• [ ] Production

[mpwsh]

Got queries from @imabdulbasit.

Cron job for Collections volume

• 1d _volume migration : every 5mins
• 7d_volume_migration : every 2 hours
• 30d_volume migration : every 12 hours

Cron job for ME Collections.

• It involves very expensive queries so maybe every 3-4 hours: updating me_collections & me_metadata_collections migration

• every 6/12 hours: collection_stats migration

Starting to work on the cronjobs.

[mpwsh]

Cronjobs added to kubes repo (dev) after merging #744 #745

All cronjob manifests can be found here alongside the script that downloads the migration and executes them with psql

Will spin up the same cronjob manifests after merging main <- dev

To create new migration cronjob, you can duplicate any of the existing ones in the indexer/app/jobs folder and change its name and the following env vars.

env:
- name: REPO_URL
  value: https://github.com/holaplex/indexer
- name: GIT_BRANCH
  value: dev
- name: REPO_PATH
  value: crates/core/migrations/2022-09-12-154321_update_1d_collection_volume/up.sql

After pushing the new file to the kubes repo, argoCD will take care of deploying it

[kespinola]

done. ty @mpwsh

[kespinola]

done.