[DEVOPS] Setup charts on DO configured for flash, and document steps to reproduce the configuration

lnflash / charts

Flash fork of Galoy helm charts

MIT License

2 stars 1 forks source link

[DEVOPS] Setup charts on DO configured for flash, and document steps to reproduce the configuration #4

Closed islandbitcoin closed 4 months ago

islandbitcoin commented 1 year ago

I have successfully run the entire galoy stack on regtest using the charts repo., running k3d on my own hardware.

I have not been able to run it successfully on mainnet and I have not been able to set it up successfully on GCP, so this task is to set up a full k8s cluster with all pods running and healthy on GCP instance along with comprehensive documentation on how to replicate the setup.

PR should include detailed documentation on how the GCP setup was done, including all settings that need to be changed for a mainnet instance. I should be able to access the following with all authentication working:

GraphQL API running on api.staging.flashapp.me
WebSocket running on ws.staging.flashapp.me
PoSURL (Flash Pay) running on pay.staging.flashapp.me
Admin running on admin.staging.flashapp.me

Finally, A DevOps resource should be able to use the documentation to replicate the steps in order to completely set up a production environment for Flash.

islandbitcoin commented 8 months ago

Might try this on DigitalOcean again....

islandbitcoin commented 7 months ago

Currently modified by @brh28 and working on DigitalOcean. leaving this open so we complete the remaining tasks:

[x] flash-pay server config: https://github.com/lnflash/charts/issues/13
[ ] admin-panel
[x] Enable persistent storage for staging and production (mongo, kratos & price-history)
[ ] Convert all default passwords to K8s secrets
[x] Automate and schedule backups
[x] Enable and track Metrics

brh28 commented 7 months ago

Kratos persistence: https://github.com/lnflash/charts/pull/16

islandbitcoin commented 7 months ago

Redeployed Staging, lost all user data :-(
Enabling Logging and Telemetry

brh28 commented 6 months ago

Enable and track Metric

[x] Write Open Telemetry data to Honeycomb via collector: https://github.com/lnflash/charts/pull/19
[x] Set up Prometheus & Grafana

brh28 commented 5 months ago

Some updates.

Removed the prometheus & grafana instances to free PVs on our single node. We weren't using them anyway because we don't have BTC/LN infrastructure that requires monitoring (main Galoy use case) and we can use Digital Ocean tooling to monitor resource utilization. Marking these tasks as complete for now as there isn't work to be done at the moment. We can reintroduce with a new issue if there's specific functionality we need.
Admin-panel. Able to run the admin-panel and authenticate using Galoy's edge image, but getting a 401 from the graphql admin api both when running locally and in staging. Seem's there's a Kratos token the api is expecting, but not being passed by the admin-panel. This likely requires a deeper dive into the internals of the admin-panel to figure out. Leaving as an open issue to be returned to

brh28 commented 4 months ago

I'm closing this task as the scope is quite large and not well-defined. Consequently, it hasn't been maintained. I've moved some of the unchecked tasks into Issues of their own: