Ship datasette metrics from fly.io using `fly-log-shipper`

[e-belfer]

Overview

fly.io currently doesn't retain logs for a long time so we need to use the fly log shipper to send logs to S3.

We should spend at most 10 hours on this.

Success Criteria

How will we know that we're done?

• [ ] We have Datasette log metrics regularly archiving to a bucket in S3

We don't need to mirror the logs into GCS or ETL them into some structured format because we are likely to deprecate the Datasette shortly anyways. This lets us do some baseline analysis without too much investment.

### Next steps
- [ ] create new AWS project to plug fly logs into - our existing one is Open Data only
- [ ] create an IAM role + access key pair so that we can plug that into the log shipper configuration

[jdangerx]

Now that Cloud Run lets you mount buckets as volumes, it's tempting to migrate back to Cloud Run. That would look like:

• build a Dockerfile that runs & mounts everything correctly locally.
  • we have a dockerfile in devtools/datasette/publish.py. it includes the nginx stuff too so that's good
  • strip out the decompression, zstd, etc. stuff
  • strip out stuff from run.sh too, I think it just needs to run datasette serve + use nginx.conf
  • run in immutable mode, do not inspect.
  • hard-code the databases into run.sh.
  • the Dockerfile should set some ports, install nginx + datasette, add nginx confs, and run run.sh
  • actually, let's try stripping out nginx too. can always put it back.
  • point datasette service at /data/nightly
• use docker compose to do a dev version of this. bind-mount $PUDL_OUTPUT at /data/nightly
• build the image locally, then push it to GHCR to test out the rest of this
• terraform: add a service that runs a datasette container
  • point at the image from ghcr
  • mount pudl.catalyst.coop as a read-only volume at /data
  • terraform apply
• this should just have a service-name-adsfasdfasdfa.run.app URL so we can check that stuff works:
  • can access the dang data
  • logs show up in GCP console & have the right IP
• remove fly.io-specific logic in nightly build, except for "restart the cloud run service"
  • remove the publish.py line in gcp_pudl_etl.sh
  • gcloud run deploy SERVICE --image IMAGE_URL
• build the image in GHA, and push to GHCR/pudl-datasette:latest - on push to main.

And to get logs:

• raw: pull logs straight from cloud logging bucket via gcloud.logging
  • do we need to do the gcloud auth stuff? i think the gha runner has an SA associated with it, just need to make sure it has the right roles
• core: parse those logs
• out: do we necessarily need to do more, really?

And if we were to stick with fly.io, the plan looks like

• keep the deployment the way it is
• push logs into S3 via fly-log-shipper
  • make a new fly app, next to the datasette one I guess
  • make an S3 bucket
  • make an AWS access key for an SA that has access to the bucket
  • run fly-log-shipper
• raw: pull logs from s3 into tables, via boto3.
• core: parse logs
• out: do we necessarily need to do more, really?

Which looks faster - but also, I'm suspicious that fly-log-shipper doesn't actually do what it purports to, or isn't as easy as it claims to be, since it doesn't look super actively maintained.

And, the final outcome of moving back into GCP is simpler & easier to take down if we decide to move to Superset.

So, I'm going to spend 30 minutes tomorrow trying to get fly-log-shipper working. If I run into problems I'll move back into GCP.