datasette.io is currently unavailable or shows the wrong site

[simonw]

Initially I saw datasette.io showing page from datasette.simonwillison.net instead

See also #140.

[simonw]

https://github.com/simonw/simonwillisonblog-backup/blob/main/.github/workflows/backup.yml should be publishing to a separate service called simonwillisonblog - I did however recently make this change to it:

https://github.com/simonw/simonwillisonblog-backup/commit/bfb573e96d8622ab52b22fdcd54724fe6e59fd24

[simonw]

Now I'm just getting 503 service unavailable errors: https://datasette.io/

[simonw]

https://github.com/simonw/datasette.io/actions last deployed 15 hours ago. I'm going to trigger a manual deploy.

[simonw]

That last run 15 hours ago is here: https://github.com/simonw/datasette.io/actions/runs/4361174962/jobs/7624818986

It finished with this output:

Deploying container to Cloud Run service [datasette-io] in project [datasette-222320] region [us-central1]
Deploying...
Setting IAM Policy...............done
Creating Revision.............................................................................................................................................................................................................................................................done
Routing traffic.............................................done
Done.
Service [datasette-io] revision [datasette-io-01211-fol] has been deployed and is serving 100 percent of traffic.
Service URL: https://datasette-io-j7hipcg4aq-uc.a.run.app/

https://datasette-io-j7hipcg4aq-uc.a.run.app/ is showing the wrong site! It's showing the same as https://datasette.simonwillison.net/

[simonw]

That manual deploy failed because it tried to fetch the previous database and could not: https://github.com/simonw/datasette.io/actions/runs/4368015559/jobs/7640059802

[simonw]

I'm attempting to run this site's deploy script like so:

Running here: https://github.com/simonw/datasette.io/actions/runs/4368028294/jobs/7640089932

[simonw]

This is the last deploy from simonw/simonwillisonblog-backup which went out - my hunch is that this is the thing that broke the main datasette.io site: https://github.com/simonw/simonwillisonblog-backup/actions/runs/4361175893/jobs/7624826610

It ends with:

Successfully built e29bbc21e72b
Successfully tagged gcr.io/datasette-222320/datasette:latest
PUSH
Pushing gcr.io/datasette-222320/datasette
The push refers to repository [gcr.io/datasette-222320/datasette]
...
latest: digest: sha256:81866c41a6b25b0e9e0c4ed67089af62543969fa8b09e40f154048df6ce9be07 size: 2007
DONE
--------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                                          IMAGES                                       STATUS
33d476b9-0267-43ab-b97d-e5b23700fdf9  2023-03-08T04:35:16+00:00  1M7S      gs://datasette-222320_cloudbuild/source/1678250109.802925-d0d284ddb36742088faf7bb9ae360c96.tgz  gcr.io/datasette-222320/datasette (+1 more)  SUCCESS
Deploying container to Cloud Run service [simonwillisonblog] in project [datasette-222320] region [us-central1]
Deploying...
Setting IAM Policy...............done
Creating Revision...............................................................................................................................................................................................................................................done
Routing traffic.......................................................done
Done.
Service [simonwillisonblog] revision [simonwillisonblog-00331-tiy] has been deployed and is serving 100 percent of traffic.
Service URL: https://simonwillisonblog-j7hipcg4aq-uc.a.run.app/

I'm worried about "Successfully tagged gcr.io/datasette-222320/datasette:latest" - could that be over-riding the same entry in the registry that's used for the datasette.io site usually?

[simonw]

I disabled the scheduled deploys on simonw/simonwillisonblog-backup while I continue to investigate.

[simonw]

My hunch as to what happened here is that it's a timing bug: I think both of those repos have been using the same name for their image in the registry, which has mostly worked (because they build and push the image and then run a deploy of it) - but here there was a timing clash and both deploys ran simultaneously, resulting in the wrong image going out to datasette.io.

[simonw]

https://github.com/simonw/datasette.io/actions/runs/4368028294/jobs/7640089932 was my attempt at a manual build-from-scratch for datasette.io but it failed with this error:

  scripts/build.sh
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.16/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.16/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.16/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.16/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.16/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.16/x64/lib
    GITHUB_TOKEN: ***
[{"rows_affected": -1}]
[{"rows_affected": -1}]
Error: Table "plugin_repos" does not exist

My guess is that the build-from-scratch script hasn't been updated in a while to reflect recent changes.

[simonw]

I'm going to do a manual deploy from my own laptop to fix the problem and restore the old content.db file.

[simonw]

I'm running this script on my Mac (after reinstalling gcloud and logging in): https://github.com/simonw/datasette.io/blob/main/scripts/deploy.sh

[simonw]

From my local logs:

Successfully built b2b2117ad276
Successfully tagged gcr.io/datasette-222320/datasette:latest

I'm really suspicious of that tagging.

[simonw]

Manual deploy worked - a slightly out-of-date version of https://datasette.io/ is now live.

[simonw]

I'm running a GitHub Actions deploy now to see if that updates to the most recent data.

Before that deploy the site looks like this:

[simonw]

Here's what it looked like after the manual deploy. I think this issue is fixed now.

[simonw]

This highlighted a bug in Datasette itself which I'll address here:

• https://github.com/simonw/datasette/issues/2036

[simonw]

I shipped Datasette 0.64.2 with the fix for that bug in datasette publish cloudrun: https://docs.datasette.io/en/stable/changelog.html#v0-64-2