Detailed upgrade instructions for metadata.yaml -> datasette.yaml

[simonw]

Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.

I think we should link directly to documentation that tells people how to perform this upgrade.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/2190#issuecomment-1759947021

[simonw]

Some options for where this could go:

• Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides.
• On a new page? We could have a "upgrade instructions" page in the documentation.
• At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

I'm leaning towards the third option at the moment.

But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.

[simonw]

Related idea: how about a datasette-upgrade plugin which adds a datasette upgrade command that can be used to automate this process?

Maybe something like this:

datasette install datasette-upgrade
datasette upgrade metadata-to-config metadata.json

This would output two new files: metadata.yaml and datasette.yaml. If files with those names existed already in the current directory they would be called metadata-new.yaml and datasette-new.yaml.

The command would tell you what it did:

Your metadata.json file has been rewritten as two files:

    metadata-new.yaml
    datasette.yaml

Start Datasette like this to try them out:

    datasette -m metadata-new.yaml -c datasette.yaml

The command is datasette upgrade metadata-to-config because metadata-to-config is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.

[asg017]

I dig it - I was thinking an Observable notebook where you paste your metadata.json/metadata.yaml and it would generate the new metadata + datasette.yaml files, but an extensible datasette upgrade plugin would be nice for future plugins.

One thing to think about: If someone has comments in their original metadata.yaml, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting

[simonw]

I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

If people want to keep their comments I'm happy to leave them to copy those over by hand.

[simonw]

Started playing with this plugin idea, now tearing myself away to work on something more important:

from datasette import hookimpl
import click
import pathlib

@hookimpl
def register_commands(cli):
    @cli.group()
    def upgrade():
        """
        Apply configuration upgrades to an existing Datasette instance
        """
        pass

    @upgrade.command()
    @click.argument(
        "metadata", type=click.Path(exists=True)
    )
    @click.option(
        "new_metadata", "-m", "--new-metadata", help="Path to new metadata.yaml file", type=click.Path(exists=False)
    )
    @click.option(
        "new_datasette", "-c", "--new-datasette", help="Path to new datasette.yaml file", type=click.Path(exists=False)
    )
    @click.option(
        "output_dir", "-e", "--output-dir", help="Directory to write new files to", type=click.Path(), default="."
    )
    def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
        """
        Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
        datasette.yaml split introduced prior to Datasette 1.0.
        """
        print("Upgrading {} to new metadata.yaml format".format(metadata))
        output_dir = pathlib.Path(output_dir)
        if not new_metadata:
            # Pick a filename for the new metadata.yaml file that does not yet exist
            new_metadata = pick_filename("metadata", output_dir)
        if not new_datasette:
            new_datasette = pick_filename("datasette", output_dir)
        print("New metadata.yaml file will be written to {}".format(new_metadata))
        print("New datasette.yaml file will be written to {}".format(new_datasette))

def pick_filename(base, output_dir):
    options = ["{}.yaml".format(base), "{}-new.yaml".format(base)]
    i = 0
    while True:
        option = options.pop(0)
        option_path = output_dir / option
        if not option_path.exists():
            return option_path
        # If we ran out
        if not options:
            i += 1
            options = ["{}-new-{}.yaml".format(base, i)]

[simonw]

Demo of that logic:

$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-1.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch metadata-new-1.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch datasette.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette-new.yaml

[simonw]

Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade