implement a command to update the version of renku-python in interactive environments

[rokroskar]

After https://github.com/SwissDataScienceCenter/renku-project-template/pull/86 is merged, it should be straightforward to update the renku-python version installed in users' docker images. We should have a command that checks if the current version needs to be updated and update it if that is the case. In the case of released versions this is trivial, it's not so obvious what to do when renku migrate is called from a development version, since the corresponding development release may not exist on PyPI.

[Panaetius]

I'm not sure this should be a part of regular renku migrate, which focuses on migrating the metadata and consists of migrations that are done once and then not repeated again. Since this increases the project version, but a new renku-python release does not necessarily need migrations or change the project version, renku migrate does not seem too suitable for this task. Unless we write a migration for this for each renku release, which I personally don't like too much.

Maybe a different, dedicated command to update the Dockerfile might make more sense.

[rokroskar]

we've talked about having some sub-commands for handling the interactive environments - maybe this could fall under that use-case? e.g.

renku interactive launch

would spin up the jupyter notebook server

and

renku interactive update

would do the Dockerfile change.

[Panaetius]

Timeboxed to 2 for design

[rokroskar]

@Panaetius I changed the title and description to reflect your proposal

[Panaetius]

Discussed in https://sdsc.atlassian.net/wiki/spaces/RENKU/pages/680067073/2020-08-06+Design+Meeting+notes design meeting

[ciyer]

It seems the scope of this issue became a little larger and includes the possibility to update the a new version of the template as well as a newer version of renku. For the UI it makes sense to have some more information from the service.

cache.migrations_check This currently just returns migration_required: bool, but we would need to get:

• renku_python_migration_required: bool
• template_migration_available: bool
• automated_renku_python_migration: bool
• automated_template_migration: bool
• new_template_url: string (URL)

This will allow us to communicate to the user what the migration will do and what to do if an automatic migration is not possible.

cache.migrate This currently returns migration_status: string , migration_error: string, but we also need:

• renku_python_migration_status: string
• renku_python_migration_error: string
• template_migration_status: string
• template_migration_error: string

The operation should be all or nothing (and reflected in migration_status), but we will want to tell the user what worked, what didn't work, and what they need to do to migrate the project.

See https://github.com/SwissDataScienceCenter/renku-ui/issues/1026 for UI perspective.

[Panaetius]

Templates have two new fields:

• __automated_update__: Whether or not the template allows automated updating.
• __immutable_template_files__: Which files a user should not edit, as they are crucial for the template to work.

The template version used is stored in the project metadata. Additionally, checksums for all templated files are saved to check for changes. The original metadata used for instantiating from the template is also stored.

cache.migrations_check will return:

• migration_required: as before, if there are migrations needed to get from the projects renku version to the core services renku version.
• template_update_possible: Whether the template can be updated (all necessary information is there for an update and there is a new version available)
• automated_template_update: If automated_template_update is set on the project (Defaults to False)
• docker_update_possible: Whether the renku-version in the dockerfile can be updated (The Dockerfile contains the relevant renku version string and it is lower than the version of the core service.) This will always return False if the current version is a dev version.

An important point is, docker_update_possible=False and migration_required=True means the project is out of date with the current renku-version, but the Dockerfile does not support updating the renku-version (e.g. if it's a custom Dockerfile that does not contain the renku version string). In this case, we shouldn't execute migrations, as it'd bring the repo into a state that doesn't work. so docker_update_possible supersedes migration_required.

cache.migrate has new parameters:

• force_template_update: set to true to update the template even if automated_template_update is not set on the template (probably not a good idea...)
• skip_template_update: Don't try to update the template (superseedes force_template_update).
• skip_docker_update: Don't try to update the Dockerfile.
• skip_migrations: Don't execute migrations.

All of the above default to False

So skip_template_update=True && skip_docker_update=True && skip_migrations=False would give the same behavior as the current endpoint.

Exceptions on this endpoint will look like:

{
    "code": -32100,
    "reason": "...error string...",
    "template_update_failed": False,
    "dockerfile_update_failed": True,
    "migrations_failed": False
}

So the 3 flags determine which part it was that failed (only one of them is True on error). This differs a bit from what was described above, but it's in line with how we dealt with errors before and should still cover everything that's needed.

I'm not sure what new_template_url: string (URL) mentioned above should contain? The template URL stays the same through all this.

[ciyer]

On the UI, we would like to

1. Let them know what has changed in the template
2. inform the user how they can update the template manually if it is necessary.

The idea is that new_template_url would provide documentation of the new features and instructions for manually upgrading.

[rokroskar]

@ciyer who should provide this? For our templates it's clear, but what about templates created by others? Should it just be a changelog in the template repo?

[ciyer]

I guess we cannot require a changelog, but we should encourage it. At least we can show a diff between the two template versions.

[rokroskar]

I think if we had a clearly-communicated convention for the changelog that would be fine. The diff could very well be incomprehensible to a normal user.

[Panaetius]

We couldn't use a CHANGELOG.md in the template as that would just be a template file, I think.

So the only place right now that we could keep that information is in the template manifest. There's no versioning of templates, though they are somewhat implicitly version by git commit at the moment. So we might be able to have a changelog in the manifest and return a diff of the changelog entry?

But I'd rather do this in a separate issue, with proper design, not duct-tape it onto this one.

[rokroskar]

Sure there's a place for the changelog, same place as the README and the manifest file (as in https://github.com/SwissDataScienceCenter/renku-project-template)

[Panaetius]

But that'd be a changelog for all templates? We'd want one per template, probably

[rokroskar]

Personally, I would be ok if I received information like

Python template bumps jupyter version to x.x.x; R images now include an improved RStudio server proxy

etc.

[Panaetius]

Having it in a machine readable format (yaml?) would probably still be a good idea.

[rokroskar]

Maybe eventually - then we'd need to sort out the formatting (which would be different than a typical changelog) and expect that people conform to it (and what do we do when they don't). It would be much simpler to just show the relevant section of the changelog in plaintext, at least as a starting point.