Add support for custom migrations

[czosel]

Currently, it's not possible to register custom migrations in caluma. There are some possibly valid use-cases for them though, e.g.:

• form answers should be transformed in some way to fit a new structure, instead of creating a new form for them
• run maintenance tasks (clean up specific records)
• fix errors that were introduced when moving forms from another system over to caluma

Not sure how this could look on a technical level. Thanks for your feedback :slightly_smiling_face: /cc @sliverc @open-dynaMIX @winged

[winged]

The obvious path would be to allow a custom Django app in Caluma, which could contain it's own migrations. But this would open doors for people to install their own models and (potentially) create a mess.

Next would be to implement a custom "data migration" mechanism that works kind of the same as Django's migrations - but we'd have to re-implement a good bit of the same logic again...

Another option would be to allow some authorized GraphQL scripting (directly executing against the schema, instead of the actual API), but again, we'd have to implement the management infrastructure for this as well.

Thinking some more about it, the first option would probably be the best option. We could implement some startup hook to ensure the "custom" app doesn't contain any models, and provide some helpers for common operations (like rename a form, rename a question, reorganize a form, etc.; implement as needed)

[sliverc]

It sounds obvious to do this in Django but should this really be done within Caluma or not better in the deployment process?

We are not really talking about schema changes but data. In the deployment process a job could be registered and depending on the taste someone could use:

• plain sql scripts
• scripts calling api directly with a privileged user
• or use a migration tool of their choice (e.g. https://flywaydb.org/)

I would prefer this than allowing any django apps to be integrated into the Caluma container.

[czosel]

This sounds very reasonable, following the general guideline to make our APIs "as powerful as needed, but not more". Let's take a look which path we want to take for our project - if it works well, it could become part of the caluma docs, and maybe we could even offer some handy utilities that help with data migrations.

[winged]

I agree with both of you. The only reason I'd opt for Django-based migrations is that it already provides a really nice infrastructure for what we need (versioning etc), and I don't think having two systems for doing the same task is a good idea.

I wouldn't allow users to inject a full app, but only the "migrations" folder of a custom app (Of course you can always abuse the container and mount whatever you want into it).

But maybe we can also build a data migration tool on top of #507, which may be closer to what the "users" should see anyway

[sliverc]

If it is not a full fledged django app then django-admin won't work and creating migrations might be come tedious and a lot of copy paste that's one of the reason why I don't see it as ideal.

Executing graphql scripts might be a solution, problem comes when one migration has to reference the result of another migration (e.g. primary key) which is currently not supported in GraphQL but discussed (see https://github.com/graphql/graphql-spec/issues/377).

[winged]

No it needs to be a full app, I agree, but the users should not be allowed to add models to it - just migrations