Backend service for plugin metadata

[bentsherman]

As the plugin ecosystem grows, it might be better to separate the plugins file into separate files for each plugin. This way Nextflow doesn't have to download the entire file just to get the metadata for a few plugins.

The problem is that older versions of Nextflow won't be able to pull plugins, or they will be stuck on the last version of the all-in-one file and won't be able to pull new plugins.

In any case, I don't think it will be an issue unless there are e.g. thousands of plugins.

[pditommaso]

I think we should start planning a backend-like system to manage plugins metadata, keep the plugins.json aligned with the new system for backward compatibility and deprecate it

[bentsherman]

And I bet you want to use surrealdb for it? 😄

[pditommaso]

now you said that, yes!

[drernie]

Is this a hackathon project? I'd be interested in helping, as it may make my merges simpler.

[bentsherman]

We will probably handle this internally since we'll need to host the service anyway.

[jagedn]

Hi Maybe by the moment, split the plugins in different json (one per plugin) and a github action to merge all of them when a pr is merged can be util

[pditommaso]

Hola Jorge, what would be the benefit?

[jagedn]

Final result will be the same, I mean, a full json with all plugins+releases but having a source json per plugin can facilitate the track and PRS as every author will edit their own json avoiding conflicts

[bentsherman]

We don't really run into conflicts between different PRs. Updating two different plugins at the same time wouldn't cause a merge conflict. I think splitting the JSON file would just increase the round-trip time. It's not an urgent issue so better to wait until we can build a full solution