search

conda-forge / conda-smithy

The tool for managing conda-forge feedstocks.
https://conda-forge.org/
BSD 3-Clause "New" or "Revised" License
147 stars 176 forks source link

Custom plugins #793

Open jakirkham opened 6 years ago

jakirkham commented 6 years ago

Have thought about this quite a bit, but it seems there is no open issue on this so opening this one.

Basically it would be nice if conda-smithy had some plugin infrastructure. Namely to provide the ability to write custom transformations on recipes. This has been done with conda-execute scripts in the past. However there is a lot of boilerplate that could easily be tucked away in conda-smithy and provided as some conda_smithy.exts module or similar for plugins to easily use. Further plugins could register for subcommands to allow easy application of them through the CLI. This could be useful in simplifying existing code in various bots, making it easy to write and perform custom migrations, etc.

Thoughts on this idea are welcome.

scopatz commented 6 years ago

Hi @jakirkham - I think we definitely need some formal infrastructure for handling migrations. It would be great if this could be shared across projects (ie migration code would be available to both conda-smithy and the bot). I think there are a few ways to accomplish this:

  1. conda-smithy and the bot each keep their own copy of migration code.
  2. All migration code lives in conda-smithy, which then the bot depends on.
  3. Migration code lives in a separate repo / project (conda-forge/migrations?), and then both conda-smithy and the bot may rely or depend on it.

I am in favor of (3) the most, because all of the migration code lives in one place AND it allows us to develop and deploy migrations quickly (that is, with out waiting for a package to be released).

In general though, I think this is a worthy goal!

jakirkham commented 6 years ago

Right, the thinking here is less about providing specific implementations for migrations. Instead it is about making it as trivial as possible for one to write migrations and leverage utilities already in conda-smithy to do it.

For example we could allow a user written function that takes a parsed meta.yaml and produces a modified meta.yaml. Then conda-smithy goes through the work of committing the changes and generating a PR. If they are common enough subclasses of these (e.g. changing script), we could expose an even simpler API for this.

IOW the goal is to make this as easy as possible for a random maintainer to pick this up and get a migration script working in a few minutes. Should add the hope would be these could be easily used locally, in the bot, or wherever else.

CJ-Wright commented 6 years ago

I'm not sure if we can operate on the parsed meta.yaml how would we handle selectors? It would be great to integrate this with the linter, since we are building filters which find "bad" syntax and fix it (thus we can point out bad syntax before it is merged).

scopatz commented 6 years ago

@jakirkham So I think between the bot and rever, we already do have a lot of the easy to use utilites for migrations. If you look at @CJ-Wright's JS migrator that ran last night (https://github.com/regro/cf-scripts/blob/master/conda_forge_tick/migrators.xsh#L224) it is pretty small as en end point for everything that it does.

Having some of the code that supports this live only in the bot does not make it super accessible. I would be happy if it happened to live in conda-smithy or somewhere else that is more of a library. We can figure out where migrations live later (though I do think a centralized place would be nice).

I'd be happy to have these tools move into conda-smithy, but I think the elephant in the room here is that the existing tools are written in xonsh, and I don't necessarily think they should be rewritten back to Python. The reason they are in xonsh in the first place is that there is a lot of interfacing with subprocesses (git, etc) and the environment, while also wanting the Python object model.

That is, users can either write their migrations in Python or xonsh. It is just that on the backend we wanted the code that touches repos, feedstocks, recipes, etc, to be able to easily run subprocesses.

I'd be thrilled if folks were open to having xonsh as a dependency of conda-smithy! I think it might even simplify other parts of conda-smithy.