cf_maintainer_bot

[pmlandwehr]

I've begun roughing out a script for helping folks do maintenance on their repos - automated version ticking, hash replacing, and rerendering. (It's definitely not a fully automated solution since it doesn't verify updated build requirements.) Its current home is at https://github.com/pmlandwehr/cf_maintainer_bot but I wouldn't mind moving it over to the main conda-forge organization if folks are interested in it.

[pmlandwehr]

(Renamed the bot package, since it isn't exactly a bot: https://github.com/pmlandwehr/tick-feedstocks)

[jakirkham]

Yeah, I'd be willing to work with you on integrating something like that at conda-forge, @pmlandwehr. Am desperately in need of some automation to maintain dependencies.

[pmlandwehr]

Ok - if you create a conda-forge repo called tick-feedstocks or something similar and make a master and develop branch, I can do a pull to put in all of my code to the develop branch. Running things via a REPL in stages works fine, but I've yet to test every part in one go. I've also not yet tested the command-line interface and am pretty sure the packaging will need some tweaks. (This is not to put those tasks on someone else; I want to just acknowledge the current status.)

[jakirkham]

Thanks for the info.

So we do have some helper scripts here. They are designed to be all in one and run with a tool called conda-execute. This is how most of the scripts operate in web services, but not all. Would be curious to hear your thoughts on this and whether that fits with your tool.

[pmlandwehr]

I think we could reformat my tool to run with conda-execute - tick-feedstocks basically runs a few scripts in serial:

• get the feedstocks a user maintains
• figure out which of those feedstocks needs updating
• figure out which of that subset don't depend on others in that subset.
• find/replace relevant hashes and keywords
• rerender and submit pull requests

My biggest concern is the rerendering, since I handle that through a separate shell script, but it looks like there's an extant script for dealing with this that avoids the shell. I'll see if I can steal the appropriate parts from it.

So yeah - I'll try to refactor for conda-execute and submit a pull.

[jakirkham]

Ok, so this is sounding pretty reasonable then. Might suggest that those steps become functions in the script. Probably will be easier to get to work that way.

As to the re-rendering bit, I think what we should do is try to isolate a usable API. One can kind of get at it by calling conda_smithy.configure_feedstock.main(...), but I think we can do better than that. Though feel free to use it for now if it helps.

Cool, looking forward to it. Please let me know if there is anything I can do to help.

[pmlandwehr]

Current pull request here: https://github.com/conda-forge/conda-forge.github.io/pull/375 Regenerating likely has some kinks, but the rest should be fine - if folks want to test locally.

[jakirkham]

Awesome, thanks for doing that @pmlandwehr. Will take a closer look soon.

[epruesse]

Will this do something like Debian's uscan?

I keep thinking that it would be useful to have a bot that monitors upstream releases and creates simple PRs that update version & hash. Debian's solution uses a regex type description in debian/watch, so as part of their "recipe", that is then utilized by the bot to keep things up to date. For PyPi, R-cran and github packages, the URLs should be simple enough to recognize. It could run in a travis cron job, too.

[pmlandwehr]

My only concern with using a bot to submit the pull requests is that it's quite possible for a version update to introduce new dependencies. As currently written, the script isn't yet sophisticated enough to attack this problem; it only ticks version numbers and replaces hashes. I'd love for it to become this sophisticated, but it isn't there yet.

Because of the minimal nature of the change, I think that it's important that a particular user maintain ownership of the submitted pull. I think that an entirely appropriate use for the script would be for a conda-forge maintainer to run it as a cron job periodically, either on their local machine or off in the cloud. That would make the maintainer directly accountable for getting each pull merged.

[epruesse]

Or that. It would be a convenience thing anyway. Just a PR for the maintainer to get started. That the package will have to be reviewed is clear. New dependencies should just lead to a fail anyway, though removed dependencies and other changes can lead to plenty of things happening that would be hard to automatically catch.

[jakirkham]

...it's quite possible for a version update to introduce new dependencies.

TBH I think we should be able to solve this problem without too much effort. conda skeleton pypi has been able to solve this for a long time by simply patching out the setup function so that it can capture all metadata that a project provides when running setup.py. We should be able to either borrow from what they have or come up with something similar.

[pmlandwehr]

I def. think conda skeleton pypi is a great starting point, but even those recipes do require some double-checking.

[CJ-Wright]

I think that even just starting a PR would be good, since I don't keep up to date with all the version changes in things that I maintain. This would serve as a good notice to let me know that a new version has come out. Discussing this with @jakirkham it may be nice to have this run independently of any single cron job (potentially a webservice?). I'm less familiar with how the webservices work in general. Are they always running or do they just react? If they are reactive then we may need to use something else as I'd imagine that this would be a constant/regularly scheduled thing.

Does anyone have an estimate of how long it would take to run tick-my-feedstocks on every feedstock?