search

bioconda / bioconda-recipes

Conda recipes for the bioconda channel.
https://bioconda.github.io
MIT License
1.6k stars 3.13k forks source link

R package migration to conda-forge #5582

Closed daler closed 6 years ago

daler commented 6 years ago

In preparation to supporting R 3.4, we are migrating any non-bio-related packages over to the conda-forge channel. This issue aims to organize and track the progress.

Below are the results of:

bioconda-utils dag recipes/ config.yml --format txt --packages "r-*" \
   | sed "/recipes/ s|recipes/|- [ ] |"

Workflow

The list is divided into independent subdags. Each subdag is topologically sorted such that packages at the top are dependencies of packages at the bottom. Within each subdag, recipes should be built in order.

subdag 0

subdag 1

subdag 2

subdag 3

subdag 4

subdag 5

subdag 6

subdag 7

subdag 8

subdag 9

subdag 10

subdag 11

subdag 12

subdag 13

subdag 14

subdag 15

subdag 16

subdag 17

subdag 18

subdag 19

subdag 20

subdag 21

subdag 22

subdag 23

subdag 24

subdag 25

subdag 26

singletons

jdblischak commented 6 years ago

I marked the remaining packages that are already available on conda-forge. The version string may be slightly different because I converted them to numeric version strings in order to identify the most recent version on conda-forge (using R's as.numeric_version()).

I can help with this effort. There are a handful of packages in this list that I was hoping to get into conda-forge. Would it be useful if I submitted a PR with a batch of the singletons? I could start from the bottom of the list to avoid duplicating effort.

daler commented 6 years ago

@jdblischak thanks for going through these; starting at the bottom of singletons would be fantastic. You could also work on any subdag that is not already being worked on, since they are independent from each other. That might be a better option for you if you have particular packages you want sooner rather than later.

bgruening commented 6 years ago

@daler @jdblischak we were able to merge the first bigger batch yesterday. It's still in the CI queue to get coverted to a feedstock but we have some progress :) Thanks for helping so much!

daler commented 6 years ago

@bgruening great! Can you remind me: once the feedstock is created, it has to be tested again before getting uploaded to the channel?

bgruening commented 6 years ago

@daler yes. The feedstock is created by the staged-recipes PR and than a normal run is triggered on the master branch of the feedstock which created the package. I needed to remove 2 recipes from my PR due to build-time outs. So I need to get these in before we can work on your follow-up one.

daler commented 6 years ago

Got it. Thanks!

daler commented 6 years ago

@bgruening @jdblischak I'm going through all packages to decide on bio-relatedness; please hold off on editing the original issue above so we don't lose any information. I'll report back here as soon as I'm done.

daler commented 6 years ago

@bgruening @jdblischak OK, I think I got everything. Should be more straightforward to create conda-forge PRs now.

souravsingh commented 6 years ago

@daler I can work on the rest of the R packages.

daler commented 6 years ago

@souravsingh that would be fantastic. I'm a bit crunched for time through the next two weeks.

bgruening commented 6 years ago

@souravsingh awesome! I have a big one pending and will update the list accordingly. Currently we are blocked by some CI shortage :(

bgruening commented 6 years ago

@souravsingh ok I got my next two pushed, if you would like to take over for today :)

jdblischak commented 6 years ago

There are currently 6 PRs submitted to conda-forge that have failed CI builds. I've reviewed some of them, but I know many are failing due to missing dependencies. We've merged lots of recipes recently. Should I restart the CI builds on the PRs? Is there any PR in particular that I should investigate? Please let me know if there is anything I can do to help.

bgruening commented 6 years ago

@jdblischak I'm on it. Part of the problem is that we have a few circular dependencies which I try to resolve and that I also need to restart several feedstock repos. This just takes a lot of time. Sorry.

bgruening commented 6 years ago

I marked a few more as resolved as I could merge a few RPs in the last day. It does not look that bad, we are getting there :)

bgruening commented 6 years ago

Ok, I added a bunch more and restarted a few feedstocks to keep the CIs busy over the weekend.

bgruening commented 6 years ago

@daler @jdblischak I spend the last days and pushed it over the finish line. I think we have most of the packages, hopefully all. One last PR is open and this will be merged in the next hours.

So we did it! I also removed a lot of packages from bioconda in the bulk branch. I must admit that I lost a little bit track, but I did my best and hopefully this all looks fine for you.

jdblischak commented 6 years ago

So we did it!

Awesome! Great work, @bgruening, on making the final push!

daler commented 6 years ago

@bgruening this is so awesome.

I can't imagine the time you've spent over the past weeks babysitting and restarting CI builds across multiple providers and keeping track of which failed due to dependency feedstocks not being complete . . .

Phew. Amazing work!

bgruening commented 6 years ago

As usual there are still things to work on, but I think we archived our goals here and with https://github.com/bioconda/bioconda-recipes/pull/6119 I'm going to close this one. Thanks all for your help.