search

conda-forge / staged-recipes

A place to submit conda recipes before they become fully fledged conda-forge feedstocks
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
710 stars 4.96k forks source link

Porting R channel to conda-forge #2009

Closed johanneskoester closed 7 years ago

johanneskoester commented 7 years ago

This issue will track the effort to port the R channel into conda-forge. Please feel free to join if you like.

How to contribute

We can now start migrating R packages from conda/conda-recipes.

Prerequisites

R packages to migrate (topological sort levels such that all r-package depencencies are satisfied)

Level 1

Cleanup script:

In order to clean up recipes created with conda skeleton cran, I apply the following script:

#!/bin/sh

RECIPE=$1/meta.yaml
BUILDSH=$1/build.sh
BUILDBAT=$1/bld.bat

sed -i -e '/^ *#.*$/d' $RECIPE
sed -i -e '/^$/N;/^\n$/D' $RECIPE

echo extra: >> $RECIPE
echo "  recipe-maintainers:" >> $RECIPE
echo "    - johanneskoester" >> $RECIPE
echo "    - bgruening" >> $RECIPE

echo '"%R%" CMD INSTALL --build .' > $BUILDBAT
echo 'if errorlevel 1 exit 1' >> $BUILDBAT

echo '#!/bin/bash' > $BUILDSH
echo '' >> $BUILDSH
echo '$R CMD INSTALL --build .' >> $BUILDSH
asmeurer commented 7 years ago

Wow, this is a great effort. I hope you can do it. I did all the original R packaging. I don't have the resources nowadays to help out with this, but if you have a question you think I might know the answer to feel free to ping me.

johanneskoester commented 7 years ago

Thanks a lot @asmeurer!

johanneskoester commented 7 years ago

@conda-forge/core could you please direct people that provide PRs for R packages to this issue? I'd like to keep track of what is already done.

jakirkham commented 7 years ago

Does the list include things like r-oce ( https://github.com/conda-forge/staged-recipes/pull/2461 ), r-mass ( https://github.com/conda-forge/staged-recipes/pull/2666 ), r-cluster ( https://github.com/conda-forge/staged-recipes/pull/2666 ), r-class ( https://github.com/conda-forge/staged-recipes/pull/2666 ), r-codetools ( https://github.com/conda-forge/staged-recipes/pull/2668 ), r-foreign ( https://github.com/conda-forge/staged-recipes/pull/2668 ), r-kernsmooth ( https://github.com/conda-forge/staged-recipes/pull/2668 ), r-matrix ( https://github.com/conda-forge/staged-recipes/pull/2668 ), r-mgcv ( https://github.com/conda-forge/staged-recipes/pull/2672 ), r-nlme ( https://github.com/conda-forge/staged-recipes/pull/2672 ), r-nnet ( https://github.com/conda-forge/staged-recipes/pull/2672 ), r-rpart ( https://github.com/conda-forge/staged-recipes/pull/2672 ), r-spatial ( https://github.com/conda-forge/staged-recipes/pull/2672 ), r-crayon ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-digest ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-jsonlite ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-lattice ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-magrittr ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-r6 ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-repr ( https://github.com/conda-forge/staged-recipes/pull/2460 ), r-stringi ( https://github.com/conda-forge/staged-recipes/pull/2460 ), or r-uuid ( https://github.com/conda-forge/staged-recipes/pull/2460 )?

johanneskoester commented 7 years ago

The list above only includes packages that are not done yet. Hence, these are not contained.

jakirkham commented 7 years ago

Cool, wanted to make sure that (a) they were listed here somehow and (b) check with you that they had been noted somehow.

jakirkham commented 7 years ago

Also PR ( https://github.com/conda-forge/staged-recipes/pull/1597 ) has a bunch of incomplete recipes that are up for grabs if anyone wants to see them through.

jrderuiter commented 7 years ago

Who should we reference as recipe maintainers? I'm happy to help port a number of packages (https://github.com/conda-forge/staged-recipes/pull/2772), but do not necessarily want to be responsible for maintaining them.

johanneskoester commented 7 years ago

@johanneskoester and @bgruening

bgruening commented 7 years ago

Thanks @jrderuiter!

jrderuiter commented 7 years ago

Just wondering, what is the rationale for skipping win32? (Not that I miss it, just curious.)

johanneskoester commented 7 years ago

We don't have an openjdk build for win32. It is not offered by Zulu (the provider) and we don't build it ourselves so far.

jrderuiter commented 7 years ago

r-nycflights13 should be moved to level 3, as it depends on r-tibble.

johanneskoester commented 7 years ago

You are right. This dependency has apparently changes since 0.2.0, on which my dependency tree was based. Let's hope this is the only case.

EDIT: fixed.

jakirkham commented 7 years ago

Updated the level 1 listing after seeing some feedstocks get added this morning. Appears most sailed through. Also looks like that part of the list is nearly completed. 🎉

However, r-rzmq appeared to have some issues with CI registration. Am trying to readd the existing recipes to staged-recipes to see if that fixes the issue. ( https://github.com/conda-forge/staged-recipes/pull/2889 )

jakirkham commented 7 years ago

Yep, re-feedstocking r-rzmq through staged-recipes seems to have done the trick. All CIs ran and passed. Packages for all supported platforms uploaded. Checked the box in the list above.

bgruening commented 7 years ago

Level 2 should be finished if the above PR gets merged, currently blocking a few failing feedstock creations. xref: https://github.com/conda-forge/staged-recipes/pull/2897

johanneskoester commented 7 years ago

You mean level 1?

bgruening commented 7 years ago

No level 2: #2890 #2891 #2892 #2893 #2895 #2896 #2898

jrderuiter commented 7 years ago

Great! :) I'll try to help with levels 3 and 4 if I have some free time this week.

johanneskoester commented 7 years ago

Oh cool!! Haven't seen that. I guess it is not possible for somebody else than me to add the PR numbers to the list at the top, right? I'll update it. Maybe mention me once you add a new PR, so that we don't duplicate work.

bgruening commented 7 years ago

@johanneskoester will do this. But I think we need to wait for now to get the failing feedstock from Level 1 in. Enjoy your weekend!

bgruening commented 7 years ago

I have created some hacks to create these recipes. Just put the package name in packages.txt in the form of r-foobar and run the ./fix.sh script. You will end up with a bunch of r-* folders. Please check them manually. Especially the LICENSE section. This sections should not contain the word license.

Hope this helps a little bit.

build_helpers.tar.gz

Update: Latest version is here: https://github.com/bgruening/conda_r_skeleton_helper

isuruf commented 7 years ago

Here are some of the packages that have failed.

Please restart the CI for these repos. Most of them are network failures or package missing failures due to race conditions.

@jakirkham, I found this by reverting https://github.com/conda-forge/conda-forge.github.io/commit/7ede26729a882f32641a543dfa18022d3cd219bc and generating locally. Can we restore CI badges in a separate webpage so that we can see what packages are failing? This would be especially useful for @johanneskoester and @bgruening to maintain the huge number of r packages.

bgruening commented 7 years ago
bgruening commented 7 years ago

Besides https://travis-ci.org/conda-forge/r-readxl-feedstock and https://travis-ci.org/conda-forge/r-cellranger-feedstock everything should be fine now. Can someone please restart these, I can't for unknown reason.

bgruening commented 7 years ago

Work on Level3 started in https://github.com/conda-forge/staged-recipes/pull/2906 and https://github.com/conda-forge/staged-recipes/pull/2907 and https://github.com/conda-forge/staged-recipes/pull/2908

lpantano commented 7 years ago

https://github.com/conda-forge/staged-recipes/pull/2909 r-rsconnect r-rsqlite r-rversions r-sandwich r-scales r-seacarb r-selectr

bgruening commented 7 years ago

@lpantano cool thanks! Here is a new batch: https://github.com/conda-forge/staged-recipes/pull/2908

bgruening commented 7 years ago

I will do the last ones:

r-tilegramsr r-tseries r-urca r-weatherdata r-xlsx r-xts

edit: https://github.com/conda-forge/staged-recipes/pull/2910

bgruening commented 7 years ago

I have pushed the latest version of my cleanup-scripts to https://github.com/bgruening/conda_r_skeleton_helper.

lpantano commented 7 years ago

r-commonmark r-desc

at https://github.com/conda-forge/staged-recipes/issues/2912

bgruening commented 7 years ago

Level 4 is here: #2915 #2916 #2918

johanneskoester commented 7 years ago

@bgruening you are crazy :-). Thanks a lot!

bgruening commented 7 years ago

Level 5 is here: https://github.com/conda-forge/staged-recipes/pull/2920 & https://github.com/conda-forge/staged-recipes/pull/2921

jrderuiter commented 7 years ago

Awesome! Are there similar plans for migrating bioconductor packages? Or will they remain in bioconda for now?

johanneskoester commented 7 years ago

We only port non - bio stuff to conda-forge. Hence bioconductor will stay part of bioconda.

jakirkham commented 7 years ago

Checked off some more stuff. Looks like a couple things got turned around in the topological ordering. In particular r-tidyverse -> r-modelr -> r-broom. However r-broom is built now. Have restarted the Travis CI build for r-modelr. Though someone will need to do the same for the Travis CI build of r-tidyverse as well after the build of r-modelr completes.

johanneskoester commented 7 years ago

Thanks! The problem is that the toposort was done on the recipes from the R channel, which were sometimes older versions with slighly different dependencies. When creating the new recipes for our PRs we naturally received newer versions, sometimes breaking the toposort. Luckily, there are only few such cases.

jakirkham commented 7 years ago

Thanks for the details @johanneskoester. AFAIAC there will inevitably be a few issues when porting this number of interdependent packages. I'm not too worried about it personally. Just trying to help catch issues as they arise.

jakirkham commented 7 years ago

@jakirkham, I found this by reverting conda-forge/conda-forge.github.io@7ede267 and generating locally. Can we restore CI badges in a separate webpage so that we can see what packages are failing?

@isuruf, we decided to remove the CI badges some time ago as it made loading the webpage very slow and we were also concerned it had negative effects for our service providers. Not to mention they were sometimes out-of-date. So I don't think we will add that back. I expect the need for this sort of thing will diminish once the porting is finished. Though am open to other suggestions of how we manage such things for future porting efforts.

xref: https://github.com/conda-forge/conda-forge.github.io/pull/210#issuecomment-241140417

bgruening commented 7 years ago

@jakirkham I was able to get the new order and most of it is solved. I was not updating the list from above as I do not have write access to this issue. However, it does not matter we are finished this week.

johanneskoester commented 7 years ago

@isuruf since you seem to be active at the moment, I'll ask you directly: if we want to update the R version in the future (via CONDA_R), what is the mechanism we are supposed to apply? I guess there is some central place where CONDA_R is defined, and when we set a new version, conda-smithy will create corresponding pull requests to all r-related feedstocks?

isuruf commented 7 years ago

I think you just need to update r-base and the maintenance bots will send PRs for updating.

johanneskoester commented 7 years ago

So you mean they will automatically figure out that an r-base version update implies that CONDA_R shall be updated?

isuruf commented 7 years ago

Yes, it will create a version matrix and there'll be 2 builds for the 2 versions. If you want to drop support for the old version, conda-smithy will have to be updated. @jakirkham, correct me if I'm wrong.

johanneskoester commented 7 years ago

We are done! Thanks to everybody who helped!

mjsteinbaugh commented 7 years ago

You all are amazing. Already done before I had a chance to start some pull requests. Thanks for all the hard work.

jakirkham commented 7 years ago

@johanneskoester:

So you mean they will automatically figure out that an r-base version update implies that CONDA_R shall be updated?

@isuruf:

Yes, it will create a version matrix and there'll be 2 builds for the 2 versions. If you want to drop support for the old version, conda-smithy will have to be updated.

Correct. This is the relevant line in conda-smithy. Similar lines exist in staged-recipes as well.

johanneskoester commented 7 years ago

So what if we want to pin to CONDA_R=3.4 instead of CONDA_R=3.4.0?