search

r-multiverse / help

Discussions, issues, and feedback for R-multiverse
https://r-multiverse.org
MIT License
2 stars 2 forks source link

R-forge Packages #1

Closed shikokuchuo closed 4 months ago

shikokuchuo commented 4 months ago

Currently not being picked up by R-universe. No errors either though.

https://github.com/r-releases/r-releases/commit/c3616e348e0bcb6625730889a515ce60e32f06c0

I think these require a field 'subdir' instead of 'branch':

See https://github.com/r-universe/r-forge/blob/master/.gitmodules vs for e.g. https://github.com/r-universe/r-lib/blob/master/.gitmodules

shikokuchuo commented 4 months ago

We need them to unblock things like shiny.

wlandau commented 4 months ago

Hmmm, that's tricky.... will we be able to install releases from the R-forge repos at all? Looks like https://github.com/r-forge/matrix is a read-only mirror from the r-forge SVN. Does that mean it's the development version?

We'll have the same requirement for paws in either case: https://github.com/paws-r/paws-r.r-universe.dev/blob/main/packages.json

The nature of the one-line text file of each package submission is super important because it means we can use the GitHub API to review PRs instead of having to clone the entire repo.

What about a query syntax for URLs? Instead of https://github.com/paws-r/paws, one could write https://github.com/paws-r/paws?subdir=paws.common. Then r.releases.utils::build_universe() could parse that right from the URL.

shikokuchuo commented 4 months ago

Oh I see - yes that's quite peculiar from paws. We'd be able to parse such a syntax robustly using nanonext.

hadley commented 4 months ago

(As a possibly work around you might be able to use https://github.com/cran, since every CRAN package is guaranteed to be there)

wlandau commented 4 months ago

I think we could detect the sourceforge github URL and have it write something different in those cases. We can probably keep this vectorized in the r.releases.utils::build_universe() step as the 'subdir' is standardized.

Along those lines: almost everyone is going to have one repo per package and no subdirectories. What do you think about a manual.json file in https://github.com/r-releases/r-releases.r-universe.dev to handle extremely rare cases like R forge and paws-r by hand? Users wouldn't add these entries themselves, but they could request that we do so in https://github.com/r-releases/help/issues. Because of all the engineering required to handle extreme cases, I think this course of action would be the least painful in the long run. I could easily modify build_universe() to merge manual.json into the community contributed stuff from https://github.com/r-releases/r-releases.

wlandau commented 4 months ago

(As a possibly work around you might be able to use https://github.com/cran, since every CRAN package is guaranteed to be there)

We would like to be independent from CRAN if we can, but if R-forge does not provide releases, then pulling from https://github.com/cran might be the best option for package quality in the short term.

shikokuchuo commented 4 months ago

Yes given the rarity it might not be worth over-engineering a solution. We can think on it for a couple of days though.

@hadley we're trying to build from the ground-up in a maintainer-centric fashion. That's why we're caring about building the dependency chain of say shiny. It is helpful to keep that in mind as a backup though. Thanks.

wlandau commented 4 months ago

Yes given the rarity it might not be worth over-engineering a solution. We can think on it for a couple of days though.

Happy to pause for a couple days to think about it.

we're trying to build from the ground-up in a maintainer-centric fashion. That's why we're caring about building the dependency chain of say shiny. It is helpful to keep that in mind as a backup though.

On the one hand, that is exactly what we are trying to do. On the other, it seems like we cannot get the releases we need from https://github.com/r-forge. The CRAN version of Matrix is 1.6-5, https://R-forge.R-project.org/src/contrib/Matrix_1.7-0.tar.gz is dated 2024-02-08, and https://github.com/r-forge/matrix/blob/master/pkg/Matrix is also version 1.7-0 but dated 2024-02-22. Matrix has a long list of reverse depends and reverse imports, and I think it is important to use the most compatible and reliable version available. That may force us to use https://github.com/cran until there is a viable alternative for production releases.

shikokuchuo commented 4 months ago

Yes it seems that's how R-forge works. The Github mirror for Matrix does keep the release branch e.g. matrix/branches/Matrix-1-6-branch But that's not the case for other projects such as zoo.

Also Matrix itself is a recommended package so we don't really need to worry about people not being able to install it :) It's reliability we're after, no need to be dogmatic about pulling from CRAN mirrors.

wlandau commented 4 months ago

I am with you on minimizing dependence on CRAN, and I hope I don’t seem dogmatic about using CRAN mirrors. It’s just that the goal of r-releases is to provide real releases that the maintainers voluntarily distribute for production. The R-forge GitHub mirror has only development versions, and even if users can install them, they may not perform as reliably. Does that make sense? The reason I think we need the CRAN for the moment is desperation, not dogma :)

wlandau commented 4 months ago

I think it’s important to build up trust in this initial phase, and it that means going to CRAN for a small number of the older well-established packages to get true releases, I think it puts us on good footing. Then later when else have built up enough trust (from users using the r-releases universe without reliability issues) then we can talk to the R-forge maintainers and ask them to put releases on GitHub. It may not be realistic to be totally independent from cran in all cases until we gain enough trust and enough traction. Does that make sense?

shikokuchuo commented 4 months ago

Yes, we're on the same page - that was a comment to myself as much as directed at anyone. Once established, we can ask mmaechler if he wishes to contribute new releases of Matrix to us as well as CRAN.

wlandau commented 4 months ago

Glad we are on the same page. Feel free to add https://github.com/cran/Matrix and other R-forge packages to https://github.com/r-releases/r-releases in the usual way.

Okay if I flag https://github.com/cran/* URLs for manual review? I think we should nudge package contributors away from the CRAN mirror.

With R-forge taken care of, that makes subdirectory cases even rarer and easier to manage manually. Could I go ahead and implement that infrastructure for cases like {paws}? I plan to add a fixed json packages file and check PRs for name conflicts with that file.

wlandau commented 4 months ago

Okay if I flag https://github.com/cran/* URLs for manual review?

Related: I think it would be good to flag all non-GitHub/non-gitlab/non-https URLs for manual review.

wlandau commented 4 months ago

With R-forge taken care of, that makes subdirectory cases even rarer and easier to manage manually. Could I go ahead and implement that infrastructure for cases like {paws}? I plan to add a fixed json packages file and check PRs for name conflicts with that file.

Actually it will be much simpler to allow full JSON lists as direct contributions to https://github.com/r-releases/r-releases/pulls (but flag them all for manual review). And it will be easy to modify build_universe() for this.

shikokuchuo commented 4 months ago

Glad we are on the same page. Feel free to add https://github.com/cran/Matrix and other R-forge packages to https://github.com/r-releases/r-releases in the usual way.

Okay if I flag https://github.com/cran/* URLs for manual review? I think we should nudge package contributors away from the CRAN mirror.

With R-forge taken care of, that makes subdirectory cases even rarer and easier to manage manually. Could I go ahead and implement that infrastructure for cases like {paws}? I plan to add a fixed json packages file and check PRs for name conflicts with that file.

Don't let me stop you or slow you down. At this stage it makes sense for you to build out as you see fit. If you think it'll work that's good enough. I kind of did the same with the CRAN repo URL checking.

wlandau commented 4 months ago

Thanks! I just opened https://github.com/r-releases/r.releases.utils/pull/2. I think that PR is enough to close the current thread here.

wlandau commented 4 months ago

Looks like there are still packages using the R-forge mirror: https://github.com/r-releases/r-releases.r-universe.dev/actions/runs/8099610675/job/22135762281#step:6:55.I will change this to CRAN.

I think we'll need to relax one of the assertions I recently added in build_universe(), since sometimes GitHub repo names disagree with the package name.