feat: use the scientific-python developer pages

[henryiii]

This moves to using the Scientific-Python Development Pages / cookie / repo-review. Pages that moved there redirect (links or menu items) and menu items have link icons.

---

I spent most of my time at the First Scientific Python Developers Summit working on moving the Scikit-HEP developer pages into Scientific Python and merging with with the NSLS-II guide. The finished result is visible at learn.scientific-python.org/development. A live version (the final version gets updated on tags) is available at scientific-python-cookie.readthedocs.io.

A lot of work has gone into this, here are the highlights that have been added or changed in the last month during and since the summit:

• Two new sections based on NSLS-II (tutorials, principles). Environment page moved to tutorials, along with four more new pages.
• A brand new section: patterns, with commonly used patterns. Three pages there now; including data, backports, and exports.
• Brand new guide page on documentation, along with new sp-repo-review checks to help with readthedocs.
• Big update to GHA CI page (in live preview only, not released yet)
• CLI entry points mentioned now.
• Python 3.12 added, Python 3.7 dropped
• Using latest Jekyll (4), Just the Docs. More colorful callout boxes. Plugins used now.
• New sp-repo-review badges throughout (more on that later!)
• Live PR preview (provided by probably the world’s first readthedocs Jekyll build!). Developed with the Zarr developers during the summit.
• Better advertising for cookie/sp-repo-review on the index page(s)
• Auto bump and auto-sync CI jobs

I’ve also done something I’ve wanted to do for a long time: the guide, the cookie-cutter, and the checks are all in a single repo! The repo is https://github.com/scientific-python/cookie, which is the moved scikit-hep/cookie (the old URL for cookiecutter still works!). A lot of work went into the cookie cutter, too!

• Generalized defaults. We still have special integration if someone sets the org to scikit-hep; the same integration can be offered to other orgs.
• All custom hooks removed; standard jinja now used throughout. Using cookiecutter computed variables idiom too. Windows still fully supported and tested. Adding new choices is much easier now.
• Added cruft (a cookiecutter updater) testing.
• Dual-supporting copier now too, a cookiecutter replacement with a huge CLI improvement (and also supports updating). Might be the first project to support both at the same time. CI (or nox locally) checks to ensure generation is identical.
• Improved CLI experience even if using cookiecutter (like no longer requesting current year). Much better with copier, including validation, descriptive text, arrow selection, requirements, etc.
• Reworked docs template.

Along with this was probably the biggest change, one requested by several people at the summit: scientific-python/repo-review (was scikit-hep/repo-review) is now a completely general framework for implementing checks in Python 3.10+. The checks have been moved to `sp-repo-review, which is now part of scientific-python/cookie. There are too many changes to list here, so just the key ones in 0.6, 0.7 & 0.8:

• Extensive, beautiful documentation for check authors at https://repo-review.readthedocs.io/ (used to help guide the new docs guide page & template update!)
• Support for four output formats, rich (improved), svg, json (new), html (new)
• Support for listing all checks
• GitHub Actions support with HTML step summary, pre-commit support
• Generalized topological sorting to fixtures, dynamic fixtures
• Dynamic check selection (via fixtures! Basically everything is powered by fixtures now.)
• URL support in all output formats (including the terminal and WebApp!)
• Support for package not at root of repo
• Support for running on a remote repo from the command line
• Support for select/ignore config in pyproject.toml or command line.
• Pretty printed and controllable sorting for families
• Supports running from Python, including inside a readme with something like cog.

The full changelog is at https://repo-review.readthedocs.io/en/latest/changelog.html - you can even see the 10 beta releases in-between 0.6.x and 0.7.0 where a lot of this refactoring work was happening. If you have configuration you’d like to write check for, feel free to write a plugin!

Finally, sp-repo-review contains the previous repo-review plugins with checks:

• Fully cross-linked with the development guide. Every check has a URL that points to a matching badge inside the development guide where the thing the check is looking for is being discussed!
• Full list of checks (including URLs), produced by cog, in https://pypi.org/project/sp-repo-review/
• Also ships with GitHub Action and pre-commit checks
• Released (in sync with cookie & guide, as they are in the same repo) as CalVer, release notes at https://github.com/scientific-python/cookie/releases
• Split CI that selects just want to run based on changed files, with green checkmark that respects skips (based on the excellent contrition to pypa/build).

I’m also working with a fellow this year to provide an even better terminal experience. I’ll be adding more documentation on setting up compiled projects, as well. I’m hoping to work on better backend validation with repo-review, possibly by writing a plugin for pyproject-validate? We’ll see.

[eduardo-rodrigues]

Hi. Give me until early next week to look into this as it seems to be a lot. Worth emailing the gang as it affects everyone, I guess ...

If I understand a lot will be removed and redirected. So we loose "control" over a lot of development tips. Is that correct? Do we have any contribution to "Scientific-Python Development Pages / cookie / repo-review"?

[jpivarski]

Since we can be as involved in scientific-python.org as we want, we still do have the opportunity to control what's in these pages. But it's true that we're now sharing the responsibility with a larger group.

[henryiii]

And quite correct about emailing the full group, I've got about 1/3 of an email drafted already and hopefully will have a chance to finish it soon. Edit: finished and sent, also pasted most of it into the issue description above.

[henryiii]

FYI, https://github.com/scikit-hep/scikit-hep.github.io/pull/344 caused conflicts that I now have to spend time undoing. Any progress on getting signoffs/reviews of this?

scientific-python/cookie has added more features in the meantime:

• Support for cookiecutter 2.2 pretty descriptions. Cookiecutter 2.2 came out about 4 hours ago, FYI. :)
• GitLab CI support from @kratsg
• A compiled projects page! Finally! With scikit-build-core, meson-python, and maturin. The page shows real outputs from the cookiecutter, kept in sync with cog (a huge benefit of a monorepo for this!)
• Updates for Ruff's move and support for requires-python
• Lots of additions for GitHub Actions.

And more: https://github.com/scientific-python/cookie/releases/tag/2023.07.06

Reminder I'm happy to request anyone interested to be added to the scientific-python/cookie repo. Or repo-review.

[eduardo-rodrigues]

Sorry for that! It seemed benign. People are really busy with commission and stuff. Please give it a couple more days, not that much for such a big change. I will try and look asap; now is rather late here.

[henryiii]

Please take your time, as long as there's progress and this doesn't keep slipping back and getting merge conflicts! :)

[henryiii]

One of the main questions I think you'll want to review is, should the pages show up in the sidebar still? I've put them there for now, but maybe it would be better not to have side bar links go to somewhere else. I can remove them from the sidebar while still keeping the page redirects (so links continue to work) if that's better. We can also keep the links in the sidebar for now, and remove them later. Current version:

[eduardo-rodrigues]

Hello @henryiii, first off, a big 🎩 for this huge and impressive amount of work. I hope the folks at Scientific Python appreciated the contribution!

The move is effectively done at this point, as we would probably be having a less maintained version here, would we not make the move you suggest. But your suggestion does make sense. It is just about the best way to engage with Scientific Python, actually, so I agree with you and @jpivarski that the move is the way to go.

To be seen how things evolve but at this point in time I quite like the fact that you left the menu but swapped content for the relevant links. I also find it nice that I read above about the move of the site to Scientific Python. If on their side there is a little sentence saying the thing came from Scikit-HEP, that's also a fair thing IMO.

So 👍 on this!

[henryiii]

If on their side there is a little sentence saying the thing came from Scikit-HEP, that's also a fair thing IMO.

Added at https://github.com/scientific-python/cookie/pull/219.

[eduardo-rodrigues]

@henryiii, just checking the site the (redirect) links at https://scikit-hep.org/developer do not seem to be there? At least I do not see how to get to Scientific Python for the various links/pages.

[henryiii]

Ahh, must not be an allowed plugin for the integrated GH Pages build. I can update this tonight to use a proper GHA builder.

[henryiii]

Should work now with #347.

[eduardo-rodrigues]

Yep. I have 2 suggestions:

• Check all links as I see at least one giving 404.
• If possible I would make the redirects to new tabs to keep the navigator on scikit-hep and move to a separate page for the devel stuff. Dunno, seems nicer, but not a big deal. Thank you for the speedy fix!

[henryiii]

Link checker is hard, there are 314 failures, including a lot of indico timeouts and such. https://scikit-hep.org/pylhe/ isn't a real page. Also not seeing https://davidyakobovitch.github.io/2019-04-01_fnal/.