Add GitHub action to copy unversioned pages over latest

[melissawm]

Supersedes napari/docs#230

Creates a GitHub action, triggered by a commit to the gh-pages branch in this repo, to copy over the contents of the developer documentation (under the Contributing rubric of the website) from latest to stable. Also copies over the index.html (landing page) from latest to stable.

[melissawm]

cc @jni

[psobolewskiPhD]

@jni in the docs meeting we decided that you should do the honors of pushing the merge button here.

[melissawm]

I will open a PR with a note in the docs about how this works :+1:

[psobolewskiPhD]

Hmm I think the theme consistency is a real issue -- e.g. we plan to update the theme soon. But at the same time, this won't be a frequent thing. Is there any way we can release stable to napari.org without changing the content, just with new theme? Or are we stuck to napari release cadence.

[melissawm]

There is.. but at that stage I'm not sure this is very sustainable.

Talking to other folks from CPython, what they do is item 3. from this comment: https://github.com/napari/docs/issues/86#issuecomment-2013475922

They have a separate repo for the contributing guide, and use intersphinx to link between it all.

[Czaki]

Is there any way we can release stable to napari.org without changing the content, just with new theme? Or are we stuck to napari release cadence.

It will work only until there will be some conflict in docs and released version dependecies.

[melissawm]

The other thing that may happen with this current PR is that links may be broken. Becuase we will be using latest version of the devguide, if pages or anchors are moved in main, they will be broken when navigating on stable. So it is something to keep in mind...

[jni]

I think that we should use same approach as in banner. So we should have JS that will download page from dev, extract content from dev and then replace current with one from dev.

Ooof. I really dislike the idea of custom js for this.

My main question is how much the theme is really expected to change here. Are we talking about minor visual inconsistencies or completely broken functionality? I guess it would be mostly the former, but potentially the latter, in some cases. (Same for the broken link issue.)

Bah! I am unhappy. Why can't things be simple. 😂

Talking to other folks from CPython, what they do is item 3. from this comment: https://github.com/napari/docs/issues/86#issuecomment-2013475922

This is also the approach for the matplotlib home page. Maybe there is an advantage to doing as others do here... But I am sad about the idea of a third (fourth if you count the built site) repo. Nevertheless, I much prefer following what the community is doing to writing some more complicated js to fiddle with document contents.

[jni]

What are you leaning towards after this discussion @melissawm?

[Czaki]

I also prefer 3 repo setup than JS, but in the past there was strong opposition to this.

[jni]

in the past there was strong opposition to this.

sounds like me 😅 but I probably did not appreciate the downsides of the copying approach. And anyway I think there will be even stronger opposition to the js approach. 😬🤣

[melissawm]

This is also the approach for the matplotlib home page. Maybe there is an advantage to doing as others do here... But I am sad about the idea of a third (fourth if you count the built site) repo. Nevertheless, I much prefer following what the community is doing to writing some more complicated js to fiddle with document contents.

In our case, we do not need another repo. Another folder in the same napari/docs repo works fine. I have explored this and got it to work, with the disadvantage of having to fix a bunch of internal links on the versioned pages (since now the two folders will act as two different sites, we can use intersphinx to link across versioned and unversioned pages. We just have to fix the links once and they should work forever).

Bah! I am unhappy. Why can't things be simple. 😂

Exactly what I said to the cpython dev I was discussing this with 🤣

What are you leaning towards after this discussion @melissawm?

I am now reconsidering the two-folder approach as more reasonable and less of a brute force solution. So I will mark this PR as draft and try to formalize my thoughts there.

Sorry about the back and forth here folks - however I have spoken to multiple people in different communities and there's just no ready-made solution for this, we are all grasping in the dark here 😆

[melissawm]

Ok, after one billion years (I'm sorry 😭 ) here's an alternate approach: https://github.com/melissawm/napari-docs/pull/11

You can see the live version here: https://melissawm.github.io/napari.github.io/dev/home.html

This is done taking an approach of having two separate folders, with two separate conf.py files and stitching the two sites together with our Makefile. There is some redirection working which is why there will be some flashing pages. There may be alternative solutions for that (one that I discussed with a couple other folks is that if we have more control over the web server we are using, if it's something other than github-pages, we could potentially make the redirect faster. Not an expert on that though).

Let me know if you have other ideas. I had extensive conversations with several people including some of the pydata-sphinx-theme and other scientific project maintainers and I don't see another easy/better solution to this.

Thanks for the patience!

[jni]

Don't apologise!!! We all have our long-neglected issues and PRs. 😅 And I know it is 110% not because we don't care. #MaintainerLife

Can you summarise the changes in melissawm/napari-docs#11 @melissawm? The diff view is useless in this kind of reorg. 😂 But also, maybe the reorg can be more minimal? docs stays the same and we only move unversioned files to a new folder? (I'm assuming the huge diff is because ~all the files are being moved?)

Anyway a couple of comments:

• in the demo site, clicking on usage doesn't take me to the stable docs version. Which makes sense cos the demo link is to dev/ rather than to stable. But I don't know how we're supposed to test this out.
• Thinking back to the discussion about the copying approach mixing themes, I don't actually know how this approach avoids it: I presume the unversioned site is getting built with the newest theme, and then when you click on a link to the stable site, it will still be the old theme... So it's the same problem still?

I'm starting to think about the following idea, which is basically this PR on steroids (no need for js): if we tag a PR with "theme-update", then that first checks out the latest release tag, rebuilds the whole stable site again with the new theme + applying the patch from that PR, and then it builds the new site and copies the files over. This allows us to have infrequent theme updates, and keeps the dev and stable sites always synced in look and feel, rather than diverging, which is kinda nice!

And, idea number 2 (could come later), if we tag a PR with (say) "reorg" or "nav-break", then we stop updating the home page until the next release. (I'm not sure how to have long-lasting effects here but I'm sure something can be figured out... For example, we can have an action that runs when applying a milestone, and checks if any PRs in that milestone have the "reorg" tag, and if so, automatically labels the PR as well. And anyway I think we can leave this till later since we are not planning any reorgs...

Sorry @melissawm I know my above comments muddy the waters again and I hope it doesn't feel disrespectful to your efforts, but sometimes it takes time for an idea to really percolate through one's brain... And I want to make sure we're not missing a simpler option here... 🙏 🙏 🙏

[jni]

@Czaki @psobolewskiPhD thoughts?

[melissawm]

No no, that's exactly what I'm hoping here, to run ideas by you folks! It's just that in this case like most cases it's so much easier ti visualize the results if you have them clickable!

Let me think about you suggestion and I'll come back soon.

[Czaki]

I would like to point that uv pip compile provide such option:

  -P, --upgrade-package <UPGRADE_PACKAGE>
          Allow upgrades for a specific package, ignoring pinned versions in the existing output file

So it is possible to upgrade only a selected package, so rebuild constraints only for docs build

[melissawm]

Can you summarise the changes in melissawm/napari-docs#11 @melissawm? The diff view is useless in this kind of reorg. 😂 But also, maybe the reorg can be more minimal? docs stays the same and we only move unversioned files to a new folder? (I'm assuming the huge diff is because ~all the files are being moved?)

Only the unversioned pages are moved, but many links need to be updated - that's why I'm touching a few of the docs files. A few redirect files also need to be created in both folders. But that's it, the main updates are to the Makefile, moving files around, and moving static files to root-level folders.

Anyway a couple of comments:

* in the demo site, clicking on usage doesn't take me to the stable docs version. Which makes sense cos the demo link is to dev/ rather than to stable. But I don't know how we're supposed to test this out.

Yeah, the demo site only has dev. This is both because the deployed demo site is only built for dev but only because this logic is not implemented for stable at this point. To have that we would have to rebuild and reorganize it too.

* Thinking back to the discussion about the copying approach mixing themes, I don't actually know how this approach avoids it: I presume the unversioned site is getting built with the newest theme, and then when you click on a link to the stable site, it will still be the old theme... So it's the same problem still?

Yes there is no way to fix that unless we rebuild the whole site to be the same theme for all previous versions. There is no solution for that.

I'm starting to think about the following idea, which is basically this PR on steroids (no need for js): if we tag a PR with "theme-update", then that first checks out the latest release tag, rebuilds the whole stable site again with the new theme + applying the patch from that PR, and then it builds the new site and copies the files over. This allows us to have infrequent theme updates, and keeps the dev and stable sites always synced in look and feel, rather than diverging, which is kinda nice!

We could do that, although I would argue that slight differences could actually be useful in signaling to users they are jumping to a different version/area of the site, but no strong feelings on it.

And, idea number 2 (could come later), if we tag a PR with (say) "reorg" or "nav-break", then we stop updating the home page until the next release. (I'm not sure how to have long-lasting effects here but I'm sure something can be figured out... For example, we can have an action that runs when applying a milestone, and checks if any PRs in that milestone have the "reorg" tag, and if so, automatically labels the PR as well. And anyway I think we can leave this till later since we are not planning any reorgs...

I'm not really sure how this would work in practice but I think I get the main idea - we could make all of this manual too (something like a forward-porting thing for the unversioned pages).

Anyway, at this point I think the best idea would be to make a decision and run with it, so maybe we need some kind of vote/reach consensus somehow?

[psobolewskiPhD]

Sorry for the late response. I think the concern about theme mismatches is real, but worse than that would be change to the top nav bar. That would be really jarring. Ultimately, I think we need a mechanism to "release" docs independently of napari releases. Obviously we'd not document un-released stuff, but we could sync themes, fix bugs, update/add content. Incidentally, I think this would make contributing to docs more rewarding.

[jni]

I think the concern about theme mismatches is real, but worse than that would be change to the top nav bar. That would be really jarring.

yeah, try going to numpy.org then click on "documentation" 😬

Ultimately, I think we need a mechanism to "release" docs independently of napari releases.

I think the copying approach in this PR would make working towards that goal easier than the two-sites. I think my proposal above sort of achieves that @psobolewskiPhD? (ie we'd have a mechanism to rebuild and re-deploy the stable site any time.)

[jni]

In short, I think we should merge this actually. 😂 Caveats and all. Then work on the steroids.

[melissawm]

We can do that at the docs Meeting today, wdyt? That means we could immediately capture feedback and I can work on any fixes before the weekend.

[psobolewskiPhD]

yeah, try going to numpy.org then click on "documentation" 😬

This isn't fair, because numpy isn't trying to make it look like numpy.org and numpy docs are the same website.

[Czaki]

I also think that we should rebuild stable docs basing on constraints files.

[Czaki]

It is good that we have constraints for 0.4.19 release. We will be able to rebuild it from scratch.