Planning to keep up with pydata sphinx theme

[jni]

Hi @codemonkey800, @neuromusic, et al,

Thanks again for all your efforts with the napari sphinx theme. As I've mentioned before, I think we're in a great place with the build right now.

However, I think we discussed early on that forking the theme was a first-pass solution. Long term, it seems hard to maintain a fork forever. (Have a look at the number of PRs in the latest pydata sphinx theme release!)

I think instead we should explore using pydata sphinx theme directly, then customizing it, and for the more elaborate things such as the calendar directive, developing additional sphinx extension packages, that we would use in addition to sphinx pydata theme + custom CSS.

If there's any missing features in pydata sphinx theme that would block this approach, we could possibly contribute them upstream?

What do you all think? I am speaking from a position of relative ignorance as regards to the engineering effort involved, but reading the release notes of the pydata theme I see quite a few features and improvements that we could benefit from, and I see that continuing going forward, as they have built some good momentum over the past year. So I think it's worth exploring this idea!

CC @melissawm

[codemonkey800]

Long term, it seems hard to maintain a fork forever.

Hmm, I imagined the direction from here was to maintain it as our own thing, but I understand the desire to use the theme directly to get all of the contributions added to it.

If there's any missing features in pydata sphinx theme that would block this approach, we could possibly contribute them upstream?

We could suggest contributing some of these things upstream, but I imagine we'd be met with some resistance to the idea of adding React, Tailwind, and the express.js dev server we use for development. Ultimately, I don't believe there are any things that we can contribute upstream since everything is specific to our design and dev environment.

What do you all think?

While the base theme is mostly sphinx, we've added a significant amount of dev tooling and additional dependencies to make it easier for us to develop the theme in a way that is fast and can get around the tooling and build speed limitations that are currently present in sphinx / jupyterbook / etc.

reading the release notes of the pydata theme I see quite a few features and improvements that we could benefit from, and I see that continuing going forward, as they have built some good momentum over the past year. So I think it's worth exploring this idea!

That's a good point, and would be worth exploring so we could get those benefits automatically for free.

However, this would mean we would need to figure out how to get the custom CSS / JS in the docs, which would involve a node environment to build the CSS and JS. Since a big motivator for this iteration of the theme was to remove the node environment, then we would need to figure out an alternative approach because the code is primarily in SCSS and TypeScript so we can't drop it to regular CSS / JS without rewriting everything. Additionally, it will be harder for us to maintain given that we leverage a lot of the benefits that SCSS and TypeScript provides.

[codemonkey800]

There's definitely a lot of value in this because then we'd get continuous updates to the sphinx theme while only maintaining napari specific scripting and styling ourselves. However, if we go with this approach, I believe this will require significantly more effort than the most recent rewrite. I think this will require at most a half for us to plan and implement since there are a lot of parts to getting this work on our end. It may be shorter than half, but considering I am also working on the hub and SDS stuff, it may require more time to deliver. This is fine, but want to be cognizant of the amount of effort it would take with respect to other priorities

Another option we could do is just using sphinx theme with extremely minimal changes to styling and functionality. Maybe changing the top bar to napari blue for example. Unfortunately, it would mean scrapping a lot of designs we've implemented again, but at least it'll reduce the surface area for design to focus on in the future

For the calendar, we can definitely extract it to its own package if:

1. pip allows us to share JS and CSS files in a pip package
2. sphinx will allow us to use JS and CSS files from a pip package

If the above are true, then we might also be able to use it for the core SCSS and TypeScript we use for the theme right now. Otherwise we can also make the calendar a separate web app and embed it in an iframe.

@neuromusic is on PTO right now, but definitely curious on his thoughts here given the amount of work it'll take to do this

[jni]

For the calendar, we can definitely extract it to its own package if:

1. pip allows us to share JS and CSS files in a pip package
2. sphinx will allow us to use JS and CSS files from a pip package

I think both of those are true? I bet @tlambert03 knows for sure. A quick search didn't turn up anything definitive though!

Hmm, I imagined the direction from here was to maintain it as our own thing, but I understand the desire to use the theme directly to get all of the contributions added to it.

A big question for me is how hard it is to merge sphinx pydata theme main with our current main. If that's basically quick and easy, then we can keep maintaining the fork. If it's a huge lift, then I don't find it sustainable given that as you point out, we have limited developer effort to apply to this.

Unfortunately, it would mean scrapping a lot of designs we've implemented again,

Can you clarify which parts of the design require custom js rather than CSS and config customizations applied on top of pydata? Also note that there is limited support for custom js in the PyData theme, e.g. here.

I also wonder how much the dev tooling (as opposed to the site design) is driving the need for a fork? I agree that bringing in a big js stack is a big ask, but ultimately I think either it's a big benefit and you should be able to convince the PyData community of that benefit, or it's a minor benefit and we can get by without it. (With the caveat that there is of course some nuance — I know for sure it's hard to move communities. But in the long term, it's usually worth it compared to going it alone.)

[codemonkey800]

Can you clarify which parts of the design require custom js rather than CSS and config customizations applied on top of pydata? Also note that there is limited support for custom js in the PyData theme, e.g. here.

Everything is located here: https://github.com/napari/napari-sphinx-theme/blob/main/src/napari_sphinx_theme/assets/napari.tsx

I think if pydata theme is mostly onboard with:

1. Adding react to the repo
2. Using react to fix templating issues

then we might be fine. Some of these things are napari specific so we could pull some of the functions out too. Or if folks from pydata are willing to refactor functionality that we have in the above script into their codebase, then I think that would be fine too. there are a lot of things we are doing in react for speed, but I bet there's a "proper" sphinx way to do it too. it's just incredibly slow to use the existing tooling, so maybe someone with more time on their repo can look into doing it

I also wonder how much the dev tooling (as opposed to the site design) is driving the need for a fork? I agree that bringing in a big js stack is a big ask, but ultimately I think either it's a big benefit and you should be able to convince the PyData community of that benefit, or it's a minor benefit and we can get by without it. (With the caveat that there is of course some nuance — I know for sure it's hard to move communities. But in the long term, it's usually worth it compared to going it alone.)

can you help facilitate that? maybe we can see if there's interest in the pydata community in adopting an express.js proxy based server to speed up development. they're using sphinx-theme-builder which is incredibly slow for development because it needs to run the webpack build from cold start, copy the assets to the sphinx dir, and re-run the sphinx build again on every change

[tlambert03]

ultimately I think either it's a big benefit and you should be able to convince the PyData community of that benefit, or it's a minor benefit and we can get by without it. (With the caveat that there is of course some nuance

yeah, lots of nuance i think. I think it really comes down to the scope of the project, and the people actually using it. I suspect that adding react to pydata would be met with a fair amount of resistance; and if I were them I would resist too on the grounds that YAGNI for 98% of what the target audience is going to use the theme for, and in the interest of keeping things lean.
On the other hand, if you are coming from a js background, and know and love react and the tooling... then obviously there's a benefit to having it... but to me it feels firmly in the "layer on top of core" camp. They might disagree though!

I think it would be ideal if it could be compartmentalized (much as we try to do in napari with things like superqt, and magicgui) ... try to determine what bits are novel and try to inject them into the theme using template overrides and such (essentially acting as a child theme that depends on rather than forks the upstream theme).

[codemonkey800]

we could drop the react stuff from the core theme JS, but we'll need it for the calendar at least. but dropping react will require refactoring some of the templating stuff that we do like

• replacing the version icons
• replace sidebar icons
• fix search container
• match search results to napari design
• use napari menu icon for app bar
• render napari icons for admonition dropdowns

a lot of these could be done with react, but it will take time to refactor, and not sure which ones are appropriate to bring back to pydata, if any.

[jni]

@codemonkey800 I think this bit from @tlambert03 is the key that I hope is feasible:

acting as a child theme that depends on rather than forks the upstream theme.

Maybe we can keep all the tooling but apply the theme by depending on sphinx-pydata-theme?

This is Python after all. Worst case, you can depend on it, then dynamically at run time copy all the files over to our own theme's directory! 😂

[jni]

Maybe we can keep all the tooling but apply the theme by depending on sphinx-pydata-theme?

It looks like this is not only possible but common in the ecosystem:

https://twitter.com/choldgraf/status/1564598627749830657

Note that the twitter thread was in the context of even more fantastic updates from the pydata sphinx theme. Can we bump this issue again @codemonkey800 @neuromusic?

[jni]

It looks like this is not only possible but common in the ecosystem:

More directly: the dask theme depends on the sphinx book theme which depends on pydata-sphinx-theme.

[jni]

The matplotlib sphinx theme also depends on pydata and adds customization. I think it's gorgeous!

https://github.com/matplotlib/mpl-sphinx-theme

[Czaki]

we could drop the react stuff from the core theme JS, but we'll need it for the calendar at least. but dropping react will require refactoring some of the templating stuff that we do like

* [replacing the version icons](https://github.com/napari/napari-sphinx-theme/blob/a322d287543f76a1b74c163d8db4792598a6bd64/src/napari_sphinx_theme/assets/napari.tsx#L195-L295)

* [replace sidebar icons](https://github.com/napari/napari-sphinx-theme/blob/a322d287543f76a1b74c163d8db4792598a6bd64/src/napari_sphinx_theme/assets/napari.tsx#L195-L336)

* [fix search container](https://github.com/napari/napari-sphinx-theme/blob/a322d287543f76a1b74c163d8db4792598a6bd64/src/napari_sphinx_theme/assets/napari.tsx#L371-L401)

* [match search results to napari design](https://github.com/napari/napari-sphinx-theme/blob/a322d287543f76a1b74c163d8db4792598a6bd64/src/napari_sphinx_theme/assets/napari.tsx#L444-L478)

* [use napari menu icon for app bar](https://github.com/napari/napari-sphinx-theme/blob/a322d287543f76a1b74c163d8db4792598a6bd64/src/napari_sphinx_theme/assets/napari.tsx#L480-L502)

* [render napari icons for admonition dropdowns](https://github.com/napari/napari-sphinx-theme/blob/a322d287543f76a1b74c163d8db4792598a6bd64/src/napari_sphinx_theme/assets/napari.tsx#L525-L533)

a lot of these could be done with react, but it will take time to refactor, and not sure which ones are appropriate to bring back to pydata, if any.

all of this should be quickly injected as an extension of pydata theme.