An overarching gallery: learn.sunpy.org?

[wafels]

Description

There's room for more extensive tutorials on how to do solar physics with SunPy. For example, solar physics coordinate systems are a complex topic in themselves, and are fundamental in understanding what you're looking at when you look at a Map. Plus there is a learning opportunity to teach users about the data and how it is acquired.

Tutorials would have a heavier teaching/learning component compared to the existing examples in the Example Gallery. Such tutorials would also allow us to explain some design choices in SunPy, as well as some basic solar physics.

[Cadair]

https://youtu.be/UQHaGhC7C2E?t=49

[dstansby]

I was thinking the other day that our Guide (https://docs.sunpy.org/en/stable/guide/index.html) might be more suited as a set of tutorials.

[Cadair]

I have moved this here as I think it's bigger than core.

We discussed this at the Coordination Meeting and there was a lot of thoughts about we need:

• Whole workflow examples
• A way to search / explore all the example galleries from one place (i.e. all the affiliated packages, instrument specific galleries (e.g. https://solar-orbiter-python.readthedocs.io/ )
• Other tutorials etc?

[wtbarnes]

I'm going to attempt to summarize some points from a discussion @ayshih and I had the other day related to this effort. Some of this is definitely repeated in the above comments.

Our example gallery currently has three main purposes:

1. Provide API documentation for anything that requires plotting something (e.g. assume_spherical_screen)
2. Give an example of fairly self-contained functionality (e.g. here is how you plot a coordinate on top of a map)
3. Demonstrate a workflow (e.g. the AIA-to-STEREO coordinate conversion example)

Mixing of three different kinds of content, especially including examples that fall into the third category, has caused our example gallery to grow quite large (a good thing!) making it harder to find the example you're looking for (a bad thing!).

One possible solution for mitigating this scalability problem would be to spin off our longer-form narrative examples (those that fall into category 3) into a set of tutorials separate from our documentation, similar to learn.astropy.org[^1]. As @dstansby points out above, this could also serve to replace the existing user guide, which I don't think we often point people toward. In the case of the user guide format, we could start with the structure of our existing user guide and then add our more advanced narrative-heavy examples (e.g. the AIA-STEREO coordinates one) as advanced topics that follow the introductory sections. How exactly to go about structuring this would require some thought.

Some pros to this approach:

• Having a dedicated resource for tutorials makes the resources more accessible to beginners. The narrative structure could help guide new users as opposed to a gallery where there is no clear starting point.
• Removing longer examples from the example gallery make it a bit easier to parse and find what you're looking for [^2]. We could scope it to examples that fall into the category of "here is how to use this one function/class in sunpy" as opposed to "here's how to do this common task in solar data analysis"
• As noted above by @Cadair, this would also allow us to demonstrate functionality across all packages in the ecosystem, not just core and would (presumably) make the functionality in our affiliated packages more discoverable.

And some cons:

• This would be yet another "thing" for us to maintain.
• By decoupling the examples from our docs, we run the risk of our examples going stale.
• Imposing a narrative structure, while useful, may make it more difficult to contribute as opposed to just adding a single self-contained example.

It would be great to have people's thoughts on this. I know we've batted many of these ideas around for a while, but I'm just trying to solidify them in my mind a bit better. This should definitely be a coordination meeting topic, but I plan to make progress on this well-ahead of the meeting as well.

[^1]: I have to say, after looking at the learn.astropy site just now, I'm not entirely sure this same tech stack could be adapted for our efforts that easily. It appears to be built entirely on top of a javascript framework and just links to static renderings of the notebooks with no option for interactivity. Something like JupyterBook seems far more attractive to me.

[^2]: This is not to say there isn't still work to be done in making the example gallery more easily searchable. See sunpy/sunpy#2164

[wtbarnes]

A project of smaller scope, but that would achieve some of these objectives is to just have a gallery page that collects all of the example gallery entries from all of the affiliated and/or sponsored packages. Ideally, this would also support search across all of these gallery examples.

[Cadair]

I found this today: https://sphinx-thebe.readthedocs.io/en/latest/index.html which lets you have executable code blocks backed by binder. Building this into the gallery would be super nifty:tm:

[wtbarnes]

A few ideas for cross-project workflows that documentation like this could demonstrate:

• Extrapolating field lines from an HMI magnetogram and aligning them with an AIA image
• Querying cutouts of AIA images from the JSOC, aligning them using reproject, stacking them into an NDCube, and performing a time lag analysis with sunkit-image
• Selecting pixels from an EIS raster and doing a DEM analysis with demcmc
• Tracing field lines from a Carrington map and transforming them to lie on top of the active region in an AIA image
• Selecting intensity that falls along a loop defined by a traced field line from pfsspy using pixelate_coord_path and sample_at_coords (for more than 1 instrument???)
• Creating stack plots/time distance diagrams from a stack of images as a function of time in an NDCube using pixelate_coord_path and sample_at_coords
• Searching for SPICE data with sunpy-soar, extracting a spectra, and fitting it with an astropy model.

The criteria should be that:

1. the analysis should be too involved for a short gallery example,
2. have significant accompanying narrative,
3. be a task that a scientist may actually want to do in the course of their work, and
4. involve more than 1 project in the SunPy ecosystem.

[wtbarnes]

The Pythia cookbooks may provide some good inspiration for this kind of an effort: https://cookbooks.projectpythia.org/

[wtbarnes]

Notes from Stuart and Will traveling to the airport and talking about scope of what this thing should be

Philosophy

• Example workflows for doing solar physics in Python
• Examples of how to use packages in the ecosystem together
• Long-form tutorial content
• Too complex for the gallery
• In the diataxis language, these will be tutorials
• Achieve a thing at the end--the user should get a result
• Users should see checkpoints along the way
• Reference more in depth explanations, but don't mix topic guides and tutorial content
• We want to show people what is possible, what is there in terms of the entire ecosystem
• Aid in discoverability, show people what is there without them reading all of the APIs
• Should be linear

Technical bits

• Format: MyST Markdown notebook files
• Use minimal amount of remote data
• Use sample data when available
• Potentially add more sample data to cover these use cases
• This acts as an integration test for our entire ecosystem
• We need a better name
• Must make use of at least 1 package in the ecosystem