Project scoping for ROSES call

[wtbarnes]

This issue to keep track of potential long term projects for the ROSES call E.7 Support for Open Source Tools, Frameworks, and Libraries.

---

Notice of Intent (optional) due 2020 Nov 19 Proposal due 2021 Jan 19 Maximum duration of 3 years Yearly funding of ~$150–300k

---

The purpose of this issue is to both organize our thoughts but also to solicit community feedback as to missing gaps in functionality in the SunPy ecosystem

[wtbarnes]

• Emphasize that these duties are not constrained to the core code base
• This role is largely focused on the project as a whole--make sure to emphasize this in the proposal

A List of things to do right now!

• Daily issue triage
  • Back catalogue
  • As they come in
• Mailing list/Discourse maintenance
• Dev-ops automation infrastructure
  • Package template
  • CI monitoring
  • Make this less human in the loop!
• Development work
  • Adopting stale PRs
  • Less development in the first year, more in later years
  • Big feature requests from the community!
  • Offload science features to community and review
  • Non-science features that are important to the package and useful, but less likely to be implemented by the community
• Fill open roles
• Organize coordination meeting
• Affiliated package
  • Review / Re-review procedures
  • Webpage
  • Recruitment -- interfacing with instrument teams (instrument teams should front some cash)
  • Updates for breaking changes in sunpy/other dependencies (e.g. Fido in radiospectra)
• Interfacing with upstream changes
  • Need to react quickly to these changes! These are the most time-sensitive
  • How do we monitor these changes?
  • Data providers
    • e.g. GOES XRS
  • Packages
  • CI
• Mentoring
  • Filling open roles
  • Educating community
  • Onboarding contributors

Reorganize

• Large future project design Scope out and describe on issues project plans for large features i.e. metadata rewrite. To facillitate contributions or funding from other places etc. Also have discussions about these things before the dev work happens.

• Day-to-day Duties

  • Issue triage
  • PR review
  • CI monitoring
  • Mailing List / Chat support
  • Helping with releases / doing bugfixes

• Technical infrastructure

  • Development of Project Automation
    • Package template & auto PR bot
    • CI dashboard
    • Release reminder bot
    • Completely autonomus bugfixes
    • Document the shit out of all of this for benefit of wider community
  • CI maintenance
  • Sponsored package template / updates

• Project infrastructure

  • Milestone policies
  • Drafting SEPs
  • Organize weekly meetings
  • Organize coordination meetings

• Interfacing with and reacting to upstream changes

  • Need to react quickly to these changes! These are the most time-sensitive
  • How do we monitor these changes?
  • Data providers
    • e.g. GOES XRS
  • Packages
  • CI

• Community Interaction

  • What features do we need in the project?
  • Fill open roles
  • Affiliated package
    • Review / Re-review procedures
    • Webpage
    • Recruitment -- interfacing with instrument teams (instrument teams should front some cash)
    • Updates for breaking changes in sunpy/other dependencies (e.g. Fido in radiospectra)

[kreardon]

I think all the infrastructure and foundational ideas that Will lays out are great! We definitely need to indicate the need for such efforts to put the project on a firm footing for future developments. Having this kind of role funded and organized in the project has immense value that I think we can well demonstrate in the proposal.

Since this is will be a proposal for three years of development effort, I think it should also have some forward-looking ideas, even if they might seem a stretch now (or we only envision building them out partially). We will still have the flexibility to adapt our development as things change over the project period, but we need some general roadmap to work from. I think having some development plans that also tie back to some specific science topics would be valuable in the review phase.

So I'll spitball some ideas for some higher-level things (and I suspect I'm up at 3000 meters here) to possibly include in the proposal:

1. ndcube 2 development - I this can be connected to the growing importance - in radio, optical , UV, etc. - of spectral information in our scientific analysis.
2. what is needed to (better) support combining PSP/Solar Orbiter/L1/Earth observations? We're doing pretty good with the coordinates thing :) but there might be needs around tracing along magnetic fields (e.g. from PSP to surface) or taking into account (variable) transit times along those field lines. This is a good science-driven need and ties into some of the broader HSO Connect efforts in ROSES.
3. metadata/visualization/mapping support for simulations/models (different grids, volumes, convolutions)? Maybe much of this infrastructure already exists? I'm in particular thinking of 3D MHD simulations of the solar atmosphere, but I'm sure there are other applications.
4. Large volumes (e.g. SDO, DKIST) data access? Or more generally how to support future "science gateway/distributed resource/virtual server" kind of analysis approaches.

Would some of these capabilities be better developed and incorporated as affiliated packages? Maybe, but I think the NASA call allows for development efforts that aren't necessarily part of the core.

[Cadair]

ping @ayshih

[ayshih]

2. what is needed to (better) support combining PSP/Solar Orbiter/L1/Earth observations?

I like this line of thinking a lot. It screams "why now?".

Some additional notions:

• Training and/or workshops. They're even mentioned in the solicitation.
• Support for spectroscopic analysis, including up to the X-ray/gamma-ray range (leveraging PyXspec, Gammapy, Sherpa, ...?)
• Support for an event list data type, for instruments that record individual photons (leveraging Stingray or Gammapy?)
• Support for a visibility data type for radio observatories or for high-energy imagers like RHESSI and STIX

[kreardon]

I like the support for spectroscopic analysis! Could this tie in to the development of ndcube 2?

Could the event lists include "solar" events, like active regions, filaments, flares, etc.? They are not nearly as discrete/distinct as the photon events you were referencing, but maybe there is overlap?

Training and/or workshops are valuable - need to think about how to organize those over the three-year lifetime. Start general, and develop more targeted workshops about particular workflows later on?

[ehsteve]

FYI, sunxspec has won funding so need to coordinate with that effort ping @DanRyanIrish @hayesla.

[kreardon]

We will have a telecon about this on Tuesday, November 10 at noon EST. We can send around a link to anyone who is interested in joining to hash out ideas.

[mbobra]

We will have another telecon, to write the optional NOI, on Tuesday November 17 at noon EST. Again, we can send a link around to anyone who'd like to join.

[Cadair]

We got the funding! 🤑🥳🥳

[rhewett]

Congrats to the project team!

On Fri, Sep 17, 2021, 6:26 PM Stuart Mumford @.***> wrote:

Closed #9 https://github.com/sunpy/sunpy-project/issues/9.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sunpy/sunpy-project/issues/9#event-5321831275, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQNT2NOXYU5H433IXNV6H3UCO6DBANCNFSM4TCUKOIQ .