Discuss possible collaborations / integrations with Pangeo

[rabernat]

Thanks for all of your amazing work on pyOpenSci. It's great to see the progress this project has made.

I'm opening this issue to discuss how we (in the Pangeo project) can leverage and collaborate with pyOpenSci. Pangeo is a platform for big data geoscience that is really just a loose collection of interoperable python packages. We have been providing guidance on our website about best practices for Pangeo-related software packages, but we don't really have a well-maintained list of the "official" Pangeo packages. In this respect, we lag far behind other coordinated efforts like Fatiando a Terra, which presents a clear and coherent list of interrelated packages to its users.

Going forward, we would love to be able to leverage the excellent pyOpenSci package standards to provide a stamp of approval to Pangeo-related packages, thereby providing clear guidance to potential package authors about how to reach this stage.

The one complication preventing us from doing this now is that we would probably want to add some additional requirements, beyond your standards (which are appropriately broad). Our additional requirements would be something like:

• Consume and produce high-level data structures (e.g. xarray datasets / pandas dataframes) wherever feasible
• Operate lazily when called on dask data structure
• Avoid file I/O unless specifically an I/O package

These requirements are pretty central to the pangeo philosopy and enable our stack to scale out on HPC and cloud.

Is there scope within the pyOpenSci effort to attach domain-specific expectations to the review process? I imagine similar issues may arise in other fields, such as neuroscience or astronomy, where there are existing best practices.

Thanks for considering my comment, and thanks again for your work on this important effort.

[xmnlab]

@rabernat your idea about a common guideline across scientific python communities sounds very nice!

maybe we could add specific sections for specific communities when needed, how does it sound? for example, we could have a topic called "coding guide" and a section there for Pangeo.

Does anyone have more comments about that?

[rabernat]

maybe we could add specific sections for specific communities when needed, how does it sound?

Yes, that's what I had in mind. Thanks for considering it.

[lwasser]

good. morning @xmnlab @rabernat i second Ivan's idea about having community specific guidelines. The one challenge will be to have a vision for it to ensure that we don't end up with too many specific subsets of guidelines that will in turn make reviews and being an editor much more challenging. Maybe there is a process for

1. defining what a community is and the scale of the community may impact how we consider new guidelines
2. Looking at our existing guidelines to see if some of what is proposed by said community should be universally enforced,
3. Determining a way to ensure reviewers know what criteria they need to follow.

it's worth chatting more about this because i could see if becoming quite complicated to manage if it's not implemented with care. At the same time i can see some supsets - pangeo, astropy, etc may have different needs and requirements.

Just a note Ryan that right now we have slowed down effort on pyopensci mostly because of funding challenges and then the extra workload that i have given COVID challenges. We do want to find a way to fund this project so that we can continue to support reviews and such so we are also open to any thoughts about that as well!

[lwasser]

we have this underway actively. so i am gonig to close this issue for now given the work that we are doing. we will be adding this collab to our peer review guide in the upcoming weeks!