[RFC] Desired elements of a package template for python-only packages.

[eacharles]

Summary

It would be good to have a software package template for python only packages. Let's figure out what we want.

Description

It looks like we are going to try and resurrect the python_package template. That's great. We should probably discuss what we want.

FWIW, here are the things that I included in qp and tables_io

• directory with python code
• docs directory with stuff needed for readthedocs
• nb directory with notebooks
• tests directory with unit tests
• .github directory with github actions
  • CI testing to require
  • basic linting
  • unit tests
  • deploy to pypi on release (i.e., when you make a new release on github)
• at the top level
  • .coveragerc # useful
  • .git_archival.txt # gets version number into git tarballs
  • .gitattributes # calls out .git_archival.txt
  • .gitignore # (what do we include?)
  • .pylintrc # (I prefer pylint, others will prefer other tools, I also prefer to tell pylint not to be quite so rigid)
  • DOCKERFILE # Needed?
  • LICENSE
  • README.md
  • requirements.txt
  • setup.py
  • I also include a script to run the unit tests with coverage (so I don't have to remember the commands)
  • I also include a script to render the notebooks (to make it easy to be sure they work)

Let's collect suggestions for a while, then start merging them, and review this at an upcoming desc-co meeting.

[heather999]

How about we set a timeline of one week for comments - so we force some discussion sooner rather than later. We'll have to come up with a way to indicate this somehow... it was on the template originally, maybe it could be added back in along with the short checklist Katrin suggested in response to the suggestion we have a light process so people understand how this should work.

and now some comments: I want to make sure that the template is bare enough that we can say the content will be 100% useful and not create files that may not be used and cause confusion later - while still being complete enough to be helpful.

• It's in that light, that I wonder about whether to include readthedocs stuff. While we suggest the use of docstrings, we haven't prescribed the use of RTD. We could try encouraging it by including the necessary bits in this template. I'm very curious to hear other comments about this.
• I don't think we need to generically provide a Dockerfile.
• For linting, pylint is fine but I wonder if we should also include flake8 as that is what DM uses?
  • Setting the linting to be less restrictive is really necessary - otherwise you risk overwhelming developers - esp newer ones.
  • Maybe add pyproject.toml now so we can start migrating to it?

[eacharles]

FWIW, I agree with all of the above.

[joezuntz]

You don't usually want to put Dockerfiles in the main directory since then their build context includes everything below them. Not sure if they're needed at all here, since putting DESC tools into the build env is a central job?

For the github actions workflow - it would be nice if we could test software against the upcoming official DESC analysis environment (where relevant), so maybe the template workflow yml could do that? It does produce a bit of a chicken-and-egg problem when the package is one already included in that environment - would that mean the first step would have to be removing the old version of the package? Or can we let pip handle replacing it?

[brianv0]

Not that it’s necessarily relevant, but a .dockerignore file is the way around that, and a best practice.

Dockerfiles are nice both for testing and CI, since they completely describe how to bootstrap an environment usually, but of course our environments may be large so using them may be slow (or not, depending on the image they depend on).

More generally, I think we can probably get a cookiecutter repo up for this RFC. It would be nice to have things like black, mypy, and precommit hooks optional in the process, if possible.

[heather999]

We are hoping to use the existing cookiecutter repo that Francois Lanusse initially set up: https://github.com/LSSTDESC/python_package_template. Francois is aware of this and we should at least to take a look. We could also choose to make a whole new repo. If we can make things optional, that would open up a lot of possibilities so that those that are ready to take advantage of some features can switch them on - while also providing something simple as the default.

In the context of an individual package template we would want to be clear what the Dockerfile is for if we were to include it. Is it for testing or building an env for distribution via docker? I'm imagining full CI testing within the context of the DESC ReleaseManager will happen as part of the ReleaseManager and not necessarily handled at the individual package level - though of course some packages developers may want to do some level of testing of their package either on their own or against the full DESC environment. We may want to encourage that. We'd want to put some thought into setting up that environment but maybe providing the desc-python-latest (once that's going) could be a start. There may be some higher priority things such as getting that python package set up for PyPI and providing adequate tests that can be run during any testing.

[eacharles]

It is possible and relatively easy to make things optional.

-e

On Sep 24, 2021, at 11:36 AM, Heather Kelly @.***> wrote:

We are hoping to use the existing cookiecutter repo that Francois Lanusse initially set up: https://github.com/LSSTDESC/python_package_template https://github.com/LSSTDESC/python_package_template. Francois is aware of this and we should at least to take a look. We could also choose to make a whole new repo. If we can make things optional, that would open up a lot of possibilities so that those that are ready to take advantage of some features can switch them on - while also providing something simple as the default.

In the context of an individual package template we would want to be clear what the Dockerfile is for if we were to include it. Is it for testing or building an env for distribution via docker? I'm imagining full CI testing within the context of the DESC ReleaseManager will happen as part of the ReleaseManager and not necessarily handled at the individual package level - though of course some packages developers may want to do some level of testing of their package either on their own or against the full DESC environment. We may want to encourage that. We'd want to put some thought into setting up that environment but maybe providing the desc-python-latest (once that's going) could be a start. There may be some higher priority things such as getting that python package set up for PyPI and providing adequate tests that can be run during any testing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/RequestForComments/issues/8#issuecomment-926842172, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRIGIX7DPGMLL5N5BU2FJ3UDTAJDANCNFSM5EUDKLOA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[heather999]

At this week's CO meeting we discussed how to move forward on this.

@joezuntz there was a desire to hear about your experience with https://github.com/LSSTDESC/pipeline-package-template. Were developers interested in using it? If not, what were the challenges?

To move forward and to keep the template as minimal as possible, we'd like to frame this in the context of release management and define what we expect for packages that are to be added to the DESC Computing Environment. Here's a very minimal laundry list:

• Tagged Release
• README, including doc on how to install the package which could be as simple as pip install <package>
• Availability on PyPI (or conda-forge), preferably automated upload via GH actions
• unit testing via pytest

If we could provide a template or even just a sample repo that developers could copy, I think that would go a long way to getting things going. In fact, I almost wonder if we should have a skeleton example repo rather than a cookie-cutter.

[heather999]

During Sprint Week, @eacharles and I had some time to discuss how we would like to proceed. Here is the preliminary proposal which is open to comments before we go ahead and mark this adopted and set up some issues to track the work.

• A new repo, potentially named desc-python-package-utils, will be set up, without cookie-cutter. The repo will contain useful bits that DESC developers can pick and choose to copy into their own repos. This will hopefully make it easier to maintain and extend - as well as lowering the bar for DESC developers who may just want a simple example for a subset of tasks to start from.
• We will focus on implementing basic examples that we really want to encourage for Release Management.
  • push to PyPI
  • Unit testing
  • linting
  • coverage
• There will be a script developed to handle name mangling so that any examples/GitHub actions will be updated with the appropriate package name and other specific details.
• Documentation for desc-python-package-utils will start with a simple README, but ultimately we anticipate needing something like GH pages to set up richer documentation.

Comments welcome - including from Eric who should feel free to update any mistaken info above!