PyPI Readiness [minor]

[ric-evans]

After a year of using SetupShop, we have learned some lessons. This PR address those in preparation for PyPI publication.

TLDR: Running "arbitrary" setup code (SetupShop) at package install is not good practice. Let's move that to the pre-publish step (CI/CD). Meanwhile, let's revisit package requirements. After all, does it matter if my client has thepackage==2.4.1 but my library used thepackage==2.4.2? Currently SetupShop thinks so :-1:.

CI/CD Steps

On every commit

Run Setup Declaration & Generate requirements.txt. Push changes back to branch

On every merge (a commit to master/main)

Run Setup Declaration, Generate requirements.txt, & Semantic Release/Publish to PYPI. Push back to main/master.

Moving Parts & Pieces

Setup Declaration: setup.cfg

PyPA (the setuptools developers) recommend using setup.cfg for setup boilerplate generation (https://setuptools.pypa.io/en/latest/userguide/declarative_config.html). The most important setup.cfg sections for us include [metadata], [options], and [options.package_data]. These are needed for PyPI publication. These sections are nearly 100% generated by CI/CD. How?

I Introduce to you the [wipac:cicd_setup_builder] section:

[wipac:cicd_setup_builder]
pypi_name = wipac-dev-tools
python_min = 3.6
keywords_spaced = python tools utilities

Along with executing setup_builder.py in CI/CD (GH Action), the following sections are produced and prepended to the existing setup.cfg (sections/fields are overridden if they already exist, see in-line comments):

[metadata]  # generated by wipac:cicd_setup_builder
name = wipac-dev-tools
version = attr: wipac_dev_tools.__version__
url = https://github.com/WIPACrepo/wipac-dev-tools
author = WIPAC Developers
author_email = developers@icecube.wisc.edu
description = Common, basic, and reusable development tools
long_description = file: README.md, CHANGELOG.md, LICENSE.md
long_description_content_type = text/markdown
keywords =
    python
    tools
    utilities
    WIPAC
    IceCube
license = MIT
classifiers =
    Development Status :: 5 - Production/Stable
    License :: OSI Approved :: MIT License
    Programming Language :: Python :: 3.6
    Programming Language :: Python :: 3.7
    Programming Language :: Python :: 3.8
    Programming Language :: Python :: 3.9
    Programming Language :: Python :: 3.10

[semantic_release]  # generated by wipac:cicd_setup_builder
version_variable = wipac_dev_tools/__init__.py:__version__
upload_to_pypi = True
patch_without_tag = True
commit_parser = semantic_release.history.tag_parser
minor_tag = [minor]
fix_tag = [fix]
branch = main

[options]  # generated by wipac:cicd_setup_builder: 'python_requires', 'packages'
install_requires =
    requests
python_requires = >=3.6, <3.11
packages = find:

[options.package_data]  # generated by wipac:cicd_setup_builder: '*'
* = py.typed

[semantic_release] is also generated for good measure, this is important for automating PyPI publication.

Requirements and requirements.txt

As a principle, packages should be developed on the cutting edge. That is, use the latest versions for their requirements. See https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/. To accomplish this, requirements will now go in setup.cfg's [options]/install_requires, unpinned (except for rare, known exceptions). setup.cfg is the source of truth for all requirements and requirements.txt is its log/snapshot. requirements.txt will be autogenerated by parsing setup.cfg (using https://github.com/jazzband/pip-tools) in CI/CD (GH Action). pip-tools pins these package requirements. This will guarantee we have an easily accessible log of the exact library versions for each of our packages' versions (reproducibility = :smile:). Since this is done in CI/CD, we'll know if something breaks (ie. a requirement's new version isn't backward compatible); see Future Dependabot.

Example:

#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile
#
certifi==2021.10.8
    # via requests
charset-normalizer==2.0.12
    # via requests
idna==3.3
    # via requests
requests==2.27.1
    # via wipac-dev-tools (setup.py)
urllib3==1.26.8
    # via requests

Tangent: There is also the [options.extras_require] section which will be useful for deploying packages with the WIPAC Telemetry package included/implemented (or excluded).

PyPI Publication

Semantic Release takes care of this for us (see setup.cfg's [semantic_release]/upload_to_pypi and the semantic-release.yaml GH Action).

Future Dependabot

Using Github's Dependabot (https://github.blog/2019-05-23-introducing-new-ways-to-keep-your-code-secure/) will catch any security vulnerability, make a PR with the fixes, and thus trigger the CI/CD above. This will build on / replace @blinkdog's useful check-requirements.py

closes https://github.com/WIPACrepo/wipac-dev-tools/issues/20

Edit: User no longer needs to provide description, url, and branch

[ric-evans]

This is the new setup.py:

from setuptools import setup

setup()

That's it :grinning:

[dsschult]

Some thoughts:

• pypi_name: should we standardize on _ or - as separators?
• description: can this be read from github's project description? would that be a good idea?
• main_or_master: a better name might be branch_name? also, can this be inferred from github?

I guess in general I'm looking at what the defaults should be, or how we can find them automatically.

In general this is looking good though.

[ric-evans]

pypi_name: should we standardize on _ or - as separators?

• Do you mean for the key or actual value?

description: can this be read from github's project description? would that be a good idea?

• I suppose it could. Currently, there's no github connection/parsing.

main_or_master: a better name might be branch_name? also, can this be inferred from github?

• I originally had this as branch_name, but I figured main_or_master gets more to the point that this means "what does your repo call the root branch?" As for inferring from github, see above. This could be done.

I guess in general I'm looking at what the defaults should be, or how we can find them automatically.

• Sure, I'm using this dataclass below. But as for user-friendliness, this could go in a README. Also, maybe in argparse's -h printout?

  
  @dataclass
  class BuilderSection:
  """Encapsulates the `BUIDLER_SECTION_NAME` section & checks for required/invalid fields."""

pypi_name: str
description: str
url: str
python_range: str  # python_requires
main_or_master: str = "main"
keywords_spaced: str = ""  # comes as "A B C"

_Edit: this is outdated--these fields have been updated. see top-most summary_

[ric-evans]

From above, I could see giving python_range a default like >=3.6. That way we don't have to make a conscious decision of whether to bump the package into a new py version

[dsschult]

pypi_name: should we standardize on _ or - as separators?

* Do you mean for the key or actual value?

I mean the actual value.

main_or_master: a better name might be branch_name? also, can this be inferred from github?

* I originally had this as `branch_name`, but I figured `main_or_master` gets more to the point that this means "what does your repo call the root branch?" As for inferring from github, see above. This could be done.

I'm just not sure if github won't change it again. And technically you could pick a different name (I've seen a project that had a different "main" branch).

[dsschult]

Other things to think about:

• consider putting the action in a separate repo, and just reference it from there for each project
• consider documentation publishing (this might be another PR?)

[ric-evans]

pypi_name: should we standardize on _ or - as separators?

* Do you mean for the key or actual value?

I mean the actual value.

• Okay, I think dashes.

main_or_master: a better name might be branch_name? also, can this be inferred from github?

* I originally had this as `branch_name`, but I figured `main_or_master` gets more to the point that this means "what does your repo call the root branch?" As for inferring from github, see above. This could be done.

I'm just not sure if github won't change it again. And technically you could pick a different name (I've seen a project that had a different "main" branch).

• Good point. I can go with branch_name. root_branch is also an option, but perhaps that's vague?

[ric-evans]

consider putting the action in a separate repo, and just reference it from there for each project

• I'm not sure I follow. Do you mean a "symlink" to the separate repo? consider documentation publishing (this might be another PR?)
• This will grab README, CHANGELOG, and LICENSE files' content to show on the PyPI package page. That's some documentation, if you mean a gh-pages/readthedocs sort of thing, that's out of scope (another PR).

[dsschult]

consider putting the action in a separate repo, and just reference it from there for each project

* I'm not sure I follow. Do you mean a "symlink" to the separate repo?

I mean giving it the full treatment for creating an action: https://docs.github.com/en/actions/creating-actions/about-custom-actions though it may not be necessary to do all of that for a private action.

I'm just thinking of the uses syntax in a step, so we can reference it from any repo:

uses: WIPACrepo/wipac-dev-tools@v1.0.0 

but I think you can only reference a single action per repo that way.

consider documentation publishing (this might be another PR?)

* This will grab `README`, `CHANGELOG`, and `LICENSE` files' content to show on the PyPI package page. That's some documentation, if you mean a `gh-pages`/readthedocs sort of thing, that's out of scope (another PR).

Yes, I meant something like readthedocs, just with a centralized action to do all the heavy lifting.

[ric-evans]

The only thing left to test is publishing to PyPI (have to wait and see at merge) and the "future dependabot" (which can wait for now)

[ric-evans]

Here's an example of CI/CD-driven commits (see history above)

[dsschult]

Let's see if it works. Note that as with Dockerhub, you may need to create the project in PyPI first before publishing.

[ric-evans]

Let's see if it works. Note that as with Dockerhub, you may need to create the project in PyPI first before publishing.

It worked, no pre-creation needed!