Review

[gilesknap]

A review meeting on 2 Nov 2022 decided on the following changes:

• Remove the current container step from code.yml
• Make the wheel and sdist outside of a container and add them to artefacts
  • this step will use lockfiles if present and create lockfiles for use as release assets
• Have a simple container build step that installs the above wheel and copies it to a runtime stage
  • have this dependent on existence of Dockerfile so that deleting this is the only step required for removing container steps
  • have it create an artefact (tarred image) for potential release in later step
  • test won't need to run in the container
• release step will push the container to GHCR for tagged builds only
• remove .vscode and .devcontainer files EDIT: keep these but minimize them.
• DOCS: recommend use of dev-u22 workspace container for working on skeleton projects in a devcontainer

[gilesknap]

@coretl please review and edit the above. Thanks.

[gilesknap]

Update after later meeting with Martin.

• Make a wheel/sdist step (this can be replaced with a C extensions version just like pythonSoftIOC)
• have no build/dev container - only runtime and make it install the wheel only
• have the test matrix create requirements_dev_py3.XX.txt for each element of the matrix
• still have the container step create requirements.txt

dependencies are

• container relies on wheel/sdist
• release relies on wheel/sdist, lint, test matrix

[gilesknap]

@coretl

still have the container step create requirements.txt

This means there is no requirements.txt for people that remove Dockerfile.

[coretl]

still have the container step create requirements.txt

This means there is no requirements.txt for people that remove Dockerfile.

I think that's fine, the container and requirements.txt are both for deploying applications, if you aren't an application you don't need either

[gilesknap]

As part of this I'm going to look at Garry's approach to use of testpypi in this project https://github.com/DiamondLightSource/numcertain

[gilesknap]

@garryod I note that you are using

• .vscode
• .devcontainer (folder)

In numcertain.

We are considering dropping .vscode and .devcontainer.json from skeleton because we are recommending the use of a workspace project such as https://github.com/epics-containers/dev-u22-workspace for working with python projects.

What is your view? Did these files provide a starting point for what you now have in numcertain? would you prefer to keep them?

[garryod]

still have the container step create requirements.txt

This means there is no requirements.txt for people that remove Dockerfile.

I think that's fine, the container and requirements.txt are both for deploying applications, if you aren't an application you don't need either

The requirements.txt file serves as a useful artifact for debugging failures observed in the regular "checkup" workflows, IMO it would be nice to have even for library code

[garryod]

@garryod I note that you are using

• .vscode
• .devcontainer (folder)

In numcertain.

We are considering dropping .vscode and .devcontainer.json from skeleton because we are recommending the use of a workspace project such as https://github.com/epics-containers/dev-u22-workspace for working with python projects.

What is your view? Did these files provide a starting point for what you now have in numcertain? would you prefer to keep them?

The vast majority of my projects require some form of system dependency - whether it be the CUDA toolkit, some specific flavour of MPI, or some C library - and therefore require their own Dockerfile. A lot of these are at best difficult to get working in tandem and at worst fundamentally incompatible, so a single development container per project has made sense.

Further, it is nice that the contributor instructions can be as simple as "clone this repo and open in a vscode dev container", especially when collaborating with folks at different institutions

[gilesknap]

update: twine check in the dist step instead of using testpypi because: its one less token to generate and gives just as much info.

• this does not check tags which is OK - if we are tagged badly then the actual push to pypi will fail anyway

[coretl]

still have the container step create requirements.txt

This means there is no requirements.txt for people that remove Dockerfile.

I think that's fine, the container and requirements.txt are both for deploying applications, if you aren't an application you don't need either

The requirements.txt file serves as a useful artifact for debugging failures observed in the regular "checkup" workflows, IMO it would be nice to have even for library code

But you'd still get dev_requirements_pyx.x.txt files for each test run and lint, are those sufficient?

[gilesknap]

thanks @garryod

After further discussion we are going to keep .devcontainer and .vscode. We will make the settings minimal for working with the project (e.g. no extensions except pylance).

Then adopters can add to these or delete them as they see fit.

[garryod]

update: twine check in the dist step instead of using testpypi because: its one less token to generate and gives just as much info.

• this does not check tags which is OK - if we are tagged badly then the actual push to pypi will fail anyway

Hadn't realised this was a thing, sounds ideal!

[gilesknap]

@garryod further to the test pypi conversation.

What kind of failures are you looking to catch with this? Thanks.

[garryod]

@garryod further to the test pypi conversation.

What kind of failures are you looking to catch with this? Thanks.

I've previously had it catch duplicate wheels for C extension builds, I would expect it to be a lot more useful when dealing with a large number of related files like with a C extension.

Also, as discussed on #72, twine check is fairly limited in what it verifies (only that stuff will be rendered correctly afaik)