feat: Add Python 3.11 to the build matrix

[jmarshall]

I have no idea what all is involved in updating this and perhaps doing initial package builds on the bulk branch, but this PR provides a starting point for this long-overdue update. Python 3.12 was released almost two months ago, so it is embarrassing that bioconda is still not building packages for the previous year's release, Python 3.11.

This is tracked as bioconda/bioconda-recipes#37805. That links to conversation on #878 which indicates that bioconda has been waiting for conda-forge to include 3.11. It appears that conda-forge's migration was completed in September.

[hasindu2008]

I hope bioconda soon supports Python 3.11. I have got a request to get pyslow5 for Python 3.11 on conda https://github.com/hasindu2008/slow5lib/issues/83.

[jmarshall]

@bioconda/core: Please comment as to whether bioconda intends to support Python 3.11.

[bgruening]

Hi John. Yes, we intend to support Python 3.11 and 3.12. However, we are currently busy rolling out the ARM builds and in parallel the BioConductor release. This keeps us and the CI already busy. Realistically we can only care about 3.11/12 in January. Sorry.

[jmarshall]

@bgruening: Thanks for the information. I suggest you post an update on bioconda/bioconda-recipes#37805, bioconda/bioconda-recipes#33333, and/or bioconda/bioconda-recipes#37068. I ask not for myself, but for the numerous bioconda users regularly asking myself and other maintainers of python-version-specific packages why our packages are not available on Python 3.11.

In general, the lack of information from the core cabal is disappointing. When bioconda/bioconda-recipes#33333 and bioconda/bioconda-recipes#37068 were pinned, I was hopeful that they would become useful conduits, but they have largely not been used.

[bgruening]

I agree, we need to improve our communications. Maybe someone wants to join us and help communicate what we are currently doing.

[corneliusroemer]

I'd be happy to help a little with communication

[corneliusroemer]

@bgruening gentle ping to ask about status of bioconductor release, ARM feature and Python 3.11/3.12 support. @jmarshall mentioned that bioconductor release seems to be complete.

Your past updates were very helpful, I'm hoping you can give another short summary of where things stand so the community at large is somewhat in the loop.

[daler]

@bgruening can chime in if there's anything I'm missing here, but here's a snapshot of the current state and short-term goals:

Current goal is to get everything cleaned up and in shape to support large-scale updating for ARM and Python 3.11 specifically regarding the build-failure yamls and their interaction with the bulk and master branches. For example, right now removing a build-failure yaml is insufficient to trigger a rebuild; we need to work out the right process for having many people work on build failures simultaneously. This requires the technical component (actually writing the infrastructure code) but also the sociological component (how, specifically, do we best coordinate across contributors without stepping on each others' toes)? This is important for both ARM and Python 3.11 and of course any future migrations/updates. The “practice run” for this, once the infra is in place, will be finalizing the remaining Bioconductor packages.

There are a last few remaining details to be taken care of with ARM containers to avoid arm64 clobbering amd6. This involves quay.io, docker manifests, and making sure all the infrastructure is updated to handle all of that. Then we’ll kick off some opt-in ARM builds to make sure everything works in practice.

Then it’s on to Python 3.11. Hopefully the infrastructure will be in place there for rapid building on bulk, kicking out build failures to the wiki, and coordinating with the community to fix everything. My dream is for the 3.11 to go quickly and smoothly. Ideally, 3.12 could follow shortly after. And by then hopefully we'll have worked out the kinks on ARM, and can scale up those builds as well.

The challenging part in all of this is that to make meaningful progress on infrastructure components like ARM containers and the build-failure process, we need to load everything into mental RAM. That is difficult to do when we all have day jobs, many of us are trying run our respective labs and working on this in our (increasingly rare) spare time. So then documentation becomes a vital part of all of this to minimize the mental-RAM-loading time (e.g. https://github.com/bioconda/bioconda-docs/pull/16/files). The docs are certainly helping. Ideally, we would document everything extensively enough so the community can jump in and help out and alleviate the bandwidth issue. We’re not there yet,but we're getting there.

We of course also need to be better about the decision-making process, planning process, and documenting all of the various processes and moving parts to make this all more transparent to the community. Developing that documentation and communication takes away from making progress on working on infrastructure, so it's a balance.

None of this helps directly with Python 3.11, I know. Just trying to give some insight into how we're trying to improve all the internal processes and make it better for everyone over the long term. Rest assured that 3.11 is definitely in the works.

[jmarshall]

@daler: Thanks for your comments. Posting them here does not really reach the audience who needs to hear them; wider communication channels might be gitter or the pinned announcement and roadmap issues.

Past Python migrations have been done without the benefit of the build-failure infrastructure. So — volunteer time issues aside — there does not seem to be a technical reason to further delay this overdue migration.

[daler]

@jmarshall agreed on both counts.

The current mechanism, where Py 3.11 builds run on bulk branch in topological sort across workers, requires heavy time investment by a single person over a couple of weeks. This is how all previous migrations have been done. Nobody currently (or regularly) has bandwidth to manage that nowadays, hence the refactor for better scaling. So not a technical limitation, but rather a "personnel" limitation.

[corneliusroemer]

Now that aarch64 is in the works (big congrats 🎉 ), what's the timeline for progress on Python 3.11

[mbargull]

what's the timeline for progress on Python 3.11

I'm meant to start this a few weeks back actually, but got sick (multiple times :facepalm:). This is high on my prio list and should start next week. (I'll also re-evaluate the status of 3.12's migration on conda-forge and may do 3.12 at the same or shortly after in case nothing major is missing for us.)

[daler]

thanks @mbargull, hope you're feeling better soon. @corneliusroemer as mentioned previously this sort of updating is currently personnel-intensive, and Marcel will be the one handling it this round. We have some pieces in place to better democratize the process and speed up future migrations which we will be testing here.

[corneliusroemer]

Excellent news, thank you both for the update!