cjw85
commented
1 year ago Would it be possible to relax the version requirements to allow tensorflow>=2.8.0 ?
In short no. We only test with tensorflow 2.8.* and over the years we've have numerous issues reported when users have gone off-piste with the tensorflow versions; tensorflow hasn't had the best record of maintaining stable APIs through minor version numbers.
The current version of tensorflow we specify derives from the matrix of precompiled wheels we wish to build and tensorflow package support in PyPI and conda.
I wish you luck.
tardigradus
Problem
I am trying to install version 1.8.0 on an HPC system running Linux using EasyBuild. This is a build framework which allows one to build from source and install multiple versions of the same piece of software on a single system and which manages dependencies between pieces of software in a reproducible manner.
Although EasyBuild makes it much easier to handle the complexity of installing software on an HPC system, one limitation is that arbitrary combinations of say, a Python version and a particular version of a Python module cannot be easily realised within the framework. Thus, the "recipe" (known as an
EasyConfig) for building, say, Python 3.9.6, automatically installscffi. There is a separate corresponding EasyConfig forSciPycontainingnumpy, but the versions of both these modules are not sufficient formedaka 1.8.0:If, on the other hand, I use the EasyConfig for Python 3.10.4, this contains
cffi 1.15.0and the correspondingSciPyEasyConfig containsnumpy 1.22.3, so the requirements are met. However, this version of Python corresponds to a specific version ofTensorFlowand I then get the errorProposed solution
Would it be possible to relax the version requirements to allow
tensorflow>=2.8.0?Due to the complexity of TensorFlow I can see that one might want to limit support to one version modulo patches, but is there a technical necessity for this?
Alternatives
Stop using EasyBuild
This is not really feasible. EasyBuild allows us as a small team to provide a much (I would guess, at least one order of magnitude) larger amount of software for our users than we could otherwise.
Use a different build framework
There are alternatives to EasyBuild, such as Spack, which probably allows for more flexibility, but it is aimed at a slightly different user-base. Apart from that, we are already fairly heavily invested in EasyBuild.
Add all the missing EasyConfigs to EasyBuild
It would in principle be possible to create all the missing EasyConfigs with exactly the versions required, but this would be quite a lot of work and would probably generate quite a lot of redundancy by producing almost identical EasyConfigs which differ in, say, just a few Python modules. Moreover, this would potentially have to be done for each release of Medaka.
Just don't install Medaka centrally and let users install in their home directories
This can be challenging, e.g. because the dependencies cannot be installed by non-admin users via the package manager (which wouldn't help any way, because users are in general on a single login node and this would also have to happen on hundreds of compute nodes). Moreover, some of the potential Medaka users do not have much of an IT, let alone a Linux or even HPC, background).
Additional context
EasyBuild does already provide an EasyConfig for
medaka 1.6.0and based on that I was trying to write one formedaka 1.8.0. This could then be used by any other HPC site which is running EasyBuild. HPC is quite a niche, but it is one that allows Medaka users access to considerable computing resources. If we can't install via EasyBuild, we won't be able to install at all, unless with subsequent the versions of the dependencies in both EasyBuild and in Medaka happen to line up, as they did for 1.6.0.