Upgrade/Installation

[eboileau]

While changing to CmdStanPy, I upgraded all dependencies/packages (most have "unpinned" versions, however Conda does not necessarily resolve the latest version, we can eventually pin important packages). Now we need to finalise the upgrade and the installation (for both rpbp and pbiotools). I have only worked out the Conda installation (with pip install for pbiotools and rp-bp) as follows:

# clone https://github.com/dieterich-lab/rp-bp.git and https://github.com/dieterich-lab/pbiotools.git
# checkout dev-ssciwr
# install environment under rp-bp
conda env create --prefix /prj/rpbp-dev/bullseye/envs/rpbp-cmdstan -f environment.yml
# then I install pbiotools, followed by rp-bp using each time (editable mode)
pip --verbose install -e . 

Now I have many questions...

• [X] I am confused now why we mix Conda and pip... ideally, we want ONE Conda install (environment with dependencies + rpbp/pbiotools, and pip only for packages/version not on Conda). We still put rp-bp and pbiotools on PyPI, but in that case users will have to handle dependencies by themselves (Flexbar, STAR, CmdStan, etc. ). When this is sorted, we can also create containers (Docker/Singularity) for the full Conda install.

• [x] Under rp-bp I ran pytest tests/ successfully, but latest commit doesn't work, same problem with pbiotools. Something is wrong with the python version(s), and with pip installation...

• [X] Why do we need a requirement.txt file (in addition to environment.yml and install_require in setup.cfg)? We need to clean this.

• [X] Do we actually need to distribute compiled Stan models with Conda (maybe this is not critical, compilation is now much faster than it was)?

• [X] With the latest Conda install, samtools seem to work fine, but I get

samtools: /beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/bin/../lib/libtinfow.so.6: no version information available (required by samtools)
samtools: /beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/bin/../lib/libncursesw.so.6: no version information available (required by samtools)
samtools: /beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/bin/../lib/libncursesw.so.6: no version information available (required by samtools)

Listed in #133

[lkeegan]

Under rp-bp I ran pytest tests/ successfully, but latest commit doesn't work, same problem with pbiotools. Something is wrong with the python version(s), and with pip installation...

• I added a couple suggestions for the failing CI:
• https://github.com/dieterich-lab/rp-bp/commit/800108e9c4b0148a316a170a313e67de1c6aa7f2#r89387115
• https://github.com/dieterich-lab/pbiotools/commit/713ad8bf3b1dc44f3f656df2b5dd5e32ac6630f1#r89386793

I am confused now why we mix Conda and pip... ideally, we want ONE Conda install (environment with dependencies + rpbp/pbiotools, and pip only for packages/version not on Conda). We still put rp-bp and pbiotools on PyPI, but in that case users will have to handle dependencies by themselves (Flexbar, STAR, CmdStan, etc. ). When this is sorted, we can also create containers (Docker/Singularity) for the full Conda install.

I agree - and everything is now on bioconda so there can be a pure conda install. There are also already docker/singularity containers for pbiotools from bioconda (see https://github.com/dieterich-lab/pbiotools/pull/9), once rp-bp is packaged in the same way on bioconda there would also be containers available for that.

[eboileau]

I just removed the pins, except for pysam. The latest version on PyPI is 0.20.0 (28 Oct), but hasn't been updated on bioconda yet.

I need to clean the requirement/environment list, in particular if we do some more cleaning (remove unused scripts, etc.)

[eboileau]

Pysam 0.20.0 is now on Conda. The environment file has been updated.