minimap2 with tiledbvcf

[lynnjo]

I am trying to setup an environment which includes both minimap2 and the latest tiledbvcf-py versions. Conda/mamba is not able to set this up due to incompatibility. Minimap2 and older versions of tiledbvcf-py that use python 3.8 work, but versions that use 3.10 do not. I asked on the minimap2 site and they believe it is a bioconda issue. Response from Heng Li of minimap2:

"Minimap2 doesn't depend on python. Only mappy does but it works with all python3 versions (and even python2.7). As such, this is not a minimap2 question. I don't know bioconda good enough to answer. You need to ask bioconda developers."

My yml file looks like this: name: phgv2-conda channels:

• conda-forge
• bioconda
• tiledb dependencies:
• python=3.10
• tiledb-py=0.32.0
• tiledbvcf-py=0.35.0
• anchorwave=1.2.3
• bcftools
• samtools
• agc=3.1

• minimap2=2.28

  When running this, the attached error message is given indicating. an error with the libraries. If I remove the version on minimap, the environment builds however minimap2 is not usable as it shows the build as "0".

Suggestions? - Thanks, Lynn CondaMinimap2Tiledbvcf_Error.pdf

[lynnjo]

Online version of error - might be easier to see: CondaMinimap2Tiledbvcf_Error.pdf

More information: If I only have python 3.10 and minimap2, there is no problem. The problem is adding the newer versions of tiledbvcf-py and tiledb-py, which (as you can see from the error I posted) require libzlib >-1.3.1.

Because the minimap2 is installed with k8, and that has a libzlib requirement >-1.2.13 and < 1.3, this cannot be loaded. If this a problem with using k8? How can we solve this?

[martin-g]

Could you please paste the text here instead of attaching it as PDF file ? It is plain text! There is no need to create binaries for it (PDFs, images, etc.).

[lynnjo]

ok - I pasted it as it was a screen shot. Here it is:

LibMambaUnsatisfiableError: Encountered problems while solving:

• package libtiledbvcf-0.35.0-hfffce77_0 requires tiledb >=2.26.0,<2.27.0a0, but none of the providers can be installed

Could not solve for environment specs The following packages are incompatible ├─ minimap2 2.28* is installable with the potential options │ ├─ minimap2 2.28 would require │ │ └─ libzlib >=1.2.13,<1.3.0a0 , which can be installed; │ └─ minimap2 2.28 would require │ └─ k8 with the potential options │ ├─ k8 0.2.5 would require │ │ └─ zlib >=1.2.11,<1.3.0a0 , which can be installed; │ ├─ k8 0.2.5 would require │ │ └─ zlib >=1.2.13,<1.3.0a0 , which can be installed; │ ├─ k8 0.2.5 would require │ │ └─ libzlib >=1.2.13,<1.3.0a0 , which can be installed; │ └─ k8 [1.0|1.1|1.2] would require │ └─ python_abi 3.8. _cp38 with the potential options │ ├─ python_abi 3.8 would require │ │ └─ python 3.8. , which can be installed; │ └─ python_abi 3.8, which can be installed; ├─ python 3.10 is uninstallable because there are no viable options │ ├─ python [3.10.0|3.10.10|...|3.10.9] conflicts with any installable versions previously reported; │ ├─ python [3.10.0|3.10.1|...|3.10.9] would require │ │ └─ python_abi 3.10. _cp310, which conflicts with any installable versions previously reported; │ └─ python 3.10.15 would require │ ├─ libsqlite >=3.46.1,<4.0a0 , which requires │ │ └─ libzlib >=1.3.1,<2.0a0 but there are no viable options │ │ ├─ libzlib 1.3.1 would require │ │ │ └─ zlib 1.3.1 _1, which conflicts with any installable versions previously reported; │ │ └─ libzlib 1.3.1 would require │ │ └─ zlib 1.3.1 _0, which conflicts with any installable versions previously reported; │ ├─ libzlib >=1.3.1,<2.0a0 , which cannot be installed (as previously explained); │ └─ python_abi 3.10. _cp310, which conflicts with any installable versions previously reported; └─ tiledbvcf-py 0.35.0 is uninstallable because it requires └─ libtiledbvcf 0.35.0 hfffce77_0, which requires └─ tiledb >=2.26.0,<2.27.0a0 , which requires └─ libzlib >=1.3.1,<2.0a0 , which cannot be installed (as previously explained).

[martin-g]

The problem is in k8. minimap2 depends on k8 and k8 needs different versions of v8/Node.js for Linux and OSX, and this causes all kind of troubles ... More info at https://github.com/bioconda/bioconda-recipes/pull/49946#issuecomment-2278466864 and below

[lynnjo]

I read through the comment you list above, and followed the links in that comment. I'm not clear if this is something fixable, if there is something I can do, or if support for Mac ARM has made this a problem not currently solvable. Do you have any suggestions? If it matters, we're currently trying to get this to run on non-ARM linux servers, though we would eventually want to support MAC and ARM. Thanks

[martin-g]

@lynnjo Please paste complete logs with the issued command and resolved versions, etc. It is hard to guess what OS and other details from the error above.

It might be a different problem. Let's ping @attractivechaos @peterjc

[lynnjo]

@martin-g The error above and yml file were initially run on a MAC - macOS Monterey, version 12.6.2. This is INTEL processor: 3.6 GHz 8-Core Intel Core i9

I just tried this on one of our linux machines - same yml file, here are the Linux system specifics:

NAME="Rocky Linux" VERSION="9.0 (Blue Onyx)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="9.0" PLATFORM_ID="platform:el9"

here is the command and results:

[lcj34@cbsubl01 phgEnvironment_minimap2Tiledb]$ conda env create --solver libmamba -f phg_environmentMinimap2Tiledb.yml Retrieving notices: ...working... done Channels:

• conda-forge
• bioconda
• tiledb
• defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: failed Channels:
• conda-forge
• bioconda
• tiledb
• defaults Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:

• package tiledbvcf-py-0.35.0-py38h0d54072_0 requires python >=3.8,<3.9.0a0, but none of the providers can be installed

Could not solve for environment specs The following packages are incompatible ├─ minimap2 2.28* is installable with the potential options │ ├─ minimap2 2.28 would require │ │ └─ libzlib >=1.2.13,<1.3.0a0 , which can be installed; │ └─ minimap2 2.28 would require │ └─ k8 with the potential options │ ├─ k8 0.2.5 would require │ │ └─ zlib >=1.2.11,<1.3.0a0 , which can be installed; │ ├─ k8 0.2.5 would require │ │ └─ zlib >=1.2.13,<1.3.0a0 , which can be installed; │ ├─ k8 0.2.5 would require │ │ └─ libzlib >=1.2.13,<1.3.0a0 , which can be installed; │ └─ k8 [1.0|1.2] would require │ └─ python_abi 3.8. _cp38 with the potential options │ ├─ python_abi 3.8 would require │ │ └─ python 3.8. , which can be installed; │ └─ python_abi 3.8, which can be installed; ├─ python 3.10 is not installable because there are no viable options │ ├─ python [3.10.0|3.10.10|...|3.10.9] conflicts with any installable versions previously reported; │ ├─ python [3.10.0|3.10.1|...|3.10.9] would require │ │ └─ python_abi 3.10. _cp310, which conflicts with any installable versions previously reported; │ ├─ python 3.10.15 would require │ │ ├─ libzlib >=1.3.1,<2.0a0 but there are no viable options │ │ │ ├─ libzlib 1.3.1 would require │ │ │ │ └─ zlib 1.3.1 _1, which conflicts with any installable versions previously reported; │ │ │ └─ libzlib 1.3.1 would require │ │ │ └─ zlib 1.3.1 _0, which conflicts with any installable versions previously reported; │ │ └─ python_abi 3.10. _cp310, which conflicts with any installable versions previously reported; │ └─ python 3.10.8 would require │ └─ graalpy >=23.0.0,<23.0.1.0a0 , which requires │ └─ python_abi 3.10. _graalpy230_310_native, which can be installed; └─ tiledbvcf-py 0.35.0 is installable with the potential options ├─ tiledbvcf-py 0.35.0 would require │ ├─ libtiledbvcf 0.35.0 h53fe7cb_0, which requires │ │ └─ tiledb >=2.26.0,<2.27.0a0 , which requires │ │ └─ libzlib >=1.3.1,<2.0a0 , which cannot be installed (as previously explained); │ └─ python >=3.8,<3.9.0a0 , which can be installed; ├─ tiledbvcf-py 0.35.0 would require │ ├─ libtiledbvcf 0.35.0 h53fe7cb_0, which cannot be installed (as previously explained); │ ├─ python >=3.10,<3.11.0a0 but there are no viable options │ │ ├─ python [3.10.0|3.10.10|...|3.10.9] conflicts with any installable versions previously reported; │ │ ├─ python [3.10.0|3.10.1|...|3.10.9], which cannot be installed (as previously explained); │ │ ├─ python 3.10.15, which cannot be installed (as previously explained); │ │ └─ python 3.10.8, which cannot be installed (as previously explained); │ └─ python_abi 3.10. _cp310 with the potential options │ ├─ python_abi 3.10 conflicts with any installable versions previously reported; │ └─ python_abi 3.10 would require │ └─ python 3.10. _cpython, which can be installed; ├─ tiledbvcf-py 0.35.0 would require │ ├─ python >=3.11,<3.12.0a0 , which can be installed; │ └─ python_abi 3.11. _cp311, which can be installed; ├─ tiledbvcf-py 0.35.0 would require │ └─ python_abi 3.12. _cp312, which can be installed; └─ tiledbvcf-py 0.35.0 would require ├─ libtiledbvcf 0.35.0 h53fe7cb_0, which cannot be installed (as previously explained); └─ python >=3.9,<3.10.0a0 , which can be installed.

[lcj34@cbsubl01 phgEnvironment_minimap2Tiledb]$

[attractivechaos]

k8 requires python for build but it doesn't need python to run. The only run-time dependency is zlib.

[lynnjo]

@attractivechaos If I'm reading the error message correctly, k8 requires zlib >= 1.2.11 but < 1.3 But tiledbvcf-py version 0.35.0 needs libzlib >= 1.3.1 so that is the incompatibility. (is libzlib the same as zlib here?)

Is that what you see from the message above? Is the only option to find an earlier version of tiledbvcf-py that can use a lower version of libzlib? (which means missing the updates we'd like)

[peterjc]

I agree, the core of the conflict is k8 package is declaring it wants an old version of zlib, while tiledbvcf-py (and the python 3.10 packages) want a newer version.

Looking at https://github.com/bioconda/bioconda-recipes/blob/master/recipes/k8/meta.yaml the k8 v1.2 recipe (which does look to be for the latest version) is explicitly wanting the obsolete Python 3.8 - I would hope that can be relaxed. See #46559 for where that came from, but unclear why. Earlier versions of that pull request tried Python 3.9. Calling @mencian

[lynnjo]

Hi All - any further thoughts on this? Can the k8 v1.2 recipe be changed to allow for more recent python versions?

[martin-g]

The easiest would be to open a new PR without restriction of the Python version or with requirement for a newer Python and/or zlib. Experiment until it works (download the artifacts locally and test them!) and add the label to ask for a review.

[lynnjo]

@martin-g @mencian The run from the pull request failed when testing linux with the messages below. What should I do from here? I tried clicking on "rerun failed jobs" but it needs a user name for http://dev.azure.com which I don't believe I have and I assume I don't have permission to re-run this manually.

[error]We stopped hearing from agent Hosted Agent. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

Agent: Hosted Agent Started: Today at 5:07 AM Duration: 1h 32m 47s

Job preparation parameters 9 queue time variables used system.pullRequest.pullRequestId : 2099927690 system.pullRequest.pullRequestNumber : 51054 system.pullRequest.mergedAt : undefined system.pullRequest.sourceBranch : k8_recipe_update system.pullRequest.targetBranch : master system.pullRequest.targetBranchName : master system.pullRequest.sourceRepositoryUri : https://github.com/bioconda/bioconda-recipes system.pullRequest.sourceCommitId : dbb3d8b59264cb80d2297a8a195658bd0a3b2961 system.pullRequest.isFork : True

[attractivechaos]

In meta files, what is difference between "host", "run" and "build"? I can roughly understand "run" and "build", but I am not sure what role "host" plays. Anyway, k8 doesn't depend on python at runtime.

[lynnjo]

According to chatGPT : Summary of Roles: under requirements of the conda meta.yaml script: build: Tools needed to compile or build the software (e.g., compilers, build systems). host: Libraries and dependencies that the software links to during the build process and that it will need when running in the target environment. run: Libraries or tools that the software requires at runtime after it is installed.

To test this locally I need to be able to download the artifacts, and because the run failed the artifacts were not produced. I don't think I can proceed until the run passes.

[attractivechaos]

Thanks. Then it seems that python, wget, tar and sed should all be moved from "host" to "build".

[lynnjo]

@attractivechaos That sounds reasonable, but because I didn't write the meta.yaml and don't know the reasons these programs were added to "host" vs "build" I'm not comfortable making that change. My aim was to make a conda package that allows minimap2 (built with k8, which hard codes python=3.8) and tiledbvcf-py=0.35.0 (which requires python of 3.10 or greater) to coexist. My only change was setting python >= 3.8.

[mencian]

I was able to get k8 to build on python >=3.8 except on osx x86_64; I'll revisit the recipe in the near future, but this should help many users in the mean time.

[lynnjo]

@mencian Does this mean that while previously we could add minimap2 (which is built with k8) to a conda env on osx x86_64, we will no longer be able to do that with this change?

That was the goal of this change - allowing minimap2 and newer versions of tiledbvcf to be in the same environment. I was able to have them both if I kept tiledbvcf at an older version. I'm concerned this means I can no longer create any environment on my intel MAC that has both minimap2 and tiledbvcf which has major implications for our work.

[peterjc]

@lynnjo the old osx x86_64 packages for k8 will still exist on the (anaconda hosted conda package) repository, so you should be fine.

[lynnjo]

@mencian @peterjc One more question: Does any of this work for ARM builds (I can't remember if they are supported). Thanks

[peterjc]

Yes, look for the file starting osx-arm64 here:

https://anaconda.org/bioconda/minimap2/files https://anaconda.org/bioconda/k8/files

As per the above, the recent k8 update to be less strict on Python covered osx-arm64, linux-64, and linux-aarch64 but NOT osx-x86_64

[lynnjo]

Thank you !!