Add library dependencies of new file formats in HB

[sveinugu]

• [ ] BigBed
• [ ] BigWig
• [ ] VCF

[radmilak]

@sveinugu Installing the libraries on insilico seems to be rather problematic.. This is the libraries we need: bx-python>=0.8.2 pyBigWig>=0.3.13 plastid>=0.4.8 When I tried to install this (from hyperbrowser-reqirements.txt): 1) There was a problem with plastid dependency scipy - it tried to install the newest version that required py3, so it failed 2) I tried to add scipy==0.16 (last version that supports py2), but the installation also failed (because of missing library (?) - _numpy.distutils.systeminfo.NotFoundError: no lapack/blas resources found) So I was wondering if we maybe already have scipy on some other instance and how did you install it? 3) pyBigWig installation failed as well - it seems like it is the same problem I had locally long time ago, though I can't reproduce it locally any more for some reason.. The problem was some encoding inconsistency we thought was due to python2/python3 differences and it caused one of the libraries to be linked like this: - L / s o f t w a r e / m i n i c o n d a 2 / l i b (with the spaces) 4) I tried to install the modified version (that worked for the local install long time ago, it has one line removed from setup.py) to my sandbox galaxy virtual env but it seems pip is too old to install that package (the error is _error: invalid command 'egginfo' which I found can be fixed by updating pip or installing a package called distribute, though I was too afraid to try that (:-D) 5) Some good news at the end - it seems that there is no problem with installing bx_python!

Do you have some ideas about what could I do to get the dependencies installed?

[sveinugu]

Scipy is a known issue on insilico which I usually solve by manuellterapeut copying a working installation. But I think it may be v0.14. Does that work?

Didn’t you provide a pull request for the PyBigWig issue?

Anyway, pip is a Galaxy patched version of pip (at least for the old version of Galaxy we are running), so it might not be so easy to update. There is a Galaxy repo somewhere with the exact versions supported so I think it should work with updating pip to one of those. Is pip itself in one of the requirements.txt files? If so, updating that file and restart should work.

Sveinung

-- Sveinung Gundersen, PhD

Senior engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Norway

On 24 Mar 2020, 16:05 +0100, radmilak notifications@github.com, wrote:

@sveinugu Installing the libraries on insilico seems to be rather problematic.. This is the libraries we need: bx-python>=0.8.2 pyBigWig>=0.3.13 plastid>=0.4.8 When I tried to install this (from hyperbrowser-reqirements.txt):

1. There was a problem with plastid dependency scipy - it tried to install the newest version that required py3, so it failed
2. I tried to add scipy==0.16 (last version that supports py2), but the installation also failed (because of missing library (?) - numpy.distutils.system_info.NotFoundError: no lapack/blas resources found) So I was wondering if we maybe already have scipy on some other instance and how did you install it?
3. pyBigWig installation failed as well - it seems like it is the same problem I had locally long time ago, though I can't reproduce it locally any more for some reason.. The problem was some encoding inconsistency we thought was due to python2/python3 differences and it caused one of the libraries to be linked like this: - L / s o f t w a r e / m i n i c o n d a 2 / l i b (with the spaces)
4. I tried to install the modified version (that worked for the local install long time ago, it has one line removed from setup.py) to my sandbox galaxy virtual env but it seems pip is too old to install that package (the error is error: invalid command 'egg_info' which I found can be fixed by updating pip or installing a package called distribute, though I was too afraid to try that (:-D)
5. Some good news at the end - it seems that there is no problem with installing bx_python!

Do you have some ideas about what could I do to get the dependencies installed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[sveinugu]

Obviously manually not manuellterapeut. Written on iPhone with Norwegian dictionary... :D

Sveinung

-- Sveinung Gundersen, PhD

Senior engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Norway

On 24 Mar 2020, 16:05 +0100, radmilak notifications@github.com, wrote:

@sveinugu Installing the libraries on insilico seems to be rather problematic.. This is the libraries we need: bx-python>=0.8.2 pyBigWig>=0.3.13 plastid>=0.4.8 When I tried to install this (from hyperbrowser-reqirements.txt):

1. There was a problem with plastid dependency scipy - it tried to install the newest version that required py3, so it failed
2. I tried to add scipy==0.16 (last version that supports py2), but the installation also failed (because of missing library (?) - numpy.distutils.system_info.NotFoundError: no lapack/blas resources found) So I was wondering if we maybe already have scipy on some other instance and how did you install it?
3. pyBigWig installation failed as well - it seems like it is the same problem I had locally long time ago, though I can't reproduce it locally any more for some reason.. The problem was some encoding inconsistency we thought was due to python2/python3 differences and it caused one of the libraries to be linked like this: - L / s o f t w a r e / m i n i c o n d a 2 / l i b (with the spaces)
4. I tried to install the modified version (that worked for the local install long time ago, it has one line removed from setup.py) to my sandbox galaxy virtual env but it seems pip is too old to install that package (the error is error: invalid command 'egg_info' which I found can be fixed by updating pip or installing a package called distribute, though I was too afraid to try that (:-D)
5. Some good news at the end - it seems that there is no problem with installing bx_python!

Do you have some ideas about what could I do to get the dependencies installed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[radmilak]

@sveinugu Haha, well it kinda is "manual therapy" :-D Hmm for plastid 0.4.18 there has to be scipy>=0.15.1 but we only use a small and non-central part of plastid so perhaps we could use some older version that has older scipy, I can check that.

No, I haven't created any pull requests for pyBigWig, I think we were not really sure if the fix is something general or more something with my environment or py2 vs py3. Also, like I said now I can't even reproduce it any more on my laptop, so that seems like it is something related to conda/pip more than python (because I still use py2 locally as well...). I was only wondering about the pip update because when installing the dependencies pip says itself it is old and you should update. Also you were somehow able to install the pytest-plugin from a local folder, but maybe that package was created in a different way or something, I can try to check if I find any differences. And pip is not in any of the requirements files (in the dependencies folder).

[sveinugu]

Manually copied scipy v0.17 as a manual fix. Managed to install plastid.

To install pyBigWig:

pip install --upgrade pip
pip install pyBigWig
pip install "pip==8.0.2"

Then start up HyperBrowser. Galaxy will then update the standard "8.0.2" version of pip to its own customly-hacked version. Updating HB to a new Galaxy version will fix all of this, so I will not do any other custom changes now.

[radmilak]

@sveinugu Right now there are two problems remaining: 1) For the BigBedComposer, we need ucsc-bedtobigbed, there is no pip package, only bioconda (or there is also a binary for Linux from ucsc..). You said we could install this globally since in the new Galaxy we will have conda support so this will then become a normal dependency. While I now have the access I need to do this myself, I am afraid of destroying the conda env since the dependency management on insilico seems a little fragile :-) so I haven't tried. 2) Since both BigWigGES and BigBedGES operates with numpy arrays we need pyBigWig to have numpy support. According to pyBigWig's github page you should have this support if you have numpy installed previously to pyBigWig installation. It seems we already had numpy but didn't get the support anyway - you can check with 'pyBigWig.numpy' in python, if it returns 1 it has the support. One of the reasons for this could possibly be that the numpy version is too old, but I haven't found any info about what the version should be. Or it could maybe be the extra steps with pip needed to get pyBigWig installed (in the previous comment). It's worth mentioning that I found some info that suggests that when you install pyBigWig with conda, the numpy support should not be an issue, but the info is old and the topic is about something else, but perhaps if we don't manage to fix it in any other way it's worth a shot - https://github.com/deeptools/pyBigWig/issues/18, second comment from 31 Oct 2016.

[sveinugu]

bedToBigBed binary is now directly installed into /software/galaxy/gocd_galaxies/hb_project_trackfind_test/.venv/bin. If you remove the requirement, I think it should work as a hack until conda is fixed on insilico

[sveinugu]

installed in incorrect instance. bedToBigBed is also now installed in radmilko_sandbox.

Manually installed pyBigWig from (hacked) source. Removed old build/ directory with non-numpy-compliant build (for some reason).

bedToBigBed dependency should be included as a conda dependency when Galaxy is fully updated (with conda support for Galaxy dependencies).