Issue with building RNAlysis under Archlinux

[kbipinkumar]

while trying to build RNAlysis from PyPI sources under archlinux i am encountering following error.

  $ python setup.py build

  File "/home/bipin/Share/test/rnalysis/src/RNAlysis-3.3.0/setup.py", line 38, in <module>
    with open('requirements.txt') as requirements_file:
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

However, the issues don't seem to affect sources downloaded from GitHub. that said GitHub source download is huge (~700mb) compared to PyPi sources. any idea reason discrepancy?

also it would be great if PyPi sources could be fixed so that package can be built using the same

furthermore, there is also the issue of setup.py installing directory named tests into site-packages folder (i.e. /usr/lib/python3.10/site-packages/tests/) which is an undesirable behaviour per the wiki due to setuptools being misconfigured to include the directory containing its tests as a top level Python package.

this causes the resulting binary packages from being installed due file conflicts

$ sudo pacman -U rnalysis-3.3.0-1-any.pkg.tar.zst

error: failed to commit transaction (conflicting files)
rnalysis: /usr/lib/python3.10/site-packages/tests/__init__.py exists in filesystem (owned by python-pairwisedist)
rnalysis: /usr/lib/python3.10/site-packages/tests/__pycache__/__init__.cpython-310.opt-1.pyc exists in filesystem (owned by python-pairwisedist)
rnalysis: /usr/lib/python3.10/site-packages/tests/__pycache__/__init__.cpython-310.pyc exists in filesystem (owned by python-pairwisedist)

it would be great to have this behaviour fixed in setup.py itself.

[GuyTeichman]

Hi, Thanks for bringing this to my attention!

Regarding the issues with 1.tests being added to site-packages; and 2.requirements text files not being included in PyPi wheel releases - I believe I managed to fix these two (see relevant commits above), and they should be appended to the next release (3.4.0) within a week or two.

Regarding the discrepancy between the PyPi release and GitHub source sizes - what accounts for the difference are the tutorial videos (~400Mb). I excluded the videos from the PyPi release due to the release size limit PyPi imposes. Instead, when users open RNAlysis for the first time, the videos are downloaded from GitHub to the users' AppData directory (as to not introduce change to the contents of the site-packages folder). I'm not sure if there is a better practice for handling this issue, and I'd be happy to hear your suggestions.

Best, Guy.

[kbipinkumar]

Hi, Thanks for bringing this to my attention!

Regarding the issues with 1.tests being added to site-packages; and 2.requirements text files not being included in PyPi wheel releases - I believe I managed to fix these two (see relevant commits above), and they should be appended to the next release (3.4.0) within a week or two.

Hi, thanks for the quick response. Hopefully, these patches will result in packages being built from PyPi source. i look forward to building the same when the new release is out.

Regarding the discrepancy between the PyPi release and GitHub source sizes - what accounts for the difference are the tutorial videos (~400Mb). I excluded the videos from the PyPi release due to the release size limit PyPi imposes. Instead, when users open RNAlysis for the first time, the videos are downloaded from GitHub to the users' AppData directory (as to not introduce change to the contents of the site-packages folder). I'm not sure if there is a better practice for handling this issue, and I'd be happy to hear your suggestions.

Best, Guy.

so long as packages can be built from pypi sources this is not a big issue. as these are tutorial videos that are not essential to the actual functioning of the software. So, downloading them to something like $HOME/.local/$Appname location upon first run seems to be a right compromise.

that said i have encountered one more issue in the course of packaging attempt. Archlinux uses something called PKGBUILD files, which are essentially bash scripts to compile the packages from source. the part of the process, especially for python based software, is to run any tests that are included in source by author using command like these

check(){
  local python_version=$(python -c 'import sys; print("".join(map(str, sys.version_info[:2])))')
  cd "$_name-$pkgver"
  python setup.py pytest
}

this step is failing for me right. i am attaching the log output of pytest as well as 'PKGBUILD` i am using. if would be great help if you could look into the same and tell

• what i am doing wrong
• whether i have missed any dependencies or is using some incompatible version of the dependencies

Also i just noticed pairwisedist maintained by you also seems to install a test folder to site-packages as well. please take look at it as well

Regards

bipin

[GuyTeichman]

that said i have encountered one more issue in the course of packaging attempt. Archlinux uses something called PKGBUILD files, which are essentially bash scripts to compile the packages from source. the part of the process, especially for python based software, is to run any tests that are included in source by author using command like these

check(){
  local python_version=$(python -c 'import sys; print("".join(map(str, sys.version_info[:2])))')
  cd "$_name-$pkgver"
  python setup.py pytest
}

this step is failing for me right. i am attaching the log output of pytest as well as 'PKGBUILD` i am using. if would be great help if you could look into the same and tell

• what i am doing wrong
• whether i have missed any dependencies or is using some incompatible version of the dependencies

It looks to me like the tests crash as soon as they reach GUI tests (using PyQt5). My guess would be that some dependencies/configuration there are missing. In particular, I suspect that libxkbcommon-x11-0 needs to be installed, and possibly some or all of the following: libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 x11-utils. Furthermore, you may need to set the environment variable DISPLAY (since PyQt5 cannot run without a display, be is a real one or a virtual one).

The troubleshooting section of pytest-qt could be helpful: https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html

And if you're still having trouble setting it up, try setting the environment variable QT_DEBUG_PLUGINS to "1", which will display more detailed debugging information for PyQt5.

Please let me know if any of these worked, and if not we will figure out what to try next.

Also i just noticed pairwisedist maintained by you also seems to install a test folder to site-packages as well. please take look at it as well

Good catch - that should be taken care of now (PyPi release version 1.3.1).

Thanks again, Guy.

[kbipinkumar]

It looks to me like the tests crash as soon as they reach GUI tests (using PyQt5). My guess would be that some dependencies/configuration there are missing. In particular, I suspect that libxkbcommon-x11-0 needs to be installed, and possibly some or all of the following: libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 x11-utils. Furthermore, you may need to set the environment variable DISPLAY (since PyQt5 cannot run without a display, be is a real one or a virtual one).

The troubleshooting section of pytest-qt could be helpful: https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html

And if you're still having trouble setting it up, try setting the environment variable QT_DEBUG_PLUGINS to "1", which will display more detailed debugging information for PyQt5.

Hi, i was able to get past test failing at GUI tests by installing python-pytest-xvfb as recommended in link given above.

although GUI tests did successfully complete, tests related to actual functionality seems to have failed. here is the truncated output below

=========================== short test summary info ============================
FAILED tests/test_clustering.py::test_clicom_cluster_wise_result - joblib.ext...
FAILED tests/test_clustering.py::test_clicom_get_cluster_similarity_matrix - ...
FAILED tests/test_clustering.py::test_clicom_inter_cluster_similarity[0-1-0.2222222222222222]
FAILED tests/test_clustering.py::test_clicom_inter_cluster_similarity[0-3-0.6666666666666666]
FAILED tests/test_clustering.py::test_clicom_inter_cluster_similarity[0-5-1]
FAILED tests/test_clustering.py::test_clicom_inter_cluster_similarity[3-5-0.6666666666666666]
FAILED tests/test_enrichment.py::test_enrichment_randomization_parallel_reliability
FAILED tests/test_enrichment.py::test_enrichment_parallel_validity - joblib.e...
FAILED tests/test_enrichment.py::test_enrich_hypergeometric_api - joblib.exte...
FAILED tests/test_enrichment.py::test_enrich_hypergeometric_pvalues - joblib....
FAILED tests/test_enrichment.py::test_go_enrichment_api[auto-hypergeometric-classic-kwargs0]
FAILED tests/test_enrichment.py::test_go_enrichment_api[auto-fisher-elim-kwargs1]
FAILED tests/test_enrichment.py::test_go_enrichment_api[caenorhabditis elegans-randomization-no-kwargs2]
FAILED tests/test_enrichment.py::test_kegg_enrichment_api[auto-hypergeometric-kwargs0]
FAILED tests/test_enrichment.py::test_kegg_enrichment_api[6239-fisher-kwargs1]
FAILED tests/test_enrichment.py::test_kegg_enrichment_api[caenorhabditis elegans-randomization-kwargs2]
FAILED tests/test_io.py::test_golr_annotation_iterator_golr_request_connectivity
===== 17 failed, 1439 passed, 1 skipped, 480 warnings in 503.53s (0:08:23) =====

i am attaching the complete log here. please take a look at your leisure an tell whats going wrong or if these errors can be ignored.

also if you have an svg version of this app icon, please add it to the repo as well. it would make it easier to package the app with proper .desktop file included so that RNAlysis will integrate well with app launchers of all major linux desktop environments.

[GuyTeichman]

Hi, It looks like all of the errors are caused by one of two things:

The first group of errors appears to be caused by an interaction between joblib and matplotlib - whenever Python tries to run a multiprocess job via joblib, the matplotlib import fails inside the workers and an error is raised. I never encountered this error before, but my guess would be that there's some version incompatibility between them.

I actually recently updated the required versions for matplotlib, joblib, and hdbscan, since a dependency bug was fixed on the hdbscan package. This requirement update is not yet in the official release (scheduled to be released in RNAlysis 3.4.0), but perhaps manually installing/updating the requirements could solve the problems you run into. Please try installing the following dependencies: matplotlib>=3.5.2, joblib>=1.2.0, hdbscan>=0.8.29 Or installing the development branch of RNAlysis from source. I suspect this will solve the problem.

The second group of errors appears to be related to the GOlr Gene Ontology Solr server - for some reason connection is refused. I'm not sure why that is, since the server is currently up and responsive (I can access it manually, and all the relevant tests passed on GitHub Actions a few minutes ago). I also didn't see any other connection issues in your test setup (to services such as UniprotKB, KEGG, etc). Perhaps there is some connection problem between the test server and the GOlr server?

also if you have an svg version of this app icon, please add it to the repo as well. it would make it easier to package the app with proper .desktop file included so that RNAlysis will integrate well with app launchers of all major linux desktop environments.

Currently the following version of the logo is available as an SVG: https://github.com/GuyTeichman/RNAlysis/blob/master/docs/source/logo.svg I could add an SVG version of the logo you attached to the next version of RNAlysis (same logo, but without the grey frame). Does it need it to have a specific filename?

Best, Guy.

[kbipinkumar]

Please try installing the following dependencies: matplotlib>=3.5.2, joblib>=1.2.0, hdbscan>=0.8.29

These are installed versions of the aforementioned dependencies, please let me know if these are okay

• python-joblib 1.2.0
• python-matplotlib 3.6.2
• python-hdbscan 0.8.29

Perhaps there is some connection problem between the test server and the GOlr server?

is this the server URL? http://golr.berkeleybop.org/ if so it seems to be timing out from inside my institutional network. later today i wil try again with non firewalled connection to see the issue goes away. in that case i will contact our network admins to see what can be done to fix the same.

Currently the following version of the logo is available as an SVG: https://github.com/GuyTeichman/RNAlysis/blob/master/docs/source/logo.svg I could add an SVG version of the logo you attached to the next version of RNAlysis (same logo, but without the grey frame). Does it need it to have a specific filename?

that icon does seem to be do the job. however as said above please remove the grey frame as well as RNAlysis text from the icon (for ui consistency)

Regarding size of icon, freedesktop.org guidelines recommend minimum icon size of 48 x 48 (square aspect ratio). you can find detailed explanation here. for global installation icons are usually installed to /usr/share/icons/hicolor/scalable/apps with appname.svg. therefore, RNAlysis.svg seems to be appropriate.

[GuyTeichman]

These are installed versions of the aforementioned dependencies, please let me know if these are okay

python-joblib 1.2.0 python-matplotlib 3.6.2 python-hdbscan 0.8.29

Yes, these should be good.

is this the server URL? http://golr.berkeleybop.org/ if so it seems to be timing out from inside my institutional network. later today i wil try again with non firewalled connection >to see the issue goes away. in that case i will contact our network admins to see what can be done to fix the same.

This is the server: http://golr-aux.geneontology.io/

Here is a sample query: http://golr-aux.geneontology.io/solr/select?q=%2A%3A%2A&wt=json&rows=0&fq=document_category%3A%22annotation%22&fq=taxon%3A%22NCBITaxon%3A6239%22&fq=aspect%3A%22P%22&fq=evidence_type%3A%22IMP%22&fq=-qualifier%3A%22not%22&fq=source%3A%22WB%22&fl=source%2Cbioentity_internal_id%2Cannotation_class

It should give you the following output: {"responseHeader":{"status":0,"QTime":0,"params":{"q":"*:*","fq":["document_category:\"annotation\"","taxon:\"NCBITaxon:6239\"","aspect:\"P\"","evidence_type:\"IMP\"","-qualifier:\"not\"","source:\"WB\""],"rows":"0","wt":"json","fl":"source,bioentity_internal_id,annotation_class"}},"response":{"numFound":9871,"start":0,"docs":[]}}

that icon does seem to be do the job. however as said above please remove the grey frame as well as RNAlysis text from the icon (for ui consistency)

Regarding size of icon, freedesktop.org guidelines recommend minimum icon size of 48 x 48 (square aspect ratio). you can find detailed explanation here. for global installation icons are usually installed to /usr/share/icons/hicolor/scalable/apps with appname.svg. therefore, RNAlysis.svg seems to be appropriate.

OK, I will add the file to the development branch later today (will also be part of V3.4.0).

[kbipinkumar]

This is the server: http://golr-aux.geneontology.io/

that server is seems to be blocked by our institutional firewall as well. i will sort out the same with our network admin team at the earliest

OK, I will add the file to the development branch later today (will also be part of V3.4.0).

Great. thanks a lot for your patience

[kbipinkumar]

The second group of errors appears to be related to the GOlr Gene Ontology Solr server - for some reason connection is refused. I'm not sure why that is, since the server is currently up and responsive (I can access it manually, and all the relevant tests passed on GitHub Actions a few minutes ago). I also didn't see any other connection issues in your test setup (to services such as UniprotKB, KEGG, etc). Perhaps there is some connection problem between the test server and the GOlr server?

This seems to be fixed when building in a system connected to an open network.

As the system was a bare installation, during build process i noticed that help videos being downloaded are on .webp format and the system required Qt 5 Image Formats addon to be installed for the tutorial videos to be played in the gui. The addon might not be installed by default in all Linux distros. therefore, please add the same as dependency in the README.rst. The dependency can be installed,

on Debian/ubuntu systems

$ sudo apt install qt5-image-formats-plugins

on redhat based distros such as fedora

$ dnf install qt5-qtimageformats

on Arch Linux based distros (The PKGBUILD script in Arch User repository for RNAlysis should take care of this dependency)

$ sudo pacman -S qt5-imageformats

also under KDE/wayland the app does not expose it icon to taskbar and instead generic wayland icon is used. this is probably due to issue mentioned here

[GuyTeichman]

The second group of errors appears to be related to the GOlr Gene Ontology Solr server - for some reason connection is refused. I'm not sure why that is, since the server is currently up and responsive (I can access it manually, and all the relevant tests passed on GitHub Actions a few minutes ago). I also didn't see any other connection issues in your test setup (to services such as UniprotKB, KEGG, etc). Perhaps there is some connection problem between the test server and the GOlr server?

This seems to be fixed when building in a system connected to an open network. OK, that's good to know!

As the system was a bare installation, during build process i noticed that help videos being downloaded are on .webp format and the system required Qt 5 Image Formats addon to be installed for the tutorial videos to be played in the gui. The addon might not be installed by default in all Linux distros. therefore, please add the same as dependency in the README.rst. The dependency can be installed,

on Debian/ubuntu systems

$ sudo apt install qt5-image-formats-plugins

on redhat based distros such as fedora

$ dnf install qt5-qtimageformats

on Arch Linux based distros (The PKGBUILD script in Arch User repository for RNAlysis should take care of this dependency)

$ sudo pacman -S qt5-imageformats

Thanks for pointing it out. I added those instructions to the README - I'll also add this as an item in the FAQ later on.

also under KDE/wayland the app does not expose it icon to taskbar and instead generic wayland icon is used. this is probably due to issue mentioned here

I followed these instructions and I hope it solved the problem, but I have no way to check. Could you have a look at the current development build?

Thanks again, Guy.

[kbipinkumar]

I followed these instructions and I hope it solved the problem, but I have no way to check. Could you have a look at the current development build?

yep commit 15a0ca6 fixed the issue. Thanks.

[GuyTeichman]

V3.4.0 is out now on PyPi. Please let me know if there are any other issues building RNAlysis under Archlinux!

Also - once you are able to successfully build RNAlysis under Archlinux, would you be able to write brief instructions how to install RNAlysis on Archlinux, so I could add it to the installation instructions?

Thanks, Guy.

[kbipinkumar]

great. at present i am away from my lab, and don't have remote access to the build machine. once i get back to lab i will update you how compilation of updated version went

[kbipinkumar]

hi, the current version 3.4.2 is failing at the testing stage i am attaching the output of python setup.py pytest | tee check.log here. these tests did complete successfully with version 3.3.0.

Please take a look at the log and let me know what the issue could be.

[GuyTeichman]

Hi, What versions of numpy and numba are you running? I suspect that the issue is related to this: https://github.com/numba/numba/issues/8615

Although I'm not sure, since the GitHub Actions CI for RNAlysis passes on all Python versions, and they should determine their numpy/numba versions the same way your tests does, through requirements.txt and extra_requirements.txt...

Could you also check if you get the same errors on V3.3.0? (which previously didn't raise this type of error)

Thanks, Guy.

[kbipinkumar]

the machine has python-numba 0.56.4 and python-numpy 1.24.1 installed.

i will try rebuild with v 3.3.0 as well to be sure.

[GuyTeichman]

FYI, the GitHub Actions test environments under Python 3.10 all built numba-0.56.4 numpy-1.23.5. I'm guessing that numba 0.56.4 doesn't support numpy 1.24 yet (according to the yet-to-be-approved pull request here: https://github.com/numba/numba/pull/8691 ). I wonder why your environment installs numpy 1.24...

[kbipinkumar]

FYI, the GitHub Actions test environments under Python 3.10 all built numba-0.56.4 numpy-1.23.5. I'm guessing that numba 0.56.4 doesn't support numpy 1.24 yet (according to the yet-to-be-approved pull request here: numba/numba#8691 ). I wonder why your environment installs numpy 1.24...

apparently, tests are proceeding with python-numpy 1.23.5 version.

As part of recommended build practices in archwiki we build packages in a clean chroot environment to prevent missing dependencies etc. as i didn't explicitly defined the version of python-numpy as 1.23.5 in PKGBUILD the build process pulled and installed the latest version of python-numpy from archlinux repository which happened to be 1.24. (a side effect of working with a rolling release distro ☹)

[GuyTeichman]

Glad to hear that issue was resolved. Were there any other issues building the package?

[GuyTeichman]

Since https://github.com/BioArchLinux/Packages/issues/107 was marked as closed a few days ago, I'm assuming everything went smoothly. Feel free to re-open the issue or post a new one if any problems pop up!

Thanks agin, Guy.