search

cgat-developers / cgat-apps

cgat-apps repository
Other
33 stars 14 forks source link

Ac githubactions #80

Closed Acribbs closed 3 years ago

Acribbs commented 3 years ago

Hi @sebastian-luna-valero, this one has been trickier.

I found issues with the way we are cythonizing our code. I fixed this by setting the language to 3chr during cythonization in setup.py and fixing a few bug fixes in the pyx files.

However, testing the imports leads to cgat.NCL failures:

AssertionError: bam2peakshape scripts/modules - ImportError: cannot import name 'cnestedlist' from partially initialized module 'cgat.NCL' (most likely due to a circular import) (/home/runner/work/cgat-apps/cgat-apps/cgat/NCL/__init__.py)

I just cant see the circular import issue though. Maybe a fresh pair of eyes can help.

I removed the test_import and tests pass for linux but fail for osx:

AssertionError: error in statement: /bin/bash -c ' cgat runGO  --filename-input=/Users/runner/work/cgat-apps/cgat-apps/tests/runGO.py/go.tsv.gz  --genes-tsv-file=/Users/runner/work/cgat-apps/cgat-apps/tests/runGO.py/genelists.tsv.gz  --min-counts=10 - --filename-ontology=/Users/runner/work/cgat-apps/cgat-apps/tests/runGO.py/ontology.obo  --fdr --sample-size=10 --fdr-method=empirical --gene2name-map-tsv-file=/Users/runner/work/cgat-apps/cgat-apps/tests/runGO.py/gene2name.tsv  --random-seed=1 > /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpo65py7ib/stdout'; stderr=b'Traceback (most recent call last):\n  File "/usr/local/miniconda/envs/cgat-a/bin/cgat", line 33, in <module>\n    sys.exit(load_entry_point(\'cgat==0.6.0\', \'console_scripts\', \'cgat\')())\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/cgat-0.6.0-py3.8-macosx-10.9-x86_64.egg/cgat/cgat.py", line 129, in main\n    module = imp.load_module(command, file, pathname, description)\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/imp.py", line 234, in load_module\n    return load_source(name, filename, file)\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/imp.py", line 171, in load_source\n    module = _load(spec)\n  File "<frozen importlib._bootstrap>", line 702, in _load\n  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked\n  File "<frozen importlib._bootstrap_external>", line 783, in exec_module\n  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/cgat-0.6.0-py3.8-macosx-10.9-x86_64.egg/cgat/tools/runGO.py", line 164, in <module>\n    import cgat.GO as GO\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/cgat-0.6.0-py3.8-macosx-10.9-x86_64.egg/cgat/GO.py", line 19, in <module>\n    import scipy.stats\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/scipy/stats/__init__.py", line 391, in <module>\n    from .stats import *\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/scipy/stats/stats.py", line 174, in <module>\n    from scipy.spatial.distance import cdist\n  File "/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/scipy/spatial/__init__.py", line 98, in <module>\n    from .qhull import *\nImportError: dlopen(/usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/scipy/spatial/qhull.cpython-38-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib\n  Referenced from: /usr/local/miniconda/envs/cgat-a/lib/python3.8/site-packages/scipy/spatial/qhull.cpython-38-darwin.so\n  Reason: image not found\n'

Looks like this is to do with numpy or scipy issues. Not sure if you know of a fix?

Lastly, I will need to re-write the test_commandline.py because I changed the option parser to argparser (a while ago) and travis never caught this. Not sure why, but the travis testing looks like its been broken for a while and now it doesn't even install the correct Conda environment. However, I was thinking that this test isn't probably required because we are testing this by running test_scripts.py anyway.

Acribbs commented 3 years ago

@sebastian-luna-valero, looks like Mac tests break with the latest version of scipy. Have temp pinned for the moment. im going to play around with the import tests and see if I can work out where the circular import is coming from.

Acribbs commented 3 years ago

So it seems like I found the circular import and corrected it. All tests seem to pass.

In summary of the changes I have made:

Lots of issues that have been spotted by migrating to actions and I think it's a lot easier to manage testing now.

@sebastian-luna-valero If you could have a quick review and merge if your happy. Thanks.

Acribbs commented 3 years ago

Looks like new version of pysam gives different output for certain scripts. May revisit this in time and identify the issue.