search

moiexpositoalonsolab / grenepipe

A flexible, scalable, and reproducible pipeline to automate variant calling from raw sequence reads, with lots of bells and whistles.
http://grene-net.org
GNU General Public License v3.0
94 stars 21 forks source link

No module named 'numpy.core._multiarray_umath' #5

Closed bensprung closed 3 years ago

bensprung commented 3 years ago

Hi, I'm trying out grenepipe--thanks for making it. I'm on Ubuntu 20.04, I installed miniconda with Python 3.9, then snakemake, the downloaded grenepipe. Very late in the process (83 of 86 steps) I get the following error

  File "/home/ben/miniconda3/envs/snakemake/lib/python3.9/site-packages/pandas/__init__.py", line 17, in <module>
    "Unable to import required dependencies:\n" + "\n".join(missing_dependencies)
ImportError: Unable to import required dependencies:
numpy: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.6 from "/home/ben/Synced/research/FitnessDistributionExperiments/2018_Sep-Dec_3654_and_3660/Temp-sequencing_Dec2020/grenepipe_try/.snakemake/conda/e2a4eb8e82f5806525944d44f65469a7/bin/python"
  * The NumPy version is: "1.20.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

I'll attach the full log file. Grateful if you are able to point me to what I need to do.

Thanks, Ben 2021-06-10T122216.640782.snakemake.log

lczech commented 3 years ago

Hi Ben,

thanks for trying grenepipe - I hope that it otherwise is helpful!

The numpy issue is a tricky one - I've had that before, and this is likely due to some weird interaction between conda and python 2/3. Not sure how I solved it back then - but I have just pushed a change to grenepipe that might fix it for good. Would you mind updating to the latest commit. That is, download via the "Code" button on https://github.com/moiexpositoalonsolab/grenepipe, instead of using a release version, and try again with that one?

So long Lucas

bensprung commented 3 years ago

Hi Lucas, thanks much for the quick fix! Trying it now, will report back.

bensprung commented 3 years ago

:( looks like the same issue.

2021-06-10T155123.927072.snakemake.log

lczech commented 3 years ago

Ah too bad, sorry. Have you tried the numpy troubleshoot thing here: https://numpy.org/devdocs/user/troubleshooting-importerror.html ? Not sure though that this will solve it. The issue most likely is a conflict between your local python/pandas/numpy setup, and the one that snakemake is trying to use...

Could you maybe edit the file envs/picard.yaml in your grenepipe directory, and add the line - python before the - numpy line, and see if that does anything?

lczech commented 3 years ago

Hm, I think I remember that I just completely uninstalled numpy locally, or something like that to get it "solved"... Obviously that is not a good solution, so hopefully there is a better way to get this done. Unfortunately, I'll be travelling for a few days, so I'll only be able to have a proper look at this towards the end of next week. But please let me know if you have any updates :-)

lczech commented 3 years ago

Ah, more things to try. Instead of the - python line that I suggested earlier, can you try both

- python =3.6 and - python =3.9

and see if either of those fixes it?

Hope it's okay to have this back-and-forth here, but I cannot reproduce your issue at the moment, so that seems the only way right now...

bensprung commented 3 years ago

Try those now! Sorry for the delay. Yes, no worries, any form of help is appreciated.

bensprung commented 3 years ago

setting - python =3.6 failed pretty early, log attached. trying 3.9 now. 2021-06-14T144820.667098.snakemake.log

bensprung commented 3 years ago

Looks like 3.9 gave the same error as originally.

2021-06-14T150421.649111.snakemake.log

lczech commented 3 years ago

Hi @bensprung,

thanks for your patience! So, although I had that error as well in the past, I cannot reproduce it any more... which makes it hard to debug.

One thing we can try is to use a Python version that is not present in your system at all. Could you do the above (edit envs/picard.yaml), but use - python ==3.7? I hope that this way, a completely new conda environment gets created that loads this python version, and hence does not conflict with your existing one.

Furthermore, we could try to request a particular numpy version that also does not exist on your system, for example the latest one. So, similar to above (in the envs/picard.yaml), could you try to replace the - numpy line in the by - numpy ==1.21.0?

Hope that works...

Cheers and so long Lucas

bensprung commented 3 years ago

So I'm not entirely sure what I did, but I thought I mangled picard.yaml, so I re-downloaded grenepipe from the latest, and it appears that in envs/picard.yaml python was already set to 3.7. However now I get a new error, which is (in logs/picard/multiple_metrics/121B01-1.log):

ERROR 2021-06-27 13:31:22 ProcessExecutor /home/ben/grenepipe_try/.snakemake/conda/3b24c4634409546dfeb3ef9ce0a88b45/lib/R/bin/exec/R: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory

That error seems to occur when Python is set to 3.6, 3.7, or 3.9.

EDIT: whoops I spoke too soon. I do not get that error about libreadline.so.6 with both - python ==3.9 and - numpy ==1.21.0. Also, I did not get the original error I reported--the entire run finished successfully!

lczech commented 3 years ago

Oh nice, that sounds good! So yes, I added 3.7 to the envs/picard.yaml last week, in an attempt to fix your issue, but apparently, that was the wrong one then.

No idea what these conda/python/numpy interactions are about - conda is meant to fix such issues, but apparently, it does not... It is not caused by our pipeline itself - it is just suffering from it - but I am glad we could resolve this! Thanks for your patience and constant feedback!

So just to be sure that I am doing the right thing now: Your envs/picard.yaml works with - python ==3.9 and - numpy ==1.21.0? I have changed it to that, and just pushed that change. I'll close the issue for now, but if this is not the right fix, feel free to re-open!

bensprung commented 3 years ago

Thank you! Yes, you have it right, it works for me with - python ==3.9 and - numpy ==1.21.0. It might be the - numpy ==1.21.0 that did the trick on its own, I didn't test it with other versions of python (but I can do it if it would be helpful).

lczech commented 3 years ago

Perfect, thanks for the feedback! I'll leave it at that then, with both versions specified, as this has (for now) proven to work. If you (or anyone) in the future has other issues with this, feel free to re-open :-D