trouble importing C extensions

nickmagnelli commented 2 months ago

General information

Corrfunc version: 2.5.3
platform: jupyter nersc
installation method (pip/source/other?): git clone

Issue description

Expected behavior

I attempted to install Corrfunc via Method 1 in the readme: https://github.com/manodeep/Corrfunc/blob/master/README.rst so that I could apply Corrfunc, including its C extensions.

Actual behavior

The first several steps seemed to work fine. On the last step... python -m pytest # run the Python tests ... I received a ton of errors which indicate the C extensions could not be imported. See below for the error message.

What have you tried so far?

I checked that my Python and Numpy were up to date, and that appears to be the case. I also verified that the C extensions were not working by trying to call them in an actual application of Corrfunc, and received an error message saying:

ImportError: Could not import the C extension for the 3-D redshift-space pair counter.

Minimal failing example

Here are the final couple lines of the error message (the entire message is too long to include, and these capture the gist of the issue):

FAILED Corrfunc/tests/test_theory.py::test_vpf[fastest-257] - ImportError: Could not import the C extension for the Counts-in-Cells  (vpf)
======================================================== 534 failed in 21.78s ========================================================

I just recently started using jupyter nersc and don't have too much else imported yet, so maybe I am missing some other piece of code that is necessary for this to work?

Thanks in advance!

lgarrison commented 2 months ago

Hi @nickmagnelli, I just tried installing Corrfunc on NERSC and I couldn't reproduce your issue. All I did first was module load python (although one should probably create a venv or conda env too). Can you share the exact commands you ran and the output, starting from a clean environment (i.e. nothing in your ~/.bashrc)? If the output is too big to paste, you can save it to a text file and upload it as an attachment.

Also, it might be worth trying a non-Jupyter terminal, although I tried both Jupyter and non-Jupyter and both worked.

manodeep commented 2 months ago

@nickmagnelli Thanks for using Corrfunc and for opening the issue!

One thing I can think of is that numpy wasn't loaded at install time, so the CPython extensions didn't get built. But there should be a message at the beginning saying Not compiling C extensions for mocks. Either python or numpy not available

The other (highly unlikely) option is that the python used to install Corrfunc is different from the python being used later

nickmagnelli commented 2 months ago

Hi @nickmagnelli, I just tried installing Corrfunc on NERSC and I couldn't reproduce your issue. All I did first was module load python (although one should probably create a venv or conda env too). Can you share the exact commands you ran and the output, starting from a clean environment (i.e. nothing in your ~/.bashrc)? If the output is too big to paste, you can save it to a text file and upload it as an attachment.

Also, it might be worth trying a non-Jupyter terminal, although I tried both Jupyter and non-Jupyter and both worked.

Hi @lgarrison . I took your suggestion, opened up a virtual environment (which did not have a bash history), and I was then able to install Corrfunc successfully. I was also able to apply it afterwards, so I know the C extensions are working. That being said, I am not sure exactly what went wrong the first time. Maybe the bash history was causing an issue or something.

In any case, I think I have it figured out for now (at least within the virtual environment). Thank you for the help!

manodeep commented 2 months ago

Hoorraay! Great that you could solve the issue. I am closing this GitHub issue but if you need more help, please feel free to reopen (or open a new issue depending ...)

manodeep / Corrfunc