Trouble installing Corrfunc 2.3.3

[julianakwan]

General information

• Corrfunc version: 2.3.3
• platform: Linux x86_64
• installation method (pip/source/other?): source

Issue description

The theory python library doesn't build.

Expected behavior

In my .local/lib/python2.7/site-packages/Corrfunc, I would expect a directory called theory, containing wp.py, DD.py etc.

Actual behavior

When I type 'make' in the Corrfunc directory:

make -C theory Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'abiflags'

What have you tried so far?

Minimal failing example

import Corrfunc

# rest of sample code goes here...

[julianakwan]

Also "mocks" didn't get installed either.

[lgarrison]

Interesting, it failed in our Travis build too, but didn't error out: https://travis-ci.org/manodeep/Corrfunc/jobs/645261386#L1330-L1333

The problem is in this line where we build PYTHON_LIBS: https://github.com/manodeep/Corrfunc/blob/master/common.mk#L450

sys.abiflags doesn't exist in Python 2.7.

There's an interesting discussion here about how abiflags may not be relevant anymore, and is an empty string in Python 3.8: https://bugs.python.org/issue36707

@manodeep But actually, are we even using the PYTHON_LIBS variable? I can't find any references to it in our code base, is it being interpreted automatically by setup.py?

[manodeep]

@julianakwan Thanks for the report - we should be able to fix this issue shortly.

@lgarrison Yeah I don't see any references for PYTHON_LIBS either. Might just comment that out for the time being.

[manodeep]

@lgarrison Maybe this is as good a time as any to discuss the timeline for dropping python2 support. Since supporting python2 comes at no additional cost (for now), should we simply retain the compatibility?

[manodeep]

@julianakwan I can see that the error occurs but the installation proceeds as per normal. I can also import Corrfunc, and run the tests. Can you please try the following under your python2 environment:

>>> import Corrfunc
>>> Corrfunc.__version__
>>> from Corrfunc.tests import tests
>>> tests()

[lgarrison]

Sure, no reason to strip it out until it starts to impose a maintenance cost, I think. In the short term, stripping out the support is more work than keeping it!

On Mon, Feb 3, 2020 at 4:34 PM Manodeep Sinha notifications@github.com wrote:

@lgarrison https://github.com/lgarrison Maybe this is as good a time as any to discuss the timeline for dropping python2 support. Since supporting python2 comes at no additional cost (for now), should we simply retain the compatibility?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/manodeep/Corrfunc/issues/212?email_source=notifications&email_token=ABLA7SYJSHZF2YGKSCJVM4DRBCEVFA5CNFSM4KPGBTXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKVPKPA#issuecomment-581629244, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLA7S2DOR4KS2Q7JHHRM43RBCEVFANCNFSM4KPGBTXA .

-- Lehman Garrison lgarrison@flatironinstitute.org Flatiron Research Fellow, Cosmology X Data Science Group Center for Computational Astrophysics, Flatiron Institute lgarrison.github.io

[julianakwan]

Hi Manodeep,

The install did proceed, but whenever I try to import Corrfunc, I get:

import Corrfunc Traceback (most recent call last): File "", line 1, in File "/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/init.py", line 25, in from . import theory ImportError: cannot import name theory

I think this is because the theory directory is missing from .local/lib/python2.7/site-packages/Corrfunc. (The mocks directory is also missing, but I think it looks for the theory directory first).

I typed python setup.py install —user, so the package should be installed in .local right?

Thanks! Juliana

On Feb 3, 2020, at 9:39 PM, Manodeep Sinha notifications@github.com<mailto:notifications@github.com> wrote:

@julianakwanhttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fjulianakwan&umid=e09a81dd-7fa3-4b52-b68d-a48c3c4a2e2b&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-3aeb216199ff7dfe4ec82647d8716fc732291975 I can see that the error occurs but the installation proceeds as per normal. I can also import Corrfunc, and run the tests. Can you please try the following under your python2 environment:

import Corrfunc Corrfunc.version from Corrfunc.tests import tests tests()

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fmanodeep%2fCorrfunc%2fissues%2f212%3femail%5fsource%3dnotifications%26email%5ftoken%3dAGUO2S43UZ4FQELRZBE3N6TRBCFIVA5CNFSM4KPGBTXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKVP3YQ%23issuecomment%2d581631458&umid=e09a81dd-7fa3-4b52-b68d-a48c3c4a2e2b&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-c6b04aed33df2476b471b266c204166bdf949616, or unsubscribehttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe%2dauth%2fAGUO2S4WZA2XHSGQWY74OT3RBCFIVANCNFSM4KPGBTXA&umid=e09a81dd-7fa3-4b52-b68d-a48c3c4a2e2b&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-53db413d23e84d7f2adc4bac34c2e5e5448b2c07.

---

Important Notice: the information in this email and any attachments is for the sole use of the intended recipient(s). If you are not an intended recipient, or a person responsible for delivering it to an intended recipient, you should delete it from your system immediately without disclosing its contents elsewhere and advise the sender by returning the email or by telephoning a number contained in the body of the email. No responsibility is accepted for loss or damage arising from viruses or changes made to this message after it was sent. The views contained in this email are those of the author and not necessarily those of Liverpool John Moores University.

[manodeep]

I tried on the local supercomputer with python2 and I still can't reproduce the issue. @lgarrison Can you please check?

@julianakwan In the meantime, will you please navigate to the /homes/jkwan/.local/lib/python2.7/site-packages/ directory and delete any directories containing *Corrfunc*, and any theory, mocks directories. After that, will you please go back to the source directory and install with the following commands:

$ make -j4 distclean 
$ python -m pip install --user --verbose . &> corrfunc_install.log

and attach the generated corrfunc_install.log file here.

[julianakwan]

Here is the install log. I don’t have pip and I can’t install it on the system so I don’t think it went well…

Thanks! Juliana

On Feb 3, 2020, at 10:51 PM, Manodeep Sinha notifications@github.com<mailto:notifications@github.com> wrote:

I tried on the local supercomputer with python2 and I still can't reproduce the issue. @lgarrisonhttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2flgarrison&umid=45d05069-3d9a-4274-ae17-3f32fe0f0333&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-fad782503bcd40ef32712c17c5c950d8ad567089 Can you please check?

@julianakwanhttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fjulianakwan&umid=45d05069-3d9a-4274-ae17-3f32fe0f0333&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-c7f9c4ea55a3e02efbb23a7330a81a61ba675b21 In the meantime, will you please navigate to the /homes/jkwan/.local/lib/python2.7/site-packages/ directory and delete any directories containing Corrfunc, and any theory, mocks directories. After that, will you please go back to the source directory and install with the following commands:

$ make -j4 distclean $ python -m pip install --user --verbose . &> corrfunc_install.log

and attach the generated corrfunc_install.log file here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fmanodeep%2fCorrfunc%2fissues%2f212%3femail%5fsource%3dnotifications%26email%5ftoken%3dAGUO2S5TVTQRSSUZCGHD2VLRBCNW5A5CNFSM4KPGBTXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKVWHNI%23issuecomment%2d581657525&umid=45d05069-3d9a-4274-ae17-3f32fe0f0333&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-90fb6f80afe1bb57a4a50af4b36adb7afd5b2372, or unsubscribehttps://hes32-ctp.trendmicro.com/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2fnotifications%2funsubscribe%2dauth%2fAGUO2S7YRSGMJVMNDZRE2DLRBCNW5ANCNFSM4KPGBTXA&umid=45d05069-3d9a-4274-ae17-3f32fe0f0333&auth=768f192bba830b801fed4f40fb360f4d1374fa7c-2dd5376d40be29d97c23c2db2bf77f89cdec3c27.

---

Important Notice: the information in this email and any attachments is for the sole use of the intended recipient(s). If you are not an intended recipient, or a person responsible for delivering it to an intended recipient, you should delete it from your system immediately without disclosing its contents elsewhere and advise the sender by returning the email or by telephoning a number contained in the body of the email. No responsibility is accepted for loss or damage arising from viruses or changes made to this message after it was sent. The views contained in this email are those of the author and not necessarily those of Liverpool John Moores University.

[lgarrison]

I don't think the log made it; you might have to attach it to your comment in GitHub rather than send it as an email attachment. But since you don't have pip, could you actually run the following instead:

$ make -j4 distclean
$ python setup.py install --user --verbose &> corrfunc_install.log

[julianakwan]

Here it is again, this time using: python setup.py install --user --verbose &> corrfunc_install.log

[julianakwan]

corrfunc_install.log

[julianakwan]

Also in /homes/jkwan/.local/lib/python2.7/site-packages/ I have 4 corrfunc related items:

Corrfunc
Corrfunc-2.3.3-py2.7.egg-info theory mocks

But in the theory and mocks directories there is only a file called 'tests' no *.py files

[manodeep]

@julianakwan Thanks for the log. The theory and mocks directories should contain a data sub-directory. From the install log, it looks like most stuff was copied across correctly, except for the theory and mocks directories under site-packages/Corrfunc/. @julianakwan Will you please report back the results from find /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/ -name *.py? Also, will you please report back the distutils (or setuptools) version that you are using?

@lgarrison Any ideas on why a seemingly successful install might not copy the Corrfunc/theory/*.py and the Corrfunc/mocks/*.py files?

(Related note: the theory and mocks directories under site-packages is an unfortunate outcome of the current Corrfunc setup and is noted in #207. By design, those two directories only contain test data and not any of the python scripts.)

A somewhat related note for @julianakwan - the compiler you are using (gcc-4.8.5) is nearly 5 years old. You will likely see better performance from a newer compiler, if that is an option for you.

[julianakwan]

This is what I get from using find:

/homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/call_correlation_functions.py /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/call_correlation_functions_mocks.py /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/init.py /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/tests.py /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/utils.py /homes/jkwan/.local/lib/python2.7/site-packages/Corrfunc/io.py

My version of distutils is 2.7.15

I've noticed that there is a later version of gcc on this cluster (8.2.0) but the modules that I'm using has been built with 4.8.5.

[lgarrison]

Hmm, I cannot reproduce yet either. I used Python 2 inside a conda environment. pip and setup.py both worked.

[manodeep]

@julianakwan Will you please try to install the previous version Corrfunc (v2.3.2) and see if that installs successfully? That way, we will at least have an idea as to what might be the root cause.

If v2.3.2 does not install correctly either, will you please try progressively earlier versions until you get one that installs correctly.

Thanks for your patience - hopefully we can get to the bottom of this.

[julianakwan]

I can go back to 2.3.1 successfully.

In the meantime, I've just been copying over the theory and mocks directories to my python path. That seems to work also.

[manodeep]

@julianakwan This will take further digging. What is your python version?

[julianakwan]

It is Python 2.7.15

[manodeep]

Hmm that python version should come with pip. What happens if you try python -m pip install -e . ?

[julianakwan]

I get: [jkwan@usernode ~]$ python -m pip install -e . DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /cosmo_tortoise/homes/jkwan WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

[manodeep]

@julianakwan My apologies - please run that command at the Corrfunc root directory

[julianakwan]

Oh sorry!

Here it is again:

[jkwan@usernode ~/Corrfunc]$ python -m pip install -e .
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///cosmo_tortoise/homes/jkwan/Corrfunc
    ERROR: Command errored out with exit status 1:
     command: /cosmo_tortoise/software/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.15-wkqj7mbdjxbiive4b2x2fvkk7xmtxn43/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"'; __file__='"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /cosmo_tortoise/homes/jkwan/Corrfunc/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[manodeep]

Ohh good - you do have pip but the install appears to be missing setuptools. ~Can you please install setuptools with:~

wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | python - --user~

~(complete instructions are here).~

Google failed - these instructions were for an immensely old version of setuptools - please disregard

[lgarrison]

@manodeep Does "pip install setuptools" not work? I tried uninstalling setuptools then installing it from pip, seemed to work. In other words, it does not appear that setuptools is required to install setuptools!

On Wed, Feb 12, 2020 at 3:37 PM Manodeep Sinha notifications@github.com wrote:

Ohh good - you do have pip but the install appears to be missing setuptools. Can you please install setuptools with:

wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | python - --user

(complete instructions are here https://pypi.org/project/setuptools/3.0/#unix-wget).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/manodeep/Corrfunc/issues/212?email_source=notifications&email_token=ABLA7S5JBLVO44BZUXUXINTRCRMYPA5CNFSM4KPGBTXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELSJNNA#issuecomment-585406132, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLA7SZM3NJ57Z4KF45G3SDRCRMYPANCNFSM4KPGBTXA .

-- Lehman Garrison lgarrison@flatironinstitute.org Flatiron Research Fellow, Cosmology X Data Science Group Center for Computational Astrophysics, Flatiron Institute lgarrison.github.io

[manodeep]

Those were the instructions on the setuptools PyPI page, but upon further checking I see that those were for setuptools in 2014!

The latest version does not support py2 - can't seem to locate a version that does. Perhaps distutils?

I am also surprised that setuptools is causing an ImportError -- the import at the top of setup.py is protected and falls back to distutils. @lgarrison Do you think this error is occuring because of the requirement of setuptools in setup_requires?

[manodeep]

All of this trouble with python2 really makes me think that we should drop python2 support sometime in the next year (after a major release).

[lgarrison]

@lgarrison https://github.com/lgarrison Do you think this error is occuring because of the requirement of setuptools in setup_requires?

Ah, maybe so! I haven't looked at the logs in detail, though.

On Wed, Feb 12, 2020 at 3:59 PM Manodeep Sinha notifications@github.com wrote:

Those were the instructions on the setuptools PyPI page, but upon further checking I see that those were for setuptools in 2014!

The latest version does not support py2 - can't seem to locate a version that does. Perhaps distutils?

I am also surprised that setuptools is causing an ImportError -- the import at the top of setup.py is protected and falls back to distutils. @lgarrison https://github.com/lgarrison Do you think this error is occuring because of the requirement of setuptools in setup_requires?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/manodeep/Corrfunc/issues/212?email_source=notifications&email_token=ABLA7S2UQ542PTYFROHNK6TRCRPKLA5CNFSM4KPGBTXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELSLQNY#issuecomment-585414711, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLA7S44HTFXJ7K6HOM32Q3RCRPKLANCNFSM4KPGBTXA .

-- Lehman Garrison lgarrison@flatironinstitute.org Flatiron Research Fellow, Cosmology X Data Science Group Center for Computational Astrophysics, Flatiron Institute lgarrison.github.io

[manodeep]

@julianakwan Will you please retry installing Corrfunc after installing setuptools with python -m pip install setuptools (please add --user if needed).

(I am assuming the issue you were facing has been resolved but would be good to know what the resolution was)

[julianakwan]

Sorry for being slow -

Yes, I was able to install Corrfunc using setuptools. Thanks for your help!

[manodeep]

@julianakwan Thanks for the update and no worries at all. Glad the installation issue was all sorted out

[lgarrison]

Just noting that I see the setuptools error is coming from the one-liner that boostraps Corrfunc's setup.py, not setup.py itself. That is, this line from @julianakwan's report:

     command: /cosmo_tortoise/software/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.15-wkqj7mbdjxbiive4b2x2fvkk7xmtxn43/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"'; __file__='"'"'/cosmo_tortoise/homes/jkwan/Corrfunc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info

I guess this is a pip-ism? I've seen this pattern before but don't totally understand where it comes from. It's not something we wrote, right?