Travis Fails to build Astroscrappy

[simontorres]

I'm having trouble with the builds in travis, "apparently" due to astroscrappy. I say apparently because I don't think astroscrappy itself is the problem but I know it is the point of failure for travis at least. Running locally it works OK.

Any feedback is appreciated. I apologize in advance if this issue does not correspond here.

My .travis.yml file is like this:

language: python
python:
- '3.5'
- '3.6'

branches:
  only:
  - master
  - development

install:
- sudo apt-get update

- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;

- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- hash -r
- conda config --set always_yes yes --set changeps1 no

- conda config --add channels http://ssb.stsci.edu/astroconda
- conda update -q conda

- conda info -a

- conda env create -q python=$TRAVIS_PYTHON_VERSION -f environment.yml
- source activate goodman

- python setup.py test
- python setup.py install

# removed unnecessary details at the bottom.

The build fails at conda env create -q python=$TRAVIS_PYTHON_VERSION -f environment.yml

and the environment.yml file is:

name: goodman
channels:
  - astroconda
dependencies:
  - numpy
  - pandas
  - matplotlib
  - scipy
  - cython
  - astropy>3
  - pip
  - pip:
    - ccdproc
    - coveralls
    - astroplan
    - sphinx
    - sphinxcontrib.napoleon

Some of the things I've tried so far to fix it.

• Specify latest version of cython
• Isolate ccdproc installation, thefore astroscrappy's as a dependency
• Isolate astroscrappy installation.

The relevan traceback is below and the full build is at https://travis-ci.org/soar-telescope/goodman/jobs/407313483

  creating build/temp.linux-x86_64-3.7/astroscrappy
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/_compiler.c -o build/temp.linux-x86_64-3.7/astroscrappy/_compiler.o
  gcc -pthread -shared -B /home/travis/miniconda/envs/goodman/compiler_compat -L/home/travis/miniconda/envs/goodman/lib -Wl,-rpath=/home/travis/miniconda/envs/goodman/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/astroscrappy/_compiler.o -o build/lib.linux-x86_64-3.7/astroscrappy/_compiler.cpython-37m-x86_64-linux-gnu.so
  building 'astroscrappy.utils.median_utils' extension
  creating build/temp.linux-x86_64-3.7/astroscrappy/utils
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/median_utils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/median_utils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/medutils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/medutils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -shared -B /home/travis/miniconda/envs/goodman/compiler_compat -L/home/travis/miniconda/envs/goodman/lib -Wl,-rpath=/home/travis/miniconda/envs/goodman/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/astroscrappy/utils/median_utils.o build/temp.linux-x86_64-3.7/astroscrappy/utils/medutils.o -o build/lib.linux-x86_64-3.7/astroscrappy/utils/median_utils.cpython-37m-x86_64-linux-gnu.so -g -fopenmp
  building 'astroscrappy.utils.image_utils' extension
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/image_utils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/image_utils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/imutils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/imutils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -shared -B /home/travis/miniconda/envs/goodman/compiler_compat -L/home/travis/miniconda/envs/goodman/lib -Wl,-rpath=/home/travis/miniconda/envs/goodman/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/astroscrappy/utils/image_utils.o build/temp.linux-x86_64-3.7/astroscrappy/utils/imutils.o -o build/lib.linux-x86_64-3.7/astroscrappy/utils/image_utils.cpython-37m-x86_64-linux-gnu.so -g -fopenmp
  building 'astroscrappy.astroscrappy' extension
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/astroscrappy.c -o build/temp.linux-x86_64-3.7/astroscrappy/astroscrappy.o -Wp,-w -Wno-unused-function
  astroscrappy/astroscrappy.c: In function ‘__Pyx__ExceptionSave’:
  astroscrappy/astroscrappy.c:27186:19: error: ‘PyThreadState’ has no member named ‘exc_type’
       *type = tstate->exc_type;
                     ^
  astroscrappy/astroscrappy.c:27187:20: error: ‘PyThreadState’ has no member named ‘exc_value’
       *value = tstate->exc_value;
                      ^
  astroscrappy/astroscrappy.c:27188:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       *tb = tstate->exc_traceback;
                   ^
  astroscrappy/astroscrappy.c: In function ‘__Pyx__ExceptionReset’:
  astroscrappy/astroscrappy.c:27195:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  astroscrappy/astroscrappy.c:27196:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  astroscrappy/astroscrappy.c:27197:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  astroscrappy/astroscrappy.c:27198:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = type;
             ^
  astroscrappy/astroscrappy.c:27199:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = value;
             ^
  astroscrappy/astroscrappy.c:27200:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = tb;
             ^
  astroscrappy/astroscrappy.c: In function ‘__Pyx__GetException’:
  astroscrappy/astroscrappy.c:27255:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  astroscrappy/astroscrappy.c:27256:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  astroscrappy/astroscrappy.c:27257:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  astroscrappy/astroscrappy.c:27258:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = local_type;
             ^
  astroscrappy/astroscrappy.c:27259:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = local_value;
             ^
  astroscrappy/astroscrappy.c:27260:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = local_tb;
             ^
  astroscrappy/astroscrappy.c: In function ‘__Pyx__ExceptionSwap’:
  astroscrappy/astroscrappy.c:27282:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  astroscrappy/astroscrappy.c:27283:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  astroscrappy/astroscrappy.c:27284:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  astroscrappy/astroscrappy.c:27285:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = *type;
             ^
  astroscrappy/astroscrappy.c:27286:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = *value;
             ^
  astroscrappy/astroscrappy.c:27287:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = *tb;
             ^
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for astroscrappy
  Running setup.py clean for astroscrappy

[krachyon]

I've run into a similar problem (same compilation errors) when I tried to install astroscrappy from pip after upgrading from python 3.6 to 3.7 Using the current master branch and running python3 setup.py build works fine. So maybe a new release to pypi including a more recent version of astroscrappy.pyx (cython spits out astroscrappy.c from it) would fix the issue

[simontorres]

I solved this problem temporarily by fixing the version of cython to 0.26.1.

This is not the best solution and I'm all ears to hear for a better one.

The version that it tries to use when it fails is 0.28

[tepickering]

can confirm that master works, but the version from pip does not. my travis workaround for this is to define:

PIP_DEPENDENCIES='git+https://github.com/astropy/astroscrappy.git'

so that pip will grab and install master. hopefully when pypi is updated, the conda version will follow shortly thereafter...

[mwcraig]

I think the underlying issue is cython/cython#1955

The latest Cython 0.29 works but only if I build from the current master of astroscrappy.

@cmccully -- would you be willing to do a new release of astroscrappy? If so, then updating helpers to 2.0.7 (the most recent release in the 2.x series) might also be helpful but building works without it.

To clarify a bit, it will not be possible to build astroscrappy on Python 3.7 without a new release (actually, just thought of a potential fix, will try in a bit, but a new release would simplify things).

[wtgee]

Would love to see this resolved. Has been frustrating my builds for a while now (it is being pulled in as part of ccdproc).

[cmccully]

@mwcraig I'm happy to release a new version. Am correctly understanding that I do not need to change anything in the code itself?

[mwcraig]

@cmccully -- I think that is the case (the current master has helpers 2.0.6 I think). Pinging @bsipocz in case there is a reason not to do a release now. There may be another one needed shortly to pull in some additional helpers changes/fixes for newer sphinx versions but that should be straightforward.

[bsipocz]

The 2.0.x helpers are currently known not to be fully sphinx 1.8 compatible, and there may be some issues with the latest pytest versions, too (there are certainly issues with the LTS astropy release, I don't yet see whether any of those would affect the helpers or not).

But if travis is all green (do a rerun to be sure, lots of stuff happened in the last 17 days), go ahead and do the release. If any serious fix comes up from the package-template/astropy-helpers side, we'll issue PRs to all affiliated packages in the coming weeks anyway.

[cmccully]

Sorry for the long delay. I just updated to the new version of astropy helpers and pushed 1.0.6 to pypi. @mwcraig and @wtgee please let me know if this fixes your install issue.

[simontorres]

I removed the cython version constraint on my builds and it works now, but travis says it downloaded astroscrappy 1.0.8?

Collecting astroscrappy (from ccdproc->-r /home/travis/build/soar-telescope/goodman_pipeline/condaenv._ai7pv0n.requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/cd/a3/0bbfcb2d22d33dc7389b5c9e61419660f7ac766dc4c80236faf441b33734/astroscrappy-1.0.8.tar.gz (449kB)

[cmccully]

Yeah. That is my bad. I saw a very minor bug in the manifest file and pypi won't let you overwrite versions so I bumped it 1.0.8 (I made two mistakes).

[wtgee]

Well I'm reporting a success for an install via pip but still see some of the numpy errors in the output (this is on my local). An import astroscrappy works after that but haven't tested much else. I can open a separate issue if unrelated.

Collecting astroscrappy
  Using cached https://files.pythonhosted.org/packages/cd/a3/0bbfcb2d22d33dc7389b5c9e61419660f7ac766dc4c80236faf441b33734/astroscrappy-1.0.8.tar.gz
Collecting astropy (from astroscrappy)
  Using cached https://files.pythonhosted.org/packages/36/23/a02c0eae427c0cad296c6c8be78d95a9638f4ec149a3d7249ae824e1fc11/astropy-3.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting numpy (from astroscrappy)
  Using cached https://files.pythonhosted.org/packages/38/39/f73e104d44f19a6203e786d5204532e214443ea2954917b27f3229e7639b/numpy-1.15.4-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: astroscrappy
  Running setup.py bdist_wheel for astroscrappy: started
  Running setup.py bdist_wheel for astroscrappy: finished with status 'error'
  Complete output from command /home/wtgee/miniconda3/envs/test-astroscrappy/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-3k62_gaj/astroscrappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-sh6zuxgz --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/cython_version.py -> build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/conftest.py -> build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/version.py -> build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/_astropy_init.py -> build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/__init__.py -> build/lib.linux-x86_64-3.7/astroscrappy
  creating build/lib.linux-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/test_utils.py -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/fake_data.py -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/test_cleaning.py -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/__init__.py -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/test_astroscrappy.py -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/setup_package.py -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  creating build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/__init__.py -> build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/setup_package.py -> build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/_compiler.c -> build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/astroscrappy.c -> build/lib.linux-x86_64-3.7/astroscrappy
  copying astroscrappy/utils/medutils.c -> build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/imutils.c -> build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/image_utils.c -> build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/median_utils.c -> build/lib.linux-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/tests/coveragerc -> build/lib.linux-x86_64-3.7/astroscrappy/tests
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-3k62_gaj/astroscrappy/setup.py", line 148, in <module>
      **package_info
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/site-packages/setuptools/__init__.py", line 143, in setup
      return distutils.core.setup(**attrs)
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 188, in run
      self.run_command('build')
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/wtgee/miniconda3/envs/test-astroscrappy/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-3k62_gaj/astroscrappy/astropy_helpers/astropy_helpers/commands/build_ext.py", line 296, in run
      np_include = get_numpy_include_path()
    File "/tmp/pip-install-3k62_gaj/astroscrappy/astropy_helpers/astropy_helpers/utils.py", line 89, in get_numpy_include_path
      import numpy
  ModuleNotFoundError: No module named 'numpy'

  ----------------------------------------
  Running setup.py clean for astroscrappy
Failed to build astroscrappy
Installing collected packages: numpy, astropy, astroscrappy
  Running setup.py install for astroscrappy: started
    Running setup.py install for astroscrappy: finished with status 'done'
Successfully installed astropy-3.1 astroscrappy-1.0.8 numpy-1.15.4

[cmccully]

Ok. This is dumb but numpy have to be installed before you pip install astroscrappy. I'm not entirely sure how to fix this. @bsipocz or @mwcraig have you seen anything like this before?

[astrofrog]

I think you can fix it by including setup_requires=['numpy'] in the setup call in setup.py.

[cmccully]

Ah. Thanks @astrofrog. We were using setup_requires, but it didn't get plumbed correctly to the actual setup() call. Whoops.

[saimn]

I think this one was fixed by the latest release?

[cmccully]

Ok. Will close for now.