Cython Build - Not installing to tmp build directory

[jlaura]

I am trying to build a Cython wrapper to a C++ library using conda. The source project has a working setup.py that I am using (python setup.py install) to build and install into a condo-env. All pytest tests then run successfully.

When I switch to doing a conda build recipe/, the build is successful, but the tests fails with an import error for cycsm (the name of the project). I tried incrementing the build number, cleaning conda, and purging the build cache thinking that that could be the issue - no joy. My final thought was that this might have to do with python3 imports, but fooling around with those was also unsuccessful. Any insights appreciated as I can not get a Cython built package to pass import tests.

The tmp directory that is created with the build has an _t_env created with python. Checking out the site-packages in that env. the cython compiled package is not installed, e.g.,:

>>> import sys
>>> sys.path
['', '/Users/me/miniconda3/conda-bld/cycsm_1510524942379/_t_env/lib/python35.zip', '/Users/me/miniconda3/conda-bld/cycsm_1510524942379/_t_env/lib/python3.5', '/Users/me/miniconda3/conda-bld/cycsm_1510524942379/_t_env/lib/python3.5/plat-darwin', '/Users/me/miniconda3/conda-bld/cycsm_1510524942379/_t_env/lib/python3.5/lib-dynload', '/Users/me/miniconda3/conda-bld/cycsm_1510524942379/_t_env/lib/python3.5/site-packages']

and

$ ls /Users/me/miniconda3/conda-bld/cycsm_1510524942379/_t_env/lib/python3.5/site-packages
README                  pip
__pycache__             pip-9.0.1-py3.5.egg-info
certifi                 pkg_resources
certifi-2017.07.27.1-py3.5.egg-info setuptools
easy_install.py             setuptools-36.6.0-py3.5.egg-info
numpy                   wheel
numpy-1.13.3-py3.5.egg-info     wheel-0.30.0-py3.6.egg-info

Conda version: 2.1.17 (freshly 'updated' via condo-forge)

Files used for the build:

meta.yml

{% set version = "0.1.0" %}

package:
  name: cycsm
  version: {{ version }}

source:
  path: /Users/jlaura/github/usgs_csm/CSM-CyCSM
  #git_url: https://github.com/USGS-Astrogeology/CSM-CyCSM
  #git_rev: master

build:
  number: 1

detect_binary_files_with_prefix: true

requirements:
  build:
    - python >=3.*
    - setuptools
    - numpy >=1.10
    - cython >=0.23
    - libcsm
  run:
    - python >=3.*
    - numpy >=1.10
    - libcsm

test:
  imports:
    - cycsm

about:
  home: https://github.com/USGS-Astrogeology/CSM-CyCSM
  license: Unlicense

build.sh

#!/bin/bash

if [ `uname` == Darwin ]; then
  export MACOSX_DEPLOYMENT_TARGET=10.9;
fi

$PYTHON setup.py install --single-version-externally-managed --record=/tmp/record.txt

setup.py

import os
import pkg_resources
import sys
import sysconfig
from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize
from Cython.Distutils import build_ext

# Look for the csmapi headers in the standard location
incdir = os.path.dirname(sysconfig.get_path('include'))
incdir = os.path.join(incdir, 'csm')

INCLUDE_DIRS = [incdir]
LIBRARY_DIRS = []  # This assumes that libcsmapi is installed in a standard place
LIBRARIES = ['csmapi']
COMPILE_ARGS = ['-g', '-std=c++11'] #, '-std=c++11']

def generate_extension(path_name, sources):
    return Extension(path_name,
                sources=sources,
                extra_compile_args=COMPILE_ARGS,
                language='c++',
                include_dirs=INCLUDE_DIRS,
                runtime_library_dirs=LIBRARY_DIRS,
                library_dirs=LIBRARY_DIRS,
                libraries=LIBRARIES)

if sys.platform == 'darwin':
    COMPILE_ARGS.append('-mmacosx-version-min=10.9')

# Create the extensions
extensions = [generate_extension('cycsm.isd', ['cycsm/isd.pyx']),
              generate_extension('cycsm.version', ['cycsm/version.pyx']),
              generate_extension('cycsm.csm', ['cycsm/csm.pyx']),
              generate_extension('cycsm.model', ['cycsm/model.pyx']),
              generate_extension('cycsm.correlationmodel', ['cycsm/correlationmodel.pyx']),
              generate_extension('cycsm.rastergm', ['cycsm/rastergm.pyx'])]

setup(
    name='cycsm',
    version='0.1.0',
    cmdclass = {'build_ext':build_ext},
    ext_modules=cythonize(extensions),
    description='Cython wrapper to the Community Sensor Model API.',
    author='Jay Laura',
    packages = find_packages(),
    include_package_data=True)

[jlaura]

Doing some additional digging, it looks like conda build is placing the built module into /lib/python3.6/site-packages, while the tests are being run using python 3.5 (also inside of the build env.).

[msarahan]

yes, by adding the python version constraint in meta.yaml, you're overriding some of conda-build's stuff about which python version it uses. If you remove >=3.* from your python specs, it should work better.

The equivalent syntax to enforce python-3-only is:

build:
    skip: True  # [not py3k]

Also, conda-build 3 may do better with this, but I'm not sure. Conda-build 3 is not yet available on conda-forge, so you'll need to get it from defaults.

conda install conda-build=3.*

should do the trick (the version spec forces it to look for the right version, even though conda-forge is a higher priority channel)

[jlaura]

@msarahan Success. Thanks! Is this documented somewhere? The desired behavior? Or a bug? Not sure if this should be closed or left open.

[github-actions[bot]]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   • What OS and version you reproduced the issue on
   • What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!