search

SciTools / cartopy

Cartopy - a cartographic python library with matplotlib support
https://scitools.org.uk/cartopy/docs/latest
BSD 3-Clause "New" or "Revised" License
1.43k stars 366 forks source link

Cartopy cannot be built on a FIPS-enabled system #1258

Closed Cynthia-Higginbotham closed 5 years ago

Cynthia-Higginbotham commented 5 years ago

Description

Cython crashes when attempting to build cython extensions to cartopy on FIPS-enabled systems because it tries to use md5 hashes internally, and FIPS-enabled systems disable md5 because it is insecure. This is a Cython issue which I have just filed (Cython fails to build code on FIPS-enabled systems cython/cython#2790), but since cartopy depends on Cython, it became a cartopy issue as well, since it cannot be built on any FIPS-enabled system.

Steps to reproduce

  1. RHEL7, FIPS-enabled kernel, python3.4 installed.
  2. pip install --prefix $PYTHONUSERBASE cartopy
  3. partway through the install, we get the crash and abort seen below:

Traceback

    building 'cartopy._crs' extension
    /usr/bin/gcc -Wno-unused-result -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python34/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/opt/rh/rh-python34/root/usr/include -I/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/numpy/core/include -I/opt/rh/rh-python34/root/usr/include/python3.4m -c lib/cartopy/_crs.c -o build/temp.linux-x86_64-3.4/lib/cartopy/_crs.o
    In file included from /opt/rh/rh-python34/root/usr/include/numpy/ndarraytypes.h:1761:0,
                     from /opt/rh/rh-python34/root/usr/include/numpy/ndarrayobject.h:17,
                     from /opt/rh/rh-python34/root/usr/include/numpy/arrayobject.h:4,
                     from lib/cartopy/_crs.c:614:
    /opt/rh/rh-python34/root/usr/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it by " \
      ^
    gcc -pthread -shared -L/opt/rh/rh-python34/root/usr/lib64-Wl,-z,relro build/temp.linux-x86_64-3.4/lib/cartopy/_crs.o -L/opt/rh/rh-python34/root/usr/lib64 -L/opt/rh/rh-python34/root/usr/lib64 -lproj -lpython3.4m -o build/lib.linux-x86_64-3.4/cartopy/_crs.cpython-34m.so
    cythoning lib/cartopy/geodesic/_geodesic.pyx to lib/cartopy/geodesic/_geodesic.c
    /tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bgk3hgjk/cartopy/lib/cartopy/geodesic/_geodesic.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    cdef extern from *:
        cdef object __pyx_test_dep(object)

    @cname('__pyx_TestClass')
    cdef class TestClass(object):
        ^
    ------------------------------------------------------------

    TestClass:8:5: Compiler crash in AnalyseDeclarationsTransform

    ModuleNode.body = StatListNode(TestClass:4:0)
    StatListNode.stats[1] = CnameDecoratorNode(TestClass:8:5,
        cname = '__pyx_TestClass')

    Compiler crash traceback from this point on:
      File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1635, in visit_CClassDefNode
        self._inject_pickle_methods(node)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1703, in _inject_pickle_methods
        checksum = '0x%s' % hashlib.md5(' '.join(all_members_names).encode('utf-8')).hexdigest()[:7]
    ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
                endpoint(s).

            """

            cdef int n_points, i
            cdef double[:, :] pts, orig_pts
                       ^
    ------------------------------------------------------------

    lib/cartopy/geodesic/_geodesic.pyx:120:20: Compiler crash in AnalyseDeclarationsTransform

    ModuleNode.body = StatListNode(_geodesic.pyx:25:0)
    StatListNode.stats[8] = StatListNode(_geodesic.pyx:46:5)
    StatListNode.stats[0] = CClassDefNode(_geodesic.pyx:46:5,
        as_name = 'Geodesic',
        class_name = 'Geodesic',
        doc = '\n    Define an ellipsoid on which to solve geodesic problems.\n\n    ',
        module_name = '',
        visibility = 'private')
    CClassDefNode.body = StatListNode(_geodesic.pyx:47:4)
    StatListNode.stats[7] = CompilerDirectivesNode(_geodesic.pyx:91:4)
    CompilerDirectivesNode.body = StatListNode(_geodesic.pyx:91:4)
    StatListNode.stats[0] = DefNode(_geodesic.pyx:91:4,
        doc = '\n        Solve the direct geodesic problem where the length of the geodesic is\n        specified in terms of distance.\n\n        Can accept and broadcast length 1 arguments. For example, given a\n        single start point and distance, an array of different azimuths can be\n        supplied to locate multiple endpoints.\n\n        Parameters\n        ----------\n        points: array_like, shape=(n *or* 1, 2)\n            The starting longitude-latitude point(s) from which to travel.\n\n        azimuths: float or array_like with shape=(n, )\n            List of azimuth values (degrees).\n\n        distances : float or array_like with shape(n, )\n            List of distances values (metres).\n\n        Returns\n        -------\n        `numpy.ndarray`, shape=(n, 3)\n            The longitudes, latitudes, and forward azimuths of the located\n            endpoint(s).\n\n        ',
        modifiers = [...]/0,
        name = 'direct',
        needs_closure = True,
        np_args_idx = [...]/0,
        num_required_args = 4,
        outer_attrs = [...]/2,
        py_wrapper_required = True,
        reqd_kw_flags_cname = '0')
    File 'Nodes.py', line 431, in analyse_declarations: StatListNode(_geodesic.pyx:92:8,
        is_terminator = True)
    File 'Nodes.py', line 1329, in analyse_declarations: CVarDefNode(_geodesic.pyx:120:13,
        modifiers = [...]/0,
        visibility = 'private')
    File 'Nodes.py', line 1088, in analyse: MemoryViewSliceTypeNode(_geodesic.pyx:120:20,
        name = 'memoryview')

    Compiler crash traceback from this point on:
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/Nodes.py", line 1084, in analyse
        axes_specs = MemoryView.get_axes_specs(env, self.axes)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/MemoryView.py", line 533, in get_axes_specs
        cythonscope.load_cythonscope()
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/CythonScope.py", line 113, in load_cythonscope
        self, cython_scope=self)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/UtilityCode.py", line 208, in declare_in_scope
        tree = self.get_tree(entries_only=True, cython_scope=cython_scope)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/UtilityCode.py", line 186, in get_tree
        assert not err, err
    AssertionError:
    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    cdef extern from *:
        cdef object __pyx_test_dep(object)

    @cname('__pyx_TestClass')
    cdef class TestClass(object):
        ^
    ------------------------------------------------------------

    TestClass:8:5: Compiler crash in AnalyseDeclarationsTransform

    ModuleNode.body = StatListNode(TestClass:4:0)
    StatListNode.stats[1] = CnameDecoratorNode(TestClass:8:5,
        cname = '__pyx_TestClass')

    Compiler crash traceback from this point on:
      File "Cython/Compiler/Visitor.py", line 180, in Cython.Compiler.Visitor.TreeVisitor._visit
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1635, in visit_CClassDefNode
        self._inject_pickle_methods(node)
      File "/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1703, in _inject_pickle_methods
        checksum = '0x%s' % hashlib.md5(' '.join(all_members_names).encode('utf-8')).hexdigest()[:7]
    ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
    building 'cartopy.geodesic._geodesic' extension
    creating build/temp.linux-x86_64-3.4/lib/cartopy/geodesic
    /usr/bin/gcc -Wno-unused-result -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python34/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/opt/rh/rh-python34/root/usr/include -I/tmp/pip-build-env-za1cx7s2/lib64/python3.4/site-packages/numpy/core/include -I/opt/rh/rh-python34/root/usr/include/python3.4m -c lib/cartopy/geodesic/_geodesic.c -o build/temp.linux-x86_64-3.4/lib/cartopy/geodesic/_geodesic.o
    lib/cartopy/geodesic/_geodesic.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
     #error Do not use this file, it is the result of a failed Cython compilation.
      ^
    error: command '/usr/bin/gcc' failed with exit status 1

    ----------------------------------------
Command "/opt/rh/rh-python34/root/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-bgk3hgjk/cartopy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-s1x4au0e/install-record.txt --single-version-externally-managed --prefix /usr/appsdev/autoadfc/share/python --compile" failed with error code 1 in /tmp/pip-install-bgk3hgjk/cartopy/
Full environment definition ### Operating system RedHat EL7 ### Cartopy version Cartopy 0.17.0, the current version on PyPi ### conda list N/A ### pip list ``` cycler 0.10.0 Cython 0.29.2 docutils 0.12 Jinja2 2.7.3 kiwisolver 1.0.1 MarkupSafe 0.23 matplotlib 2.2.3 nose 1.3.4 numpy 1.15.2 pip 18.0 Pygments 1.6 pyparsing 2.2.2 pyshp 2.0.1 python-dateutil 2.7.3 pytz 2018.5 scipy 0.13.3 setuptools 11.3.1 Shapely 1.6.4.post2 simplejson 3.6.5 six 1.11.0 Sphinx 1.2.2 SQLAlchemy 0.9.8 virtualenv 1.11.6 wheel 0.32.0 ```
ajdawson commented 5 years ago

As you say, this is an issue with the Cython compiler rather than any code specific to Cartopy cython/cython#2790. What is not quite clear to me is whether this issue only occurs during processing of the pyx file to turn it into C code, or if the C code is trying to use a forbidden function? If it is the former then distributing the C code could be a solution to this, see #1035 for some related discussion.

scoder commented 5 years ago

Including the generated C file(s) in source distributions (and making the recompilation optional) would solve this problem.

pelson commented 5 years ago

@Cynthia-Higginbotham - this issue was closed because #1264 means that users will no longer need Cython when installing source distributions (e.g. from pypi). Thanks to @scoder for the advice.

This will be part of the next release (v0.18).