Building module pyirt.util.clib failed

[SiddhantAgarwal]

Just importing the package throws this error

Error compiling Cython file:

... double y0, double theta, double alpha, double beta, double c=0.0): cdef extern from "math.h": ^

/usr/local/lib/python3.6/site-packages/pyirt/util/clib.pyx:12:9: Compiler crash in AnalyseDeclarationsTransform

ModuleNode.body = StatListNode(clib.pyx:3:0) StatListNode.stats[2] = StatListNode(clib.pyx:6:0) StatListNode.stats[0] = DefNode(clib.pyx:6:0, modifiers = [...]/0, name = 'log_likelihood_2PL', np_args_idx = [...]/0, num_required_args = 5, py_wrapper_required = True, reqd_kw_flags_cname = '0') File 'Nodes.py', line 427, in analyse_declarations: StatListNode(clib.pyx:12:4, is_terminator = True) File 'Nodes.py', line 495, in analyse_declarations: CDefExternNode(clib.pyx:12:9, include_file = 'math.h')

Compiler crash traceback from this point on: File "/usr/local/lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 495, in analyse_declarations env.add_include_file(self.include_file, self.verbatim_include, late) TypeError: add_include_file() takes 2 positional arguments but 4 were given /tmp/pyximport/temp.macosx-10.13-x86_64-3.6/pyrex/pyirt/util/clib.c:1:2: error: Do not use this file, it is the result of a failed Cython compilation.

error Do not use this file, it is the result of a failed Cython compilation.

^ 1 error generated. Traceback (most recent call last): File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile extra_postargs) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/ccompiler.py", line 909, in spawn spawn(cmd, dry_run=self.dry_run) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/spawn.py", line 36, in spawn _spawn_posix(cmd, search_path, dry_run=dry_run) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix % (cmd, exit_status)) distutils.errors.DistutilsExecError: command 'clang' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/pyximport/pyximport.py", line 215, in load_module inplace=build_inplace, language_level=language_level) File "/usr/local/lib/python3.6/site-packages/pyximport/pyximport.py", line 191, in build_module reload_support=pyxargs.reload_support) File "/usr/local/lib/python3.6/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll dist.run_commands() File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/local/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/local/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions self.build_extension(ext) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension depends=ext.depends) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/ccompiler.py", line 574, in compile self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile raise CompileError(msg) distutils.errors.CompileError: command 'clang' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "IRT_data_analyser.py", line 1, in from pyirt import irt File "/usr/local/lib/python3.6/site-packages/pyirt/init.py", line 2, in from ._pyirt import irt File "/usr/local/lib/python3.6/site-packages/pyirt/_pyirt.py", line 2, in from .solver import model File "/usr/local/lib/python3.6/site-packages/pyirt/solver/model.py", line 19, in from ..util import clib, tools File "/usr/local/lib/python3.6/site-packages/pyirt/util/init.py", line 8, in from . import clib File "/usr/local/lib/python3.6/site-packages/pyximport/pyximport.py", line 458, in load_module language_level=self.language_level) File "/usr/local/lib/python3.6/site-packages/pyximport/pyximport.py", line 231, in load_module raise exc.with_traceback(tb) File "/usr/local/lib/python3.6/site-packages/pyximport/pyximport.py", line 215, in load_module inplace=build_inplace, language_level=language_level) File "/usr/local/lib/python3.6/site-packages/pyximport/pyximport.py", line 191, in build_module reload_support=pyxargs.reload_support) File "/usr/local/lib/python3.6/site-packages/pyximport/pyxbuild.py", line 102, in pyx_to_dll dist.run_commands() File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/local/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/local/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions self.build_extension(ext) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension depends=ext.depends) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/ccompiler.py", line 574, in compile self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile raise CompileError(msg) ImportError: Building module pyirt.util.clib failed: ["distutils.errors.CompileError: command 'clang' failed with exit status 1\n"]

[junchenfeng]

I think it is the problem of cython, not of pyirt.

[bryketos]

junchenfeng, do you have any advice on this error? I got this with some environments but not others. Thanks

[junchenfeng]

I ran into a similar problem earlier (although I am not sure that it is the same problem) when the user does not have root access because cython build creates and writes into /tmp folder.

I don't know what the Devops guy did to fix the issue.

PS: from the trace stack and a bit of google, I think it may also has something to do with clang installation.

[bryketos]

Thanks Junchen, I can try compiling this on an EC2 instance as root or as other users if you would like some additional testing. It has worked on one EC2 instance but not another which is puzzling. I hope I can make this pyirt package work because the other option is the mirt package for R which doesn't handle sparsity so well (convergence failures with large, sparse dataframes) and flat-out requires in-memory data structures.

[junchenfeng]

It also depends the OS version. Ubuntu and Mac are probably fine, but Centos 6 and Centos 7 are totally different beasts. If it works on EC2 instance, I bet the problem is the system environment, which is tricky.

I would suggest asking the operation manager to install Cython for python 2.7 and 3.5, then use pipenv to install pyirt. If that fails, then we will do something.

The pyirt package should not be in the business of operations.

[junchenfeng]

@SiddhantAgarwal @bryketos Our new colleague also runs into this exact problem with his new mac. We strongly suspect it is because Cython requires Xcode to be installed. As a simple test, try run a c program that imports math.h. If that fails, it means you need Xcode. For more, see here

[junchenfeng]

Guys, it is our fault. After Cython upgrade, the previous clib.pyx does not compile properly. We are in the process of fixing it

[junchenfeng]

fixed at #12