Closed moorepants closed 7 years ago
This seems to be the main issue. For some reason when Cython checks for cygwin it calls:
ipdb> check_output(['gcc', '-dumpmachine'])
*** FileNotFoundError: [WinError 2] The system cannot find the file specified
and that gets and error but if you have mingw installed (like we do in our env) I can call this at the command line and it works:
(codegen17) C:\Users\Jason>gcc -dumpmachine
x86_64-w64-mingw32
Looks like distutils needs a patch:
out_string = check_output(['gcc', '-dumpmachine'], shell=True)
And this looks like some Aaron has already investigated:
Regarding the win64 packages, I thought they were covered here: https://github.com/sympy/scipy-2017-codegen-tutorial/blob/master/.drone.yml#L15
23: That is "intended" (Euler forward deviates)
40: Nice to see you came to the same conclusion regarding the cygwincompiler (https://github.com/sympy/scipy-2017-codegen-tutorial/issues/2#issuecomment-307812559). Do you know how to include the patch in our conda package?
+1 for skipping theano
Geez is that still an issue? I thought Continuum moved everything from .bat over to .exe.
I don't remember if there are some better workarounds for this. Might be worth some further investigation.
For theano, let's skip it. It isn't even really printed anyway. I couldn't even test it on my own computer because I couldn't install theano on the plane (and plus it's dumb to make theano a dependency of the tutorial just for this).
How does the distutils code get called? Can we monkeypatch it?
Interesting that the %%cython
magic command compiles fine but the pyximport doesn't. They must not follow the same code paths.
Maybe related to the fact that pyximport uses Cython.Distutils
and the cell magic doesn't.
This shows the win-64 packages: https://anaconda.org/SymPy/scipy2017codegen, not sure why it failed to download. Was it maybe because of the channel name: sympy ==3.5
? What does it mean to version the channel?
What error is this in reference to?
With respect to:
WARNING: The remote server could not find the noarch directory for the
requested channel with url: https://conda.anaconda.org/sympy ==3.5
It is possible you have given conda an invalid channel. Please double-check
your conda configuration using `conda config --show`.
If the requested url is in fact a valid conda channel, please request that the
channel administrator create `noarch/repodata.json` and associated
`noarch/repodata.json.bz2` files, even if `noarch/repodata.json` is empty.
$ mkdir noarch
$ echo '{}' > noarch/repodata.json
$ bzip2 -k noarch/repodata.json
Do you know how to include the patch in our conda package?
I think we'd have to make a custom disutils package on the sympy channel that had priority in downloading. Then we could include a patch via the meta.yml file.
Compiler: Using msvc from: http://landinghub.visualstudio.com/visual-cpp-build-tools worked for me. But then we should try to make people download this before the conference (4 GiB download size).
The channel should not have been versioned. A silly mistake by me. I just changed it.
Regarding pyximport: what examples are failing? (note that I used a .pyxbld file)
@moorepants I just changed so that msvc
is the default:
https://github.com/sympy/scipy-2017-codegen-tutorial/blob/d7d213aea043cbba623b7f116df3b87dbfccd2d0/scipy2017codegen/template.pyxbld#L3
We could introduce a conda package called e.g. scipy2017codegen_mingw
which sets that environment variable to "mingw" (the conda package would contain bat files as per: https://conda.io/docs/using/envs.html#more-notes-on-environment-variable-scripts). But this is all very complicated :/
Let's try to avoid patching the standard library with conda if possible. I know there is a way to change the default compiler with some config file (Continuum has a package that does this, but I forgot what it's called).
I think everything here is fixed, no?
Except I don't know the status of the disutils patch needs? @bjodah Did you work around that?
The distutils patch was needed for Python 3 when using mingw32, I changed the default to msvc
.
gcc
without shell=True
. Monkey-patching the relatively short python function: distutils.cygwinccompiler.is_cygwingcc
comes to mind. I actually think I could make that work.I think monkey patching is fine for this purpose if we need to do it. The other option would be to ship a patched distutils in the sympy conda channel. I'm not completely sure how to monkey patch it without shipping a custom distutils. We could copy the relevant distutils modules into our scipy2017codegen package and then import from that?
I'm thinking something dead simple:
>>> import distutils.cygwinccompiler
>>> distutils.cygwinccompiler.is_cygwingcc()
...
TraceBack ...
>>> distutils.cygwinccompiler.is_cygwingcc = lambda: True
>>> distutils.cygwinccompiler.is_cygwingcc()
True
the monkeypatching would be performed in pyxbld when $SCIPY2017CODEGEN_COMPILER == mingw32
. (I'm trying this right now)
That's a nice way to do it.
Thought you were going to bed :)
I was... and now I am :)
Tried it, but it didn't work unfortunately (it's a bit of whack-a-mole):
~\Anaconda2\envs\codegen17\lib\distutils\cygwinccompiler.py in __init__(self, verbose, dry_run, force)
155 # Include the appropriate MSVC runtime library if Python was built
156 # with MSVC 7.0 or later.
--> 157 self.dll_libraries = get_msvcr()
158
159 def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts):
~\Anaconda2\envs\codegen17\lib\distutils\cygwinccompiler.py in get_msvcr()
84 return ['msvcr100']
85 else:
---> 86 raise ValueError("Unknown MS Compiler version %s " % msc_ver)
87
88
ImportError: Building module ode_cython_311c73e7fc failed: ['ValueError: Unknown MS Compiler version 1900 \n']
As a side note: there is a traceback from _msvccompiler.py in distutils when trying to use %%cython magic in our codegen17 environment, but it works in Anaconda2. :/
Not sure mingw is worth the trouble..
Is this conda-forge
's python? It was fixed here, https://github.com/conda-forge/python-feedstock/pull/143
Thanks @isuruf, that got me a bit further:
c:/users/user/anaconda2/envs/codegen17/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: c:\users\user\scipy-2017-codegen-tutorial\notebooks\ode_c_c847e93dc6.cp36-win_amd64.def:1: syntax error
...
collect2.exe: error: ld returned 1 exit status
contents of ode_c_c847e93dc6.cp36-win_amd64.def
:
LIBRARY ode_c_c847e93dc6.cp36-win_amd64.pyd
EXPORTS
PyInit_ode_c_c847e93dc6
(I still need to monkey-patch distutils.cygwinccompiler.is_cygwingcc
to get this far)
For some reason I also had to manually add distutils/distutils.cfg
:
[build]
compiler = ming32
to the conda environment (it was present in the root environment, this may have been caused by me manually upgrading python).
Thanks for all help. But I'm closing this for now (I need to focus more on content and less on mingw :) ).
@ git commit 7bf20ce217dd032a519fd86effd41e9890d2cf1c tested on Windows 7 with the latest Anaconda.
It is possible you have given conda an invalid channel. Please double-check your conda configuration using
conda config --show
.If the requested url is in fact a valid conda channel, please request that the channel administrator create
noarch/repodata.json
and associatednoarch/repodata.json.bz2
files, even ifnoarch/repodata.json
is empty. $ mkdir noarch $ echo '{}' > noarch/repodata.json $ bzip2 -k noarch/repodata.jsonC:\Users\Jason\Anaconda3\envs\codegen17\lib\site-packages\ipykernel_launcher.py:4: RuntimeWarning: overflow encountered in double_scalars after removing the cwd from sys.path. C:\Users\Jason\Anaconda3\envs\codegen17\lib\site-packages\ipykernel_launcher.py:11: RuntimeWarning: invalid value encountered in add
This is added back by InteractiveShellApp.init_path()
Compiling C:\Users\Jason\Desktop\scipy-2017-codegen-tutorial-master\notebooks\ode_cython_9a390683f6.pyx because it changed. [1/1] Cythonizing C:\Users\Jason\Desktop\scipy-2017-codegen-tutorial-master\notebooks\ode_cython_9a390683f6.pyx
FileNotFoundError Traceback (most recent call last) ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in load_module(name, pyxfilename, pyxbuild_dir, is_package, build_inplace, language_level, so_path) 215 so_path = build_module(module_name, pyxfilename, pyxbuild_dir, --> 216 inplace=build_inplace, language_level=language_level) 217 mod = imp.load_dynamic(name, so_path)
~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in build_module(name, pyxfilename, pyxbuild_dir, inplace, language_level) 191 inplace=inplace, --> 192 reload_support=pyxargs.reload_support) 193 assert os.path.exists(so_path), "Cannot find: %s" % so_path
~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyxbuild.py in pyx_to_dll(filename, ext, force_rebuild, build_in_temp, pyxbuild_dir, setup_args, reload_support, inplace) 101 obj_build_ext = dist.get_command_obj("build_ext") --> 102 dist.run_commands() 103 so_path = obj_build_ext.get_outputs()[0]
~\Anaconda3\envs\codegen17\lib\distutils\dist.py in run_commands(self) 954 for cmd in self.commands: --> 955 self.run_command(cmd) 956
~\Anaconda3\envs\codegen17\lib\distutils\dist.py in run_command(self, command) 973 cmd_obj.ensure_finalized() --> 974 cmd_obj.run() 975 self.have_run[command] = 1
~\Anaconda3\envs\codegen17\lib\site-packages\Cython\Distutils\old_build_ext.py in run(self) 184 --> 185 _build_ext.build_ext.run(self) 186
~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in run(self) 307 dry_run=self.dry_run, --> 308 force=self.force) 309 customize_compiler(self.compiler)
~\Anaconda3\envs\codegen17\lib\distutils\ccompiler.py in new_compiler(plat, compiler, verbose, dry_run, force) 1030 # argument. -> 1031 return klass(None, dry_run, force) 1032
~\Anaconda3\envs\codegen17\lib\distutils\cygwinccompiler.py in init(self, verbose, dry_run, force) 300 --> 301 if is_cygwingcc(): 302 raise CCompilerError(
~\Anaconda3\envs\codegen17\lib\distutils\cygwinccompiler.py in is_cygwingcc() 410 '''Try to determine if the gcc that would be used is from cygwin.''' --> 411 out_string = check_output(['gcc', '-dumpmachine']) 412 return out_string.strip().endswith(b'cygwin')
~\Anaconda3\envs\codegen17\lib\subprocess.py in check_output(timeout, *popenargs, *kwargs) 335 return run(popenargs, stdout=PIPE, timeout=timeout, check=True, --> 336 **kwargs).stdout 337
~\Anaconda3\envs\codegen17\lib\subprocess.py in run(input, timeout, check, *popenargs, *kwargs) 402 --> 403 with Popen(popenargs, **kwargs) as process: 404 try:
~\Anaconda3\envs\codegen17\lib\subprocess.py in init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors) 706 errread, errwrite, --> 707 restore_signals, start_new_session) 708 except:
~\Anaconda3\envs\codegen17\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session) 989 cwd, --> 990 startupinfo) 991 finally:
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)