Jason's Windows Notes

[moorepants]

@ git commit 7bf20ce217dd032a519fd86effd41e9890d2cf1c tested on Windows 7 with the latest Anaconda.

• We need to make it explicit which environment.yml file windows users should use.
• There are no scipy2017codegen win-64 packages in the conda channel. It is likely to do with this:

  
  WARNING: The remote server could not find the noarch directory for the
  requested channel with url: https://conda.anaconda.org/sympy ==3.5

It is possible you have given conda an invalid channel. Please double-check your conda configuration using conda config --show.

If the requested url is in fact a valid conda channel, please request that the channel administrator create noarch/repodata.json and associated noarch/repodata.json.bz2 files, even if noarch/repodata.json is empty. $ mkdir noarch $ echo '{}' > noarch/repodata.json $ bzip2 -k noarch/repodata.json

# code printers

- I get this error on the theano print: https://pastebin.com/KKBgrydB

# 23-ordinary-differential-equations

Runs but with this odeint warning:

C:\Users\Jason\Anaconda3\envs\codegen17\lib\site-packages\ipykernel_launcher.py:4: RuntimeWarning: overflow encountered in double_scalars after removing the cwd from sys.path. C:\Users\Jason\Anaconda3\envs\codegen17\lib\site-packages\ipykernel_launcher.py:11: RuntimeWarning: invalid value encountered in add

This is added back by InteractiveShellApp.init_path()

# 25-chemical-kinetics-intro

Runs great.

# 32-chemical-kinetics-symbolic-contruction

Runs great

# 40-chemical-kinetics-cython

Fails on cell 5:

Compiling C:\Users\Jason\Desktop\scipy-2017-codegen-tutorial-master\notebooks\ode_cython_9a390683f6.pyx because it changed. [1/1] Cythonizing C:\Users\Jason\Desktop\scipy-2017-codegen-tutorial-master\notebooks\ode_cython_9a390683f6.pyx

---

FileNotFoundError Traceback (most recent call last) ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in load_module(name, pyxfilename, pyxbuild_dir, is_package, build_inplace, language_level, so_path) 215 so_path = build_module(module_name, pyxfilename, pyxbuild_dir, --> 216 inplace=build_inplace, language_level=language_level) 217 mod = imp.load_dynamic(name, so_path)

~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in build_module(name, pyxfilename, pyxbuild_dir, inplace, language_level) 191 inplace=inplace, --> 192 reload_support=pyxargs.reload_support) 193 assert os.path.exists(so_path), "Cannot find: %s" % so_path

~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyxbuild.py in pyx_to_dll(filename, ext, force_rebuild, build_in_temp, pyxbuild_dir, setup_args, reload_support, inplace) 101 obj_build_ext = dist.get_command_obj("build_ext") --> 102 dist.run_commands() 103 so_path = obj_build_ext.get_outputs()[0]

~\Anaconda3\envs\codegen17\lib\distutils\dist.py in run_commands(self) 954 for cmd in self.commands: --> 955 self.run_command(cmd) 956

~\Anaconda3\envs\codegen17\lib\distutils\dist.py in run_command(self, command) 973 cmd_obj.ensure_finalized() --> 974 cmd_obj.run() 975 self.have_run[command] = 1

~\Anaconda3\envs\codegen17\lib\site-packages\Cython\Distutils\old_build_ext.py in run(self) 184 --> 185 _build_ext.build_ext.run(self) 186

~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in run(self) 307 dry_run=self.dry_run, --> 308 force=self.force) 309 customize_compiler(self.compiler)

~\Anaconda3\envs\codegen17\lib\distutils\ccompiler.py in new_compiler(plat, compiler, verbose, dry_run, force) 1030 # argument. -> 1031 return klass(None, dry_run, force) 1032

~\Anaconda3\envs\codegen17\lib\distutils\cygwinccompiler.py in init(self, verbose, dry_run, force) 300 --> 301 if is_cygwingcc(): 302 raise CCompilerError(

~\Anaconda3\envs\codegen17\lib\distutils\cygwinccompiler.py in is_cygwingcc() 410 '''Try to determine if the gcc that would be used is from cygwin.''' --> 411 out_string = check_output(['gcc', '-dumpmachine']) 412 return out_string.strip().endswith(b'cygwin')

~\Anaconda3\envs\codegen17\lib\subprocess.py in check_output(timeout, *popenargs, *kwargs) 335 return run(popenargs, stdout=PIPE, timeout=timeout, check=True, --> 336 **kwargs).stdout 337

~\Anaconda3\envs\codegen17\lib\subprocess.py in run(input, timeout, check, *popenargs, *kwargs) 402 --> 403 with Popen(popenargs, **kwargs) as process: 404 try:

~\Anaconda3\envs\codegen17\lib\subprocess.py in init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors) 706 errread, errwrite, --> 707 restore_signals, start_new_session) 708 except:

~\Anaconda3\envs\codegen17\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session) 989 cwd, --> 990 startupinfo) 991 finally:

FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

ImportError Traceback (most recent call last)

in () ----> 1 cython_sys = mk_rsys(CythonODEsys, **watrad_data) c:\users\jason\desktop\scipy-2017-codegen-tutorial-master\scipy2017codegen\chem.py in mk_rsys(ODEcls, reactions, names, params, **kwargs) 23 def mk_rsys(ODEcls, reactions, names, params=(), **kwargs): 24 f, symbs = mk_exprs_symbs(reactions, names) ---> 25 return ODEcls(f, symbs, params=map(sym.S, params), **kwargs) 26 27 c:\users\jason\desktop\scipy-2017-codegen-tutorial-master\scipy2017codegen\odesys.py in __init__(self, f, y, t, params, tex_names, lambdify) 16 self.j = sym.Matrix(self.ny, 1, f).jacobian(y) 17 self.lambdify = lambdify or sym.lambdify ---> 18 self.setup() 19 20 @property in setup(self) 44 library_dirs=[], libraries=[], extra_compile_args=[], extra_link_args=[] 45 )) ---> 46 mod = __import__(self.mod_name) 47 self.f_eval = mod.f 48 self.j_eval = mod.j ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in load_module(self, fullname) 443 self.pyxbuild_dir, 444 build_inplace=self.inplace, --> 445 language_level=self.language_level) 446 return module 447 ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in load_module(name, pyxfilename, pyxbuild_dir, is_package, build_inplace, language_level, so_path) 230 name, traceback.format_exception_only(*sys.exc_info()[:2]))) 231 if sys.version_info[0] >= 3: --> 232 raise exc.with_traceback(tb) 233 else: 234 exec("raise exc, None, tb", {'exc': exc, 'tb': tb}) ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in load_module(name, pyxfilename, pyxbuild_dir, is_package, build_inplace, language_level, so_path) 214 module_name = name 215 so_path = build_module(module_name, pyxfilename, pyxbuild_dir, --> 216 inplace=build_inplace, language_level=language_level) 217 mod = imp.load_dynamic(name, so_path) 218 if is_package and not hasattr(mod, '__path__'): ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyximport.py in build_module(name, pyxfilename, pyxbuild_dir, inplace, language_level) 190 setup_args=sargs, 191 inplace=inplace, --> 192 reload_support=pyxargs.reload_support) 193 assert os.path.exists(so_path), "Cannot find: %s" % so_path 194 ~\Anaconda3\envs\codegen17\lib\site-packages\pyximport\pyxbuild.py in pyx_to_dll(filename, ext, force_rebuild, build_in_temp, pyxbuild_dir, setup_args, reload_support, inplace) 100 try: 101 obj_build_ext = dist.get_command_obj("build_ext") --> 102 dist.run_commands() 103 so_path = obj_build_ext.get_outputs()[0] 104 if obj_build_ext.inplace: ~\Anaconda3\envs\codegen17\lib\distutils\dist.py in run_commands(self) 953 """ 954 for cmd in self.commands: --> 955 self.run_command(cmd) 956 957 # -- Methods that operate on its Commands -------------------------- ~\Anaconda3\envs\codegen17\lib\distutils\dist.py in run_command(self, command) 972 cmd_obj = self.get_command_obj(command) 973 cmd_obj.ensure_finalized() --> 974 cmd_obj.run() 975 self.have_run[command] = 1 976 ~\Anaconda3\envs\codegen17\lib\site-packages\Cython\Distutils\old_build_ext.py in run(self) 183 optimization.disable_optimization() 184 --> 185 _build_ext.build_ext.run(self) 186 187 def build_extensions(self): ~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in run(self) 306 verbose=self.verbose, 307 dry_run=self.dry_run, --> 308 force=self.force) 309 customize_compiler(self.compiler) 310 # If we are cross-compiling, init the compiler now (if we are not ~\Anaconda3\envs\codegen17\lib\distutils\ccompiler.py in new_compiler(plat, compiler, verbose, dry_run, force) 1029 # with classes that expect verbose to be the first positional 1030 # argument. -> 1031 return klass(None, dry_run, force) 1032 1033 ~\Anaconda3\envs\codegen17\lib\distutils\cygwinccompiler.py in __init__(self, verbose, dry_run, force) 299 entry_point = '' 300 --> 301 if is_cygwingcc(): 302 raise CCompilerError( 303 'Cygwin gcc cannot be used with --compiler=mingw32') ~\Anaconda3\envs\codegen17\lib\distutils\cygwinccompiler.py in is_cygwingcc() 409 def is_cygwingcc(): 410 '''Try to determine if the gcc that would be used is from cygwin.''' --> 411 out_string = check_output(['gcc', '-dumpmachine']) 412 return out_string.strip().endswith(b'cygwin') ~\Anaconda3\envs\codegen17\lib\subprocess.py in check_output(timeout, *popenargs, **kwargs) 334 335 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, --> 336 **kwargs).stdout 337 338 ~\Anaconda3\envs\codegen17\lib\subprocess.py in run(input, timeout, check, *popenargs, **kwargs) 401 kwargs['stdin'] = PIPE 402 --> 403 with Popen(*popenargs, **kwargs) as process: 404 try: 405 stdout, stderr = process.communicate(input, timeout=timeout) ~\Anaconda3\envs\codegen17\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors) 705 c2pread, c2pwrite, 706 errread, errwrite, --> 707 restore_signals, start_new_session) 708 except: 709 # Cleanup if the child failed starting. ~\Anaconda3\envs\codegen17\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session) 988 env, 989 cwd, --> 990 startupinfo) 991 finally: 992 # Child is launched. Close the parent's copy of those pipe ImportError: Building module ode_cython_9a390683f6 failed: ['FileNotFoundError: [WinError 2] The system cannot find the file specified\n'] ``` # 45-chemical-kinetics-cython-vode Fails on cell [6] with same error as above notebook. # javascript_viz Runs great. # the_harder_way - Fails on line 54 because cat is not an available program on windows - gcc compilation works on line 55 (assumingly with mingw) but `!run.exe` needs to be called for windows instead of the linux command. # the_easy_hard_way fails on first cython cell with: ``` --------------------------------------------------------------------------- DistutilsPlatformError Traceback (most recent call last) in () ----> 1 get_ipython().run_cell_magic('cython', '--annotate', 'def cython_fib(int n):\n cdef int i\n cdef double a = 0.0, b = 1.0\n for i in range(n):\n a, b = a + b, a\n return a') ~\Anaconda3\envs\codegen17\lib\site-packages\IPython\core\interactiveshell.py in run_cell_magic(self, magic_name, line, cell) 2101 magic_arg_s = self.var_expand(line, stack_depth) 2102 with self.builtin_trap: -> 2103 result = fn(magic_arg_s, cell) 2104 return result 2105 in cython(self, line, cell) ~\Anaconda3\envs\codegen17\lib\site-packages\IPython\core\magic.py in (f, *a, **k) 185 # but it's overkill for just that one bit of state. 186 def magic_deco(arg): --> 187 call = lambda f, *a, **k: f(*a, **k) 188 189 if callable(arg): ~\Anaconda3\envs\codegen17\lib\site-packages\Cython\Build\IpythonMagic.py in cython(self, line, cell) 289 build_extension.build_temp = os.path.dirname(pyx_file) 290 build_extension.build_lib = lib_dir --> 291 build_extension.run() 292 self._code_cache[key] = module_name 293 ~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in run(self) 337 338 # Now actually compile and link everything. --> 339 self.build_extensions() 340 341 def check_extensions_list(self, extensions): ~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in build_extensions(self) 446 self._build_extensions_parallel() 447 else: --> 448 self._build_extensions_serial() 449 450 def _build_extensions_parallel(self): ~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in _build_extensions_serial(self) 471 for ext in self.extensions: 472 with self._filter_build_errors(ext): --> 473 self.build_extension(ext) 474 475 @contextlib.contextmanager ~\Anaconda3\envs\codegen17\lib\distutils\command\build_ext.py in build_extension(self, ext) 531 debug=self.debug, 532 extra_postargs=extra_args, --> 533 depends=ext.depends) 534 535 # XXX outdated variable, kept here in case third-part code ~\Anaconda3\envs\codegen17\lib\distutils\_msvccompiler.py in compile(self, sources, output_dir, macros, include_dirs, debug, extra_preargs, extra_postargs, depends) 302 303 if not self.initialized: --> 304 self.initialize() 305 compile_info = self._setup_compile(output_dir, macros, include_dirs, 306 sources, depends, extra_postargs) ~\Anaconda3\envs\codegen17\lib\distutils\_msvccompiler.py in initialize(self, plat_name) 195 plat_spec = PLAT_TO_VCVARS[plat_name] 196 --> 197 vc_env = _get_vc_env(plat_spec) 198 if not vc_env: 199 raise DistutilsPlatformError("Unable to find a compatible " ~\Anaconda3\envs\codegen17\lib\distutils\_msvccompiler.py in _get_vc_env(plat_spec) 83 vcvarsall, vcruntime = _find_vcvarsall(plat_spec) 84 if not vcvarsall: ---> 85 raise DistutilsPlatformError("Unable to find vcvarsall.bat") 86 87 try: DistutilsPlatformError: Unable to find vcvarsall.bat ``` - Tried install http://landinghub.visualstudio.com/visual-cpp-build-tools based on https://matthew-brett.github.io/pydagogue/python_msvc.html to see if this will provide the right compiler for Python 3.5+. Note that https://github.com/cython/cython/wiki/CythonExtensionsOnWindows says not to compile with Minggw, as it has to match whatever python was compiled with on windows (not sure what this is for anaconda) - need to add "import setuptools" right above (in same cell) as `%load_ext cython - Ok that worked! I just had to install that cpp-build-tools and add the `import setuptools` line and it found the correct compiler and ran! # 50-chemical-kinetics-C Same file not found error as Bjorn's other notebooks above # 60-chemical-kinetics-reaction-diffusion Got this warning on cell [3] ``` C:\Users\Jason\Anaconda3\envs\codegen17\lib\site-packages\sympy\core\decorators.py:38: SymPyDeprecationWarning: _as_finite_diff has been deprecated since SymPy 1.1. Use Derivative.as_finite_difference instead. See https://github.com/sympy/sympy/issues/11410 for more info. _warn_deprecation(wrapped, 3) ``` Notebook runs otherwise. # Main conclusions - Add the visual cpp build tools to our windows installation instructions - Figure out what is going on with the missing file. It generates the c, pyx, and pyxbld files in the directory. - SKip theano?

[moorepants]

This seems to be the main issue. For some reason when Cython checks for cygwin it calls:

ipdb> check_output(['gcc', '-dumpmachine'])
*** FileNotFoundError: [WinError 2] The system cannot find the file specified

and that gets and error but if you have mingw installed (like we do in our env) I can call this at the command line and it works:

(codegen17) C:\Users\Jason>gcc -dumpmachine
x86_64-w64-mingw32

Looks like distutils needs a patch:

https://stackoverflow.com/questions/24291506/building-minimal-cython-file-with-python-3-3-anaconda-under-windows-7

out_string = check_output(['gcc', '-dumpmachine'], shell=True)

[moorepants]

And this looks like some Aaron has already investigated:

http://bugs.python.org/issue21821

[bjodah]

Regarding the win64 packages, I thought they were covered here: https://github.com/sympy/scipy-2017-codegen-tutorial/blob/master/.drone.yml#L15

23: That is "intended" (Euler forward deviates)

40: Nice to see you came to the same conclusion regarding the cygwincompiler (https://github.com/sympy/scipy-2017-codegen-tutorial/issues/2#issuecomment-307812559). Do you know how to include the patch in our conda package?

+1 for skipping theano

[asmeurer]

Geez is that still an issue? I thought Continuum moved everything from .bat over to .exe.

I don't remember if there are some better workarounds for this. Might be worth some further investigation.

For theano, let's skip it. It isn't even really printed anyway. I couldn't even test it on my own computer because I couldn't install theano on the plane (and plus it's dumb to make theano a dependency of the tutorial just for this).

[asmeurer]

How does the distutils code get called? Can we monkeypatch it?

[moorepants]

Interesting that the %%cython magic command compiles fine but the pyximport doesn't. They must not follow the same code paths.

[asmeurer]

Maybe related to the fact that pyximport uses Cython.Distutils and the cell magic doesn't.

[moorepants]

This shows the win-64 packages: https://anaconda.org/SymPy/scipy2017codegen, not sure why it failed to download. Was it maybe because of the channel name: sympy ==3.5? What does it mean to version the channel?

[asmeurer]

What error is this in reference to?

[moorepants]

With respect to:

WARNING: The remote server could not find the noarch directory for the
requested channel with url: https://conda.anaconda.org/sympy ==3.5

It is possible you have given conda an invalid channel. Please double-check
your conda configuration using `conda config --show`.

If the requested url is in fact a valid conda channel, please request that the
channel administrator create `noarch/repodata.json` and associated
`noarch/repodata.json.bz2` files, even if `noarch/repodata.json` is empty.
$ mkdir noarch
$ echo '{}' > noarch/repodata.json
$ bzip2 -k noarch/repodata.json

[moorepants]

Do you know how to include the patch in our conda package?

I think we'd have to make a custom disutils package on the sympy channel that had priority in downloading. Then we could include a patch via the meta.yml file.

[bjodah]

• Compiler: Using msvc from: http://landinghub.visualstudio.com/visual-cpp-build-tools worked for me. But then we should try to make people download this before the conference (4 GiB download size).

• The channel should not have been versioned. A silly mistake by me. I just changed it.

• Regarding pyximport: what examples are failing? (note that I used a .pyxbld file)

[bjodah]

@moorepants I just changed so that msvc is the default: https://github.com/sympy/scipy-2017-codegen-tutorial/blob/d7d213aea043cbba623b7f116df3b87dbfccd2d0/scipy2017codegen/template.pyxbld#L3

[bjodah]

We could introduce a conda package called e.g. scipy2017codegen_mingw which sets that environment variable to "mingw" (the conda package would contain bat files as per: https://conda.io/docs/using/envs.html#more-notes-on-environment-variable-scripts). But this is all very complicated :/

[asmeurer]

Let's try to avoid patching the standard library with conda if possible. I know there is a way to change the default compiler with some config file (Continuum has a package that does this, but I forgot what it's called).

[moorepants]

I think everything here is fixed, no?

Except I don't know the status of the disutils patch needs? @bjodah Did you work around that?

[bjodah]

The distutils patch was needed for Python 3 when using mingw32, I changed the default to msvc.

• If we require msvc for our tutorial we can close this.
• If we want to support mingw32 we still need to figure out how to workaround distutils calling gcc without shell=True. Monkey-patching the relatively short python function: distutils.cygwinccompiler.is_cygwingcc comes to mind. I actually think I could make that work.

[moorepants]

I think monkey patching is fine for this purpose if we need to do it. The other option would be to ship a patched distutils in the sympy conda channel. I'm not completely sure how to monkey patch it without shipping a custom distutils. We could copy the relevant distutils modules into our scipy2017codegen package and then import from that?

[bjodah]

I'm thinking something dead simple:

>>> import distutils.cygwinccompiler
>>> distutils.cygwinccompiler.is_cygwingcc()
...
TraceBack ...
>>> distutils.cygwinccompiler.is_cygwingcc = lambda: True
>>> distutils.cygwinccompiler.is_cygwingcc()
True

the monkeypatching would be performed in pyxbld when $SCIPY2017CODEGEN_COMPILER == mingw32. (I'm trying this right now)

[moorepants]

That's a nice way to do it.

[moorepants]

Thought you were going to bed :)

[bjodah]

I was... and now I am :)

Tried it, but it didn't work unfortunately (it's a bit of whack-a-mole):

~\Anaconda2\envs\codegen17\lib\distutils\cygwinccompiler.py in __init__(self, verbose, dry_run, force)
    155             # Include the appropriate MSVC runtime library if Python was built
    156             # with MSVC 7.0 or later.
--> 157             self.dll_libraries = get_msvcr()
    158 
    159     def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts):

~\Anaconda2\envs\codegen17\lib\distutils\cygwinccompiler.py in get_msvcr()
     84             return ['msvcr100']
     85         else:
---> 86             raise ValueError("Unknown MS Compiler version %s " % msc_ver)
     87 
     88 

ImportError: Building module ode_cython_311c73e7fc failed: ['ValueError: Unknown MS Compiler version 1900 \n']

As a side note: there is a traceback from _msvccompiler.py in distutils when trying to use %%cython magic in our codegen17 environment, but it works in Anaconda2. :/

Not sure mingw is worth the trouble..

[isuruf]

Is this conda-forge's python? It was fixed here, https://github.com/conda-forge/python-feedstock/pull/143

[bjodah]

Thanks @isuruf, that got me a bit further:

c:/users/user/anaconda2/envs/codegen17/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: c:\users\user\scipy-2017-codegen-tutorial\notebooks\ode_c_c847e93dc6.cp36-win_amd64.def:1: syntax error
...
collect2.exe: error: ld returned 1 exit status

contents of ode_c_c847e93dc6.cp36-win_amd64.def:

LIBRARY ode_c_c847e93dc6.cp36-win_amd64.pyd
EXPORTS
PyInit_ode_c_c847e93dc6

(I still need to monkey-patch distutils.cygwinccompiler.is_cygwingcc to get this far) For some reason I also had to manually add distutils/distutils.cfg:

[build]
compiler = ming32

to the conda environment (it was present in the root environment, this may have been caused by me manually upgrading python).

[bjodah]

Thanks for all help. But I'm closing this for now (I need to focus more on content and less on mingw :) ).