Installation Question

[sandeepjella]

I'm using Cantera + Python in Anaconda and have installed pyjac. Works well and generates the C code but I am running into an issue trying to generate the C wrapper that works on that output for later use in Python (i.e. a cantera program).

It is fairly silly: Pyjac doesn't find the gcc compiler, which is installed through MingW in the Anaconda environment. Outside Pyjac's run, the conda prompt is able to find gcc.

I don't think this is a PyJac issue but perhaps someone could spot the obvious and put me out of my misery?

Best wishes,

Sandeep

[skyreflectedinmirrors]

Hi Sandeep,

Just so I'm clear here, the steps involved in this issue are:

1) Install pyJac on windows
2) Generate the c-code
3) use pyjac.pywrap to generate a python wrapper for the generated C-code

Is that right?

If so, I think the issue is with paths / Conda. I don't know that anyone's ever tested pyJac with MinGW, but this is something we should support.

First things to check:

1) Can you call gcc via the Anaconda prompt (that you run python from)?

For example, on my MinGW installed Conda env, the command gcc -v yields:

gcc -v
Using built-in specs.
COLLECT_GCC=C:\ProgramData\Miniconda3\Scripts\gcc.bat\..\..\MinGW\bin\gcc.exe
COLLECT_LTO_WRAPPER=c:/programdata/miniconda3/mingw/bin/../libexec/gcc/x86_64-w64-mingw32/4.7.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../build/gcc/src/configure --target=x86_64-w64-mingw32 --prefix=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --with-sysroot=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --enable-languages=all,obj-c++ --enable-fully-dynamic-string --disable-multilib
Thread model: win32
gcc version 4.7.0 20111220 (experimental) (GCC)

2) However, when I call gcc within python:

python
>>> import subprocess
>>> subprocess.call(['gcc', '-v'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Miniconda3\lib\subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\ProgramData\Miniconda3\lib\subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "C:\ProgramData\Miniconda3\lib\subprocess.py", line 992, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

I think the issue here is that subprocess doesn't inherit the conda path when looking for gcc (but it would, once it actually launches gcc).

For instance,

subprocess.call(['gcc', '-v'], env=os.environ, shell=True)

yields:

COLLECT_GCC=C:\ProgramData\Miniconda3\Scripts\gcc.bat\..\..\MinGW\bin\gcc.exe
COLLECT_LTO_WRAPPER=c:/programdata/miniconda3/mingw/bin/../libexec/gcc/x86_64-w64-mingw32/4.7.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../build/gcc/src/configure --target=x86_64-w64-mingw32 --prefix=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --with-sysroot=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --enable-languages=all,obj-c++ --enable-fully-dynamic-string --disable-multilib
Thread model: win32
gcc version 4.7.0 20111220 (experimental) (GCC)

So the quick fix is to put shell=True in the subprocess call on lines 111 & 207 in libgen.py. In order to do so, you should uninstall pyJac via conda, clone this repo, make these changes and then install your modified version of pyJac via pip

I'll investigate a longer-term fix

[sandeepjella]

Nick,

Spot on with everything. This is exactly my scenario.

Sounds like the quick fix will work since you show that python can find the path. I'll update again when this works.

Thanks again for your help,

Sandeep.

On Fri, May 25, 2018 at 10:35 AM, arghdos notifications@github.com wrote:

Hi Sandeep,

Just so I'm clear here, the steps involved in this issue are:

1. Install pyJac on windows
2. Generate the c-code
3. use pyjac.pywrap to generate a python wrapper for the generated C-code

Is that right?

If so, I think the issue is with paths / Conda. I don't know that anyone's ever tested pyJac with MinGW, but this is something we should support.

First things to check:

1. Can you call gcc via the Anaconda prompt (that you run python from)?

For example, on my MinGW installed Conda env, the command gcc -v yields:

gcc -v Using built-in specs. COLLECT_GCC=C:\ProgramData\Miniconda3\Scripts\gcc.bat....\MinGW\bin\gcc.exe COLLECT_LTO_WRAPPER=c:/programdata/miniconda3/mingw/bin/../libexec/gcc/x86_64-w64-mingw32/4.7.0/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../../../build/gcc/src/configure --target=x86_64-w64-mingw32 --prefix=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --with-sysroot=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --enable-languages=all,obj-c++ --enable-fully-dynamic-string --disable-multilib Thread model: win32 gcc version 4.7.0 20111220 (experimental) (GCC)

1. However, when I call gcc within python:

python

import subprocess subprocess.call(['gcc', '-v']) import sys import subprocess subprocess.call(['gcc', '-v']) Traceback (most recent call last): File "", line 1, in File "C:\ProgramData\Miniconda3\lib\subprocess.py", line 267, in call with Popen(*popenargs, **kwargs) as p: File "C:\ProgramData\Miniconda3\lib\subprocess.py", line 707, in init restore_signals, start_new_session) File "C:\ProgramData\Miniconda3\lib\subprocess.py", line 992, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified

I think the issue here is that subprocess doesn't inherit the conda path https://bugs.python.org/msg166473 when looking for gcc (but it would, once it actually launches gcc).

For instance,

subprocess.call(['gcc', '-v'], env=os.environ, shell=True)

yields:

COLLECT_GCC=C:\ProgramData\Miniconda3\Scripts\gcc.bat....\MinGW\bin\gcc.exe COLLECT_LTO_WRAPPER=c:/programdata/miniconda3/mingw/bin/../libexec/gcc/x86_64-w64-mingw32/4.7.0/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../../../build/gcc/src/configure --target=x86_64-w64-mingw32 --prefix=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --with-sysroot=/c/bb/vista64-mingw32/mingw-x86-x86_64/build/build/root --enable-languages=all,obj-c++ --enable-fully-dynamic-string --disable-multilib Thread model: win32 gcc version 4.7.0 20111220 (experimental) (GCC)

So the quick fix is to put shell=True in the subprocess call on lines 111 & 207 in libgen.py https://github.com/SLACKHA/pyJac/tree/master/pyjac/libgen. In order to do so, you should uninstall pyJac via conda, clone this repo, make these changes and then install your modified version of pyJac via pip

I'll investigate a longer-term fix

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SLACKHA/pyJac/issues/32#issuecomment-392077555, or mute the thread https://github.com/notifications/unsubscribe-auth/Alvbc3ra-Ygeh15x2uRfzh1lbLefwB-Sks5t2Ba0gaJpZM4UNOVq .

[skyreflectedinmirrors]

Sandeep, I'm still running into some issues getting the python wrapper to fully compile here on windows (you have to do similar things to the linker and there's some Cython weirdness), but I should have a pull request for you to test later this afternoon

[sandeepjella]

OK, thanks for that. I'll wait to hear from you. GCC is now found by pyjac and some object files are created but like you mention, there is some weirdness happening.

Not sure if its of use but here's a screenshot of my output. Some crazy path mangling is evident (Line 11)

Sandeep.

On Fri, May 25, 2018 at 11:54 AM, arghdos notifications@github.com wrote:

Sandeep, I'm still running into some issues getting the python wrapper to fully compile here on windows (you have to do similar things to the linker and there's some Cython weirdness), but I should have a pull request for you to test later this afternoon

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SLACKHA/pyJac/issues/32#issuecomment-392102312, or mute the thread https://github.com/notifications/unsubscribe-auth/Alvbc_6Spxiw7fT0TVqhXd-Gv8irv5sCks5t2ClIgaJpZM4UNOVq .

[skyreflectedinmirrors]

Ok, so this is actually a bit more complicated than I hoped. Unfortunately Conda compiles everything with MSVC compilers on windows, and there are some serious incompatibilities between the MinGW and MSVC python run-time libraries.

The mingwpy project was started to try and get around this type of thing, but has since been abandoned.

I could get the C-library to compile, but linking it into a python module for import is proving difficult. So we have some options here:

1) Implement a MSVC C target -- a bit annoying since it'll require you either have MSVC installed (or the Microsoft SDK's)

2) Go to linux where life becomes much less of an issue!

I've also added a mingw branch you could try out.

For me, this fails with:

python -m pyjac.pywrap -l c -so out
...
subprocess.CalledProcessError: Command '['C:\\ProgramData\\Miniconda3\\python.exe', 'C:\\ProgramData\\Miniconda3\\lib\\site-packages\\pyjac\\pywrap\\pyjacob_setup.py', 'build_ext', '--build-lib', 'C:\\Users\\njc07003\\Desktop\\test', '--compiler=mingw32']' returned non-zero exit status 1.

which if you run directly, yields:

C:\ProgramData\Miniconda3\python.exe C:\ProgramData\Miniconda3\lib\site-packages\pyjac\pywrap\pyjacob_setup.py build_ext --build-lib C:\Users\njc07003\Desktop\test --compiler=mingw32
...
writing c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def
C:\ProgramData\Miniconda3\pkgs\mingw-4.7-1\MinGW\bin\gcc.exe -shared -s c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob_wrapper.o C:\Users\njc07003\Desktop\test\build\temp.win-amd64-3.6\libc_pyjac.a c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def -LC:\ProgramData\Miniconda3\libs -LC:\ProgramData\Miniconda3\PCbuild\amd64 -lpython36 -lmsvcr140 -o C:\Users\njc07003\Desktop\test\pyjacob.cp36-win_amd64.pyd
c:/programdata/miniconda3/pkgs/mingw-4.7-1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def:1: syntax error
c:/programdata/miniconda3/pkgs/mingw-4.7-1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe:c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def: file format not recognized; treating as linker script
c:/programdata/miniconda3/pkgs/mingw-4.7-1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe:c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def:1: syntax error
collect2.exe: error: ld returned 1 exit status
error: command 'C:\\ProgramData\\Miniconda3\\pkgs\\mingw-4.7-1\\MinGW\\bin\\gcc.exe' failed with exit status 1

Pretty sure those "file format not recognized; treating as a linker script" is the root of the error here, but I can't find much info about it

[sandeepjella]

No worries, the Linux route is probably best. Thanks for looking at this in detail,

Sandeep.

Sent from my phone.

On May 25, 2018 1:02 PM, "arghdos" notifications@github.com wrote:

Ok, so this is actually a bit more complicated than I hoped. Unfortunately Conda compiles everything with MSVC compilers https://conda.io/docs/user-guide/tutorials/build-windows.html on windows, and there are some serious https://stackoverflow.com/questions/49088038/pyd-modules-build-failed-with-ld-exe-complaining-def-file-def1-syntax-error incompatibilities between the MinGW https://stackoverflow.com/questions/24291506/building-minimal-cython-file-with-python-3-3-anaconda-under-windows-7 and MSVC python run-time libraries.

The mingwpy https://github.com/mingwpy/mingwpy project was started to try and get around this type of thing, but has since been abandoned.

I could get the C-library to compile, but linking it into a python module for import is proving difficult. So we have some options here:

1.

Implement a MSVC C target -- a bit annoying since it'll require you either have MSVC installed (or the Microsoft SDK's) 2.

Go to linux where life becomes much less of an issue!

I've also added a mingw branch you could try out:

For me, this fails with:

python -m pyjac.pywrap -l c -so out ... subprocess.CalledProcessError: Command '['C:\ProgramData\Miniconda3\python.exe', 'C:\ProgramData\Miniconda3\lib\site-packages\pyjac\pywrap\pyjacob_setup.py', 'build_ext', '--build-lib', 'C:\Users\njc07003\Desktop\test', '--compiler=mingw32']' returned non-zero exit status 1.

which if you run directly, yields:

C:\ProgramData\Miniconda3\python.exe C:\ProgramData\Miniconda3\lib\site-packages\pyjac\pywrap\pyjacob_setup.py build_ext --build-lib C:\Users\njc07003\Desktop\test --compiler=mingw32 ... writing c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def C:\ProgramData\Miniconda3\pkgs\mingw-4.7-1\MinGW\bin\gcc.exe -shared -s c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob_wrapper.o C:\Users\njc07003\Desktop\test\build\temp.win-amd64-3.6\libc_pyjac.a c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def -LC:\ProgramData\Miniconda3\libs -LC:\ProgramData\Miniconda3\PCbuild\amd64 -lpython36 -lmsvcr140 -o C:\Users\njc07003\Desktop\test\pyjacob.cp36-win_amd64.pyd c:/programdata/miniconda3/pkgs/mingw-4.7-1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe: c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def:1: syntax error c:/programdata/miniconda3/pkgs/mingw-4.7-1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe:c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def: file format not recognized; treating as linker script c:/programdata/miniconda3/pkgs/mingw-4.7-1/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-w64-mingw32/bin/ld.exe:c:\programdata\miniconda3\lib\site-packages\pyjac\pywrap\pyjacob.cp36-win_amd64.def:1: syntax error collect2.exe: error: ld returned 1 exit status error: command 'C:\ProgramData\Miniconda3\pkgs\mingw-4.7-1\MinGW\bin\gcc.exe' failed with exit status 1

Pretty sure those "file format not recognized; treating as a linker script" is the root of the error here, but I can't find much info about it

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SLACKHA/pyJac/issues/32#issuecomment-392120518, or mute the thread https://github.com/notifications/unsubscribe-auth/Alvbc1xeY7DoESCfDpw7_4e3k6S2Ulswks5t2Dk_gaJpZM4UNOVq .