search

AnacondaRecipes / python-feedstock

A conda-smithy repository for python.
BSD 3-Clause "New" or "Revised" License
4 stars 7 forks source link

implement sysconfig default, to not require our new compiler #4

Closed msarahan closed 7 years ago

msarahan commented 7 years ago

CC @jjhelmus @nehaljwani @mingwandroid

This doesn't quite work yet. The default one does work, and does allow things to be built with gcc. However, I saw this link error:

statsmodels/tsa/statespace/_statespace.c:4494: note: initialized from here
gcc -pthread -shared -L$PREFIX/lib -Wl,-rpath=$PREFIX/lib,--no-as-needed build/temp.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.o -L/opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib -L$PREFIX/lib -lnpymath -lm -lpython3.6m -o build/lib.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.cpython-36m-x86_64-linux-gnu.so
/usr/bin/ld: /opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib/libnpymath.a(npy_math_complex.o)(.text+0x2): unresolvable H�\$�H�l$�H��L�d$�H��H�GH��I��H��H�;�PxH�EI�tH��PxH�EH�{I�t$H�$H�lL�d$H�@xH����f�H�\$�H�l$�H��L�d$�H��H�GI��H��H���PhI�$H�CH��PhH�$I�DH�lL�d$H��� relocation against symbol `cabsf@@GLIBC_2.2.5'
/usr/bin/ld: BFD version 2.20.51.0.2-5.46.el6 20100205 internal error, aborting at reloc.c line 443 in bfd_get_reloc_size

/usr/bin/ld: Please report this bug.

collect2: ld returned 1 exit status
/usr/bin/ld: /opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib/libnpymath.a(npy_math_complex.o)(.text+0x2): unresolvable H�\$èH�l$ðH�óL�d$øH�ìH�GH�ýI�ÔH�ÖH�;ÿPxH�EI�tH�ÿPxH�EH�{I�t$H�$H�lL�d$H�@xH�Äÿàf�H�\$èH�l$ðH�ûL�d$øH�ìH�GI�ÔH�õH�÷ÿPhI�$H�CH�ÿPhH�$I�DH�lL�d$H�Äà relocation against symbol `cabsf@@GLIBC_2.2.5'
/usr/bin/ld: BFD version 2.20.51.0.2-5.46.el6 20100205 internal error, aborting at reloc.c line 443 in bfd_get_reloc_size

/usr/bin/ld: Please report this bug.

collect2: ld returned 1 exit status
error: Command "gcc -pthread -shared -L$PREFIX/lib -Wl,-rpath=$PREFIX/lib,--no-as-needed build/temp.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.o -L/opt/conda/conda-bld/statsmodels_1506636065943/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib -L$PREFIX/lib -lnpymath -lm -lpython3.6m -o build/lib.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.cpython-36m-x86_64-linux-gnu.so" failed with exit status 1

I have tried with gcc 4.4 in centos 6, and also with conda-forge's devtoolset2 docker image. in the docker image, I got a slightly different, but really similar error:

gcc -pthread -shared -L$PREFIX/lib -Wl,-rpath=$PREFIX/lib,--no-as-needed build/temp.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.o -L/opt/conda/conda-bld/statsmodels_1506653245397/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib -L$PREFIX/lib -lnpymath -lm -lpython3.6m -o build/lib.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.cpython-36m-x86_64-linux-gnu.so
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/ld: /opt/conda/conda-bld/statsmodels_1506653245397/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib/libnpymath.a(npy_math_complex.o): unrecognized relocation (0x29) in section `.text'
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/ld: /opt/conda/conda-bld/statsmodels_1506653245397/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib/libnpymath.a(npy_math_complex.o): unrecognized relocation (0x29) in section `.text'
/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
error: Command "gcc -pthread -shared -L$PREFIX/lib -Wl,-rpath=$PREFIX/lib,--no-as-needed build/temp.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.o -L/opt/conda/conda-bld/statsmodels_1506653245397/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.6/site-packages/numpy/core/lib -L$PREFIX/lib -lnpymath -lm -lpython3.6m -o build/lib.linux-x86_64-3.6/statsmodels/tsa/statespace/_statespace.cpython-36m-x86_64-linux-gnu.so" failed with exit status 1

I have been setting _PYTHON_SYSCONFIGDATA_NAME using a variable in conda_build_config.yaml. Note that it must not be set when building python. I have:

_PYTHON_SYSCONFIGDATA_NAME:
  - _sysconfig_x86_64_conda_cos6_linux_gnu  # [linux]

Unfortunately, this doesn't seem to be importable. The error I see is:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site.py", line 544, in <module>
    main()
  File "/opt/conda/lib/python3.6/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/opt/conda/lib/python3.6/site.py", line 282, in addusersitepackages
    user_site = getusersitepackages()
  File "/opt/conda/lib/python3.6/site.py", line 258, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/opt/conda/lib/python3.6/site.py", line 248, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/opt/conda/lib/python3.6/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/opt/conda/lib/python3.6/sysconfig.py", line 550, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/opt/conda/lib/python3.6/sysconfig.py", line 421, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfig_x86_64_conda_cos6_linux_gnu'

I have verified that the _sysconfig_x86_64_conda_cos6_linux_gnu.py file is in the (root)/lib/python3.6 folder, and I have also tried putting it into (root)/lib/python3.6/site-packages, both of which are on sys.path when I activate _h_env...

@jjhelmus told me that he was successful with py2.7 (his work is at https://github.com/jjhelmus/python-feedstock/tree/jjh_27_wip ) - I think I need sleep.

msarahan commented 7 years ago

PS: for anyone playing with this, I advise you to set _OPTIMIZE=0 in build.sh, or else you'll be waiting a while on PGO.

msarahan commented 7 years ago

Latest commit copies prefixed ld into $PREFIX/bin/ld and sets gcc flags to ensure that it gets used preferentially over system ld. This is necessary likely due to LTO.

This now works for me, and needs a bit of testing before uploading to defaults.

msarahan commented 7 years ago

I guess I still haven't resolved the import error - so not quite ready yet.

msarahan commented 7 years ago

The import error comes when that env var is set, but the root python is an older python that doesn't yet have that file. Once the root python is updated to a build that has the appropriate file, all is fine.