search

ContinuumIO / anaconda-issues

Anaconda issue tracking
646 stars 220 forks source link

SyntaxError on import of mayavi.api (double CR in apptools/persistence/state_pickler.py) #10984

Open grelston opened 5 years ago

grelston commented 5 years ago

Actual Behavior

>>> import mayavi
>>> mayavi.__version__
'4.6.2'
>>> import mayavi.api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Miniconda3\envs\mayavi\lib\site-packages\mayavi\api.py", line 2, in <module>
    from mayavi.core.engine import Engine
  File "C:\Miniconda3\envs\mayavi\lib\site-packages\mayavi\core\engine.py", line 23, in <module>
    from apptools.persistence import state_pickler
  File "C:\Miniconda3\envs\mayavi\lib\site-packages\apptools\persistence\state_pickler.py", line 1210
    ^
SyntaxError: invalid syntax

Expected Behavior

>>> import mayavi
>>> mayavi.__version__
'4.6.2'
>>> import mayavi.api
>>>

Steps to Reproduce

(base) C:\Users\User>conda create --name=mayavi python=3.7 mayavi mock nose

(base) C:\Users\User>conda activate mayavi

(mayavi) C:\Users\User>python
Python 3.7.3 (default, Apr 24 2019, 13:20:13) [MSC v.1915 32 bit (Intel)] :: Anaconda, Inc. on win32

Type "help", "copyright", "credits" or "license" for more information.
>>> import mayavi
>>> mayavi.__version__
'4.6.2'
>>> import mayavi.api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Miniconda3\envs\mayavi\lib\site-packages\mayavi\api.py", line 2, in <module>
    from mayavi.core.engine import Engine
  File "C:\Miniconda3\envs\mayavi\lib\site-packages\mayavi\core\engine.py", line 23, in <module>
    from apptools.persistence import state_pickler
  File "C:\Miniconda3\envs\mayavi\lib\site-packages\apptools\persistence\state_pickler.py", line 1210
    ^
SyntaxError: invalid syntax
Anaconda or Miniconda version:

Miniconda3-4.6.14-Windows-x86

Operating System:

Windows 7 Enterprise, Service Pack 1, 64-bit Operating System

conda info
``` active environment : mayavi active env location : C:\Miniconda3\envs\mayavi shell level : 2 user config file : C:\Users\User\.condarc populated config files : C:\Users\User\.condarc conda version : 4.6.14 conda-build version : not installed python version : 3.7.2.final.0 base environment : C:\Miniconda3 (writable) channel URLs : https://conda.anaconda.org/r/win-32 https://conda.anaconda.org/r/noarch https://repo.anaconda.com/pkgs/main/win-32 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/win-32 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/win-32 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-32 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Miniconda3\pkgs C:\Users\User\.conda\pkgs C:\Users\User\AppData\Local\conda\conda\pkgs envs directories : C:\Miniconda3\envs C:\Users\User\.conda\envs C:\Users\User\AppData\Local\conda\conda\envs platform : win-32 user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.2 Windows/7 Windows/6.1.7601 administrator : True netrc file : None offline mode : False ```
conda list --show-channel-urls
``` # packages in environment at C:\Miniconda3\envs\mayavi: # # Name Version Build Channel apptools 4.4.0 py37_1 defaults blas 1.0 mkl defaults bzip2 1.0.6 hfa6e2cd_5 defaults ca-certificates 2019.5.15 0 defaults certifi 2019.3.9 py37_0 defaults configobj 5.0.6 py37_1 defaults curl 7.64.1 h2a8f88b_0 defaults envisage 4.7.2 py_0 defaults expat 2.2.5 he025d50_0 defaults freetype 2.9.1 ha9979f8_1 defaults future 0.17.1 py37_0 defaults hdf4 4.2.13 hf3d92eb_2 defaults hdf5 1.10.4 h7ebc959_0 defaults icc_rt 2019.0.0 h0cc432a_1 defaults icu 58.2 h3fcc66b_1 defaults intel-openmp 2019.4 245 defaults jpeg 9b he27b436_2 defaults jsoncpp 1.8.4 h74a9793_0 defaults krb5 1.16.1 hc04afaa_7 defaults libcurl 7.64.1 h2a8f88b_0 defaults libiconv 1.15 hc4a7105_7 defaults libnetcdf 4.6.1 h411e497_2 defaults libpng 1.6.37 h2a8f88b_0 defaults libssh2 1.8.2 h7a1dbc1_0 defaults libtiff 4.0.10 hb898794_2 defaults libxml2 2.9.9 h464c3ec_0 defaults lz4-c 1.8.1.2 h2fa13f4_0 defaults mayavi 4.6.2 py37h3ccff09_4 defaults mkl 2019.4 245 defaults mkl_fft 1.0.12 py37h14836fe_0 defaults mkl_random 1.0.2 py37h343c172_0 defaults mock 3.0.5 py37_0 defaults nose 1.3.7 py37_2 defaults numpy 1.16.4 py37h19fb1c0_0 defaults numpy-base 1.16.4 py37hc3f5095_0 defaults openssl 1.1.1c he774522_1 defaults pip 19.1.1 py37_0 defaults pyface 6.0.0 py37_0 defaults pygments 2.4.0 py_0 defaults pyqt 5.9.2 py37h6538335_0 defaults python 3.7.3 h8c8aaf0_1 defaults qt 5.9.7 vc14h73c81de_0 defaults setuptools 41.0.1 py37_0 defaults sip 4.19.8 py37h6538335_0 defaults six 1.12.0 py37_0 defaults sqlite 3.28.0 he774522_0 defaults tbb 2019.4 h74a9793_0 defaults tk 8.6.8 hfa6e2cd_0 defaults traits 5.1.1 py37he774522_0 defaults traitsui 6.0.0 py37_1 defaults vc 14.1 h0510ff6_4 defaults vs2015_runtime 14.15.26706 h3a45250_4 defaults vtk 8.2.0 py37h1e53df8_200 defaults wheel 0.33.4 py37_0 defaults wincertstore 0.2 py37_0 defaults xz 5.2.4 h2fa13f4_4 defaults zlib 1.2.11 h62dcd97_3 defaults zstd 1.3.7 h508b16e_0 defaults ```
grelston commented 5 years ago

Related issues

grelston commented 5 years ago

Steps to fix the SyntaxError


>>> pyfile = r"C:\Miniconda3\envs\mayavi\lib\site-packages\apptools\persistence\state_pickler.py"
>>> txt_str = open(pyfile, 'rt').read()
>>> txt_lines = open(pyfile, 'rt').readlines()
>>> bin_str = open(pyfile, 'rb').read()
>>> bin_lines = open(pyfile, 'rb').readlines()
>>> import hashlib
>>> hashlib.md5(bin_str).hexdigest()
'474262401f975920a18c82e763783c37'
>>> len(txt_lines)
2045
>>> len(bin_lines)
1023
>>> bin_lines[0:5]
[b'"""This module provides code that allows one to pickle the state of a\r\r\n', b'Python object to a dictionary.\r\r\n', b'\r\r\n', b'The motivation for this is simple.  The standard Python\r\r\n', b'pickler/unpickler is best used to pickle simple objects and does not\r\r\n']
>>> txt_lines[0:10]
['"""This module provides code that allows one to pickle the state of a\n', '\n', 'Python object to a dictionary.\n', '\n', '\n', '\n', 'The motivation for this is simple.  The standard Python\n', '\n', 'pickler/unpickler is best used to pickle simple objects and does not\n', '\n']
>>> open(pyfile+'.orig', 'wb').write(bin_str)
37425
>>> bin_str.replace(b'\r\r', b'\r')[0:100]
b'"""This module provides code that allows one to pickle the state of a\r\nPython object to a dictionary'
>>> open(pyfile, 'wb').write(bin_str.replace(b'\r\r', b'\r'))
36403
>>> import mayavi.api
>>> # an unrelated NameError. See Issue #10985.
grelston commented 5 years ago

This issue also affects the win-64 versions of mayavi and apptools. Once the above fix is applied, the 64-bit mayavi.api imports with no further error.

grelston commented 5 years ago

Could the apptools packages for the main/win-32 and main/win-64 channels be rebuilt with (single) CR-LF line-endings instead of the existing (double) CR-CR-LF line-endings? Thanks.

grelston commented 5 years ago

I suspect this is an issue with the conda build of https://github.com/enthought/apptools because a pip install of mayavi (and its apptools and envisage dependencies) works as expected (see https://github.com/ContinuumIO/anaconda-issues/issues/10985#issuecomment-499627263 for more details).

msarahan commented 5 years ago

I vaguely remember something about this a long time ago. My memory is that a git clone or something like that introduced the double line endings. I'm confused, though, because the apptools recipe is using source from a pypi package: https://github.com/AnacondaRecipes/apptools-feedstock/blob/master/recipe/meta.yaml

If you can get a good build working on the conda-forge feedstock, we'll be happy to bring it in. Otherwise, I'm afraid it will be a while before we can debug this and fix it ourselves.

grelston commented 5 years ago

Thanks @msarahan.

The conda-forge feedstock already builds a working apptools-4.4.0 for python=3.7 on both win-32 and win-64 (conda-forge also provides a working mayavi-4.6.2 on win-64; however, there is no mayavi for python=3.7 on win-32 in conda-forge).

### Install and test apptools-4.4.0 from conda-forge on win-32 ```bash (base) C:\Users\User\win32>conda create --name=apptools-cf python=3.7 vtk numpy traitsui configobj six (base) C:\Users\User\win32>conda activate apptools-cf (apptools-cf) C:\Users\User\win32>conda install --override-channels -c conda-forge apptools (apptools-cf) C:\Users\User\win32>python ``` ```python Python 3.7.3 (default, Apr 24 2019, 13:20:13) [MSC v.1915 32 bit (Intel)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import apptools.persistence.state_pickler >>> apptools.persistence.state_pickler >>> ```
### Install and test apptools-4.4.0 (as part of mayavi-4.6.2) from conda-forge on win-64 ```bash (base) C:\Users\User\win-64>conda create --name mayavi-cf python=3.7 vtk numpy traitsui configobj six (base) C:\Users\User\win-64>conda activate mayavi-cf (mayavi-cf) C:\Users\User\win-64>conda install --override-channels -c conda-forge mayavi (mayavi-cf) C:\Users\User\win-64>python ``` ```python Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mayavi.api >>> import apptools.persistence >>> apptools.persistence >>> apptools.persistence.state_pickler >>> ```
### Check which apptools and mayavi versions are available from conda-forge on win-32 ```bash (apptools-cf) C:\Users\User\win-32>conda search --override-channels -c conda-forge apptools Loading channels: done # Name Version Build Channel apptools 4.4.0 py27_0 conda-forge apptools 4.4.0 py34_0 conda-forge apptools 4.4.0 py35_0 conda-forge apptools 4.4.0 py36_0 conda-forge apptools 4.4.0 py_1 conda-forge (apptools-cf) C:\Users\User\win-32>conda search --override-channels -c conda-forge mayavi Loading channels: done # Name Version Build Channel mayavi 4.5.0 np111py27_vc9_1 conda-forge mayavi 4.5.0 np111py27_vc9_2 conda-forge mayavi 4.5.0 np111py35_vc14_1 conda-forge mayavi 4.5.0 np111py35_vc14_2 conda-forge mayavi 4.5.0 np111py36_vc14_2 conda-forge mayavi 4.5.0 np112py27_vc9_1 conda-forge mayavi 4.5.0 np112py27_vc9_2 conda-forge mayavi 4.5.0 np112py35_vc14_1 conda-forge mayavi 4.5.0 np112py35_vc14_2 conda-forge mayavi 4.5.0 np112py36_vc14_2 conda-forge mayavi 4.5.0 np113py27_vc9_2 conda-forge mayavi 4.5.0 np113py35_vc14_2 conda-forge mayavi 4.5.0 np113py36_vc14_2 conda-forge mayavi 4.5.0 py27_vc9_3 conda-forge mayavi 4.5.0 py35_vc14_3 conda-forge mayavi 4.5.0 py36_vc14_3 conda-forge ```

I have compared the recipe/meta.yaml files from AnacondaRecipes/apptools-feedstock and conda-forge/apptools-feedstock. They both refer to the same pypi file for apptools-4.4.0.tar.bz2 and report the same md5 hash. The only major difference seems to be that conda-forge installs apptools using pip, whereas AnacondaRecipes uses "python setup.py install".

```bash diff --git a/AnacondaRecipes/apptools-feedstock/recipe/meta.yaml b/conda-forge/apptools-feedstock/recipe/meta.yaml index b50ced7..df46af6 100644 --- a/AnacondaRecipes/apptools-feedstock/recipe/meta.yaml +++ b/conda-forge/apptools-feedstock/recipe/meta.yaml @@ -10,11 +10,12 @@ source: md5: cf122251faec4d3dff0a9480d6f6f9f7 build: - number: 1 - script: python setup.py install --single-version-externally-managed --record record.txt + number: 1002 + script: python -m pip install --no-deps --ignore-installed . requirements: host: + - pip - python - setuptools - traits @@ -31,11 +32,14 @@ requirements: test: imports: - apptools + # the following import will fail on Python 3 if 2to3 wasn't run + - apptools.logger.api about: home: http://docs.enthought.com/apptools/ license: BSD 3-clause summary: application tools + license_file: LICENSE.txt extra: recipe-maintainers: ```
grelston commented 5 years ago

The double-CR line-ending was a new issue introduced by python/cpython#6483. The problem in lib2to3 was fixed, committed, merged and backported to python 3.7 in July 2018.

Both py37 builds of apptools in the main/win-32 Anaconda repo (py37_0 and py37_1) have CRCRLF line-endings in Lib/site-packages/apptools/persistence/state_pickler.py even though their timestamps are August and September 2018. Could a pre-July-2018 version of (python-3.7) 2to3 have been used to generate them?

For completeness, I also checked the line-endings of state_pickler.py in the apptools-4.4.0-*.tar.bz2 files for py36_1 (CRLF), py35_1 (CRLF) and py27_1 (LF).