Sharing my experience of not being able to instantiate 'wrapt' in python 3.9, but I resolved this issue - with reservation

AnthonyTheKoala commented 3 years ago

Dear Mr Dumpleton or other authorised assignee,

Summary

I wish to share my experience of installing the wrapt package. It has been resolved but say it with caution.

The story

I have recently changed from Python 3.8.10 to Python 3.9.5

In the course of reinstalling "Spyder" which uses the wrapt package, wrapt could not be successfully installed.

So I tried installing wrapt by pip install wrapt

The result is a failure to install wrapt.

Typical output from the pipping is: That was a glimpse. The error messages went for a about "two" dos command windows.

What was done to resolve the problem - indeed I use caution in this matter.

When instantiating 'wrapt' in Python 3.9.5 by import wrapt
the result was a "no module found error"
Fortunately I had my Python3.8.10 installation and performed the following tasks in file explorer
Go to the c::\python38\lib\site-packages
highlight the wrapt folder
copy by right-clicking the wrapt folder and select 'copy'
Go to the c:\python39\lib\site-packages
right-click in free space of the folder and select "paste"

Remarks - I caution against the use of copying from previous versions of python to new versions of python

I did not put the the "dist-info" folder from the previous version of python
instantiated 'wrapt' in python
listed the functions in wrapt
Again this was a transfer of the 'wrapt' folder from Python 3.8.10 to the Python 3.9.5 folder.
Caution this should not imply that a mere transfer of a package from an older version to a newer version of python will work all the time.
Caution it was successful in this instance of 'wrapt'. Such a technique may or may not work with other packages.

Summary:

wrapt was successfully installed using the pipping of wrapt in Python 3.8.10, but unsuccessful in the pipping of wrapt in Python 3.9.5,
A mere copying of the wrapt folder from the lib\site-packages of Python38 to Python39's lib\site-packages worked in this instance. BUT a mere transfer of a package from an older version of python to the newest version of python may or may not work.
The method worked without copying the associated "dist-info" folder

Other information:

Python: 3.8.10 to Python 3.9.5 OS: Windows 10 MSVC: from the Idle program: Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32 Note: when originally pipping 'wrapt' the installation failed to recognise MSC 1928. Even did not work when using MSVC for Visual Studio 2017, version 19.16.27045

If Mr Dumpleton or other authorised assignee likes to comment on this and/or close it.

Thank you, Anthony of Sydney

GrahamDumpleton commented 3 years ago

Which suggests there is something wrong with your compiler/Python installation that no C extensions for Python will be able to be compiled, with it affecting not just wrapt.

In the case of wrapt, if the compiler doesn't exist or fails it should fall back to install a pure Python version, unless the compiler setup is really wonky.

You can force it to install the pure Python version by setting the:

WRAPT_INSTALL_EXTENSIONS=false

environment variable, and then install wrapt using:

pip install --no-cache-dir -U wrapt

The --no-cache-dir is to ensure it ignores any prior cached build, and -U forces an install if a version is already installed.

AnthonyTheKoala commented 3 years ago

Dear Mr Dumpleton, Thank you for your response, it is appreciated. One point and two questions please:

First I followed your instructions on the inclusion of the environment variable WRAPT_INSTALL_EXTENSIONS=falseand pipped wrapt by pip install --no-cache-dir -U wrapt.

Note that I included the environment variable WRAPT_INSTALL_EXTENSIONS=false in both the system and user environments.
Question 1 please - it may help in addition to installing wrapt, affecting not just wrapt: Where you said that:

....Which suggests there is something wrong with your compiler/Python installation that no C extensions for Python will be able to be compiled, with it affecting not just wrapt....

When I installed the compiler, I installed the MSVC compilers including Visual Studio 2019 and let the installation program decide whether to add any necessary environment variables and/or addition.

In addition I have also installed MSYS 32-bit and 64-bit and let the installation program take care of environment variables and path.

Question 1 Could you elaborate please about "something wrong " with the compiler installation such that no C extensions for python will be able to be compiled.

Is it something to do with the environmet variables and/or paths in system and user departments?
I have Cython 0.29.23
Question 2 please - when you mentioned pure python and python using C extensions for python, is the pure python slower than the one with the C extensions?

Very much appreciated, Anthony of Sydney

GrahamDumpleton commented 3 years ago

I can't help you determine what may be wrong with the compiler installation. One thing you need to be aware of though is that Python always requires a specific version of the compiler. Details in:

https://wiki.python.org/moin/WindowsCompilers

Unfortunately that site hasn't been updated for Python 3.9, so I don't know what compiler version should be used.

As to whether it is slower, yes it is. See:

https://wrapt.readthedocs.io/en/latest/benchmarks.html

Actual differences in speed will depend on the Python version as pure Python performance has improved over time.

BTW, you only need to set that environment variable when you installed wrapt. So could have been just for the terminal session. It did not need to be set for user or globally in the system.

AnthonyTheKoala commented 3 years ago

Dear Mr Dumpleton, Thank you again for your reply.

Summary incompatible compile

For your information, I wanted to see when I compiled from source instead of pipping. Result: The MS's cl version of 1929 was not recognized.

Method to investigate non-recognition of compiler

Compiling from source, python setup.py install Indicates that the msvc visual studio 2019 is version 1928

 File "c:\python39\lib\distutils\cygwinccompiler.py", line 84, in get_msvcr
    raise ValueError("Unknown MS Compiler version %s " % msc_ver)/
ValueError: Unknown MS Compiler version 1928

I have both Visual Studio 2019 and 2017 In the 2019 version

cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30037

In the 2017 version

cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27045 f

Even when I go to the dos window for version 2017 version and do python setup.py install I still get

File "c:\python39\lib\distutils\cygwinccompiler.py", line 84, in get_msvcr
    raise ValueError("Unknown MS Compiler version %s " % msc_ver)/
ValueError: Unknown MS Compiler version 1928

That occurs even when setup.py is included to include Python 3.9

classifiers = [
    'Development Status :: 5 - Production/Stable',
    'License :: OSI Approved :: BSD License',
    'Programming Language :: Python :: 2',
    'Programming Language :: Python :: 2.7',
    'Programming Language :: Python :: 3',
    ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''#other versions not displayed here
     'Programming Language :: Python :: 3.8',
     'Programming Language  :: Python :: 3.9',
    'Programming Language :: Python :: Implementation :: CPython',
    'Programming Language :: Python :: Implementation :: PyPy',
]

AND by modifying cygwincompiler.py

    msc_pos = sys.version.find('MSC v.')
    if msc_pos != -1:
        msc_ver = sys.version[msc_pos+6:msc_pos+10]
        if msc_ver == '1300':
            # MSVC 7.0
            return ['msvcr70']
        elif msc_ver == '1310':
            # MSVC 7.1
            return ['msvcr71']
        elif msc_ver == '1400':
            # VS2005 / MSVC 8.0
            return ['msvcr80']
        elif msc_ver == '1500':
            # VS2008 / MSVC 9.0
            return ['msvcr90']
        elif msc_ver > '1600':
            # vs 2017, 2019  
            return ['msvcp140']

Produced the error following python setup.py install

running install
running build
running build_py
running build_ext
dllwrap: WARNING: dllwrap is deprecated, use gcc -shared or ld -shared instead

building 'wrapt._wrappers' extension
gcc -mdll -O -Wall -Ic:\python39\include -Ic:\python39\include -c src/wrapt/_wrappers.c -o build\temp.win32-3.9\Release\src\wrapt\_wrappers.o
writing build\temp.win32-3.9\Release\src\wrapt\_wrappers.cp39-win_amd64.def
gcc -shared -s build\temp.win32-3.9\Release\src\wrapt\_wrappers.o build\temp.win32-3.9\Release\src\wrapt\_wrappers.cp39-win_amd64.def -Lc:\python39\libs -Lc:\python39\PCbuild\win32 -lpython39 -lmsvcp140 -o build\lib.win32-3.9\wrapt\_wrappers.cp39-win_amd64.pyd
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lmsvcp140
collect2.exe: error: ld returned 1 exit status
***************************************************************************

WARNING: The C extension component for wrapt could not be compiled.

INFO: Trying to build without extensions.

***************************************************************************
running install
running build
running build_py
running install_lib
running install_egg_info
Removing c:\python39\Lib\site-packages\wrapt-1.12.1-py3.9.egg-info
Writing c:\python39\Lib\site-packages\wrapt-1.12.1-py3.9.egg-info
***************************************************************************

WARNING: The C extension component for wrapt could not be compiled.

INFO: Only pure Python version of package installed.

***************************************************************************

Yet msvcp140.dll exists.

Mini summary

Changing setup.py to accommodate python 3.9 and cygwincompiler.py to accommodate msvcp140.dll only reverted to the pure python installation.

Conclusion and question:

There was no way of accommodating Python 3.9 and MSVC version 1929 to install a non-pure version of wrapt.
The setup installation reverted to the pure python version.
I do have the latest pacmaned version of MSYS. Is there any way to use MSYS to install wrapt?

Thank you, Anthony of Sydney

AnthonyTheKoala commented 3 years ago

Dear Mr Dumpleton, All those attempted mini-corrections and modifications to the wrapt setup.py code and the cygwincompiler.py to accommodate Python 3.9 and MSVC 19.** did not work. I understand that the method of installation of wrapt is by distutils.

I have read https://github.com/GrahamDumpleton/wrapt/issues/178 and it appears that there may be a forthcoming version that does not rely on distutils as a method of installation.

I have gone back to the method of copying the wrapt folder from the Python38's c:\python38\lib\site-packages folder to the c:\python39\lib\site-packages folder

Thank you, Anthony of Sydney

MohammadKhalid commented 3 years ago

Hi @AnthonyTheKoala I'm unable to install wrapt can you please share your wrapt folder with me on my email, khalid.180695@gmail.com

@GrahamDumpleton Your build is failing. Please fix this issue. I'm unable to install TensorFlow 2 because of this issue. Thanks.

GrahamDumpleton commented 3 years ago

Have you tried:

pip install -U --no-cache-dir https://github.com/GrahamDumpleton/wrapt/archive/refs/heads/develop.zip

GrahamDumpleton commented 3 years ago

Closing out this issue. The original problem boiled down to something wrong with compiler installation on Windows. From 1.13.0 onwards of wrapt, binary wheels are included on PyPi which should avoid the need to have a working compiler installation, although if need other third party modules which have to be compiled from source code, you are still going to have. the same issue.

GrahamDumpleton / wrapt