Lastest release on pip (4.3.0.38) fails install on Windows (Python 2/3)

Breakthrough commented 4 years ago

Expected behaviour

pip install opencv-python should succeed gracefully on all targets. Works correctly for all versions earlier than 4.3.0.38 (running 4.2.0.32 locally).

Actual behaviour

Running pip install opencv-python fails on both Python 2 and 3 on Windows with the latest version (4.3.0.38).

Result under Python 3.8:

packages (from opencv-python) (1.19.1)
Building wheels for collected packages: opencv-python
  Building wheel for opencv-python (PEP 517) ... error

[[[[[ ATTEMPTS TO FIND A C COMPILER ]]]]]

  ERROR: Failed building wheel for opencv-python
Failed to build opencv-python
ERROR: Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly

Result under Python 2.7:

(py27) C:\Users\Breakthrough\Documents\VirtualEnvs\py27\Scripts>pip install opencv-python --upgrade
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting opencv-python
  Using cached opencv-python-4.3.0.38.tar.gz (88.0 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Users\Breakthrough\Documents\VirtualEnvs\py27\Scripts\python.exe' 'C:\Users\Breakthrough\Documents\VirtualEnvs\py27\lib\site-packages\pip\_vendor\pep517\_in_process.py' get_requires_for_build_wheel 'c:\users\Breakthrough\appdata\local\temp\tmpae1fpf'
       cwd: c:\users\Breakthrough\appdata\local\temp\pip-install-oxqgbg\opencv-python
  Complete output (22 lines):
  Traceback (most recent call last):
    File "C:\Users\Breakthrough\Documents\VirtualEnvs\py27\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 280, in <module>
      main()
    File "C:\Users\Breakthrough\Documents\VirtualEnvs\py27\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "C:\Users\Breakthrough\Documents\VirtualEnvs\py27\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "c:\users\Breakthrough\appdata\local\temp\pip-build-env-alnx4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 146, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "c:\users\Breakthrough\appdata\local\temp\pip-build-env-alnx4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "c:\users\Breakthrough\appdata\local\temp\pip-build-env-alnx4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 243, in run_setup
      self).run_setup(setup_script=setup_script)
    File "c:\users\Breakthrough\appdata\local\temp\pip-build-env-alnx4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 448, in <module>
      main()
    File "setup.py", line 99, in main
      % {"ext": re.escape(sysconfig.get_config_var("EXT_SUFFIX"))}
    File "C:\Python27\lib\re.py", line 210, in escape
      s = list(pattern)
  TypeError: 'NoneType' object is not iterable
  ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\Breakthrough\Documents\VirtualEnvs\py27\Scripts\python.exe' 'C:\Users\Breakthrough\Documents\VirtualEnvs\py27\lib\site-packages\pip\_vendor\pep517\_in_process.py' get_requires_for_build_wheel 'c:\users\Breakthrough\appdata\local\temp\tmpae1fpf' Check the logs for full command output.

Steps to reproduce

Install 4.3.0.38 via pip on any Windows machine. It fails on both Python 2.7 and 3.x. Linux seems okay for Python 3.x, haven't tested Python 2.7, and not an OSX user.

This seems to be affecting my Travis.CI and ReadTheDocs builds as well, which makes me think this might not be related to the latest changes I pushed to my repo, but rather, the latest release of this package (or one of the releases since version 4.2.0.32).

harlowja commented 4 years ago

Can confirm we are seeing this in our CI as well (running python2.7 on ubuntu).

[31m ERROR: Command errored out with exit status 1:

command: /home/jenkins/workspace/datavault_PR-1768/.venv/bin/python /home/jenkins/workspace/datavault_PR-1768/.venv/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpMPFq4u

cwd: /tmp/pip-install-jdznwV/opencv-python

Complete output (22 lines):

Traceback (most recent call last):

File "/home/jenkins/workspace/datavault_PR-1768/.venv/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>

main()

File "/home/jenkins/workspace/datavault_PR-1768/.venv/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main

json_out['return_val'] = hook(**hook_input['kwargs'])

File "/home/jenkins/workspace/datavault_PR-1768/.venv/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel

return hook(config_settings)

File "/tmp/pip-build-env-qO9YN0/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 146, in get_requires_for_build_wheel

return self._get_build_requires(config_settings, requirements=['wheel'])

File "/tmp/pip-build-env-qO9YN0/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires

self.run_setup()

File "/tmp/pip-build-env-qO9YN0/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 243, in run_setup

self).run_setup(setup_script=setup_script)

File "/tmp/pip-build-env-qO9YN0/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 142, in run_setup

exec(compile(code, __file__, 'exec'), locals())

File "setup.py", line 448, in <module>

main()

File "setup.py", line 99, in main

% {"ext": re.escape(sysconfig.get_config_var("EXT_SUFFIX"))}

File "/usr/lib/python2.7/re.py", line 210, in escape

s = list(pattern)

TypeError: 'NoneType' object is not iterable

----------------------------------------[0m

[31mERROR: Command errored out with exit status 1: /home/jenkins/workspace/datavault_PR-1768/.venv/bin/python /home/jenkins/workspace/datavault_PR-1768/.venv/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpMPFq4u Check the logs for full command output.[0m

skvark commented 4 years ago

Python 2.7 support was dropped in 4.2.0.34 (https://github.com/skvark/opencv-python/releases/tag/34). However, the setup.py is missing python_requires='>=3.5', I will add it in the next release. After that Python 2.7 should be ignored properly in new releases. Currently pip picks up the source distribution from the latest release for 2.7 because it thinks it's the latest release for 2.7 while it really isn't. I will see if I can manually edit the PyPI metadata to fix this issue after all builds have completed.

If you are using Python 3, then the issue is that the new source distribution (https://github.com/skvark/opencv-python/releases/tag/38) is kicking in because the Windows builds haven't yet completed (it takes almost a day to build all Windows wheels...):

https://ci.appveyor.com/project/skvark/opencv-python/builds/34563450

I highly recommend to pin your dependencies to certain versions and manually bump the version when needed to avoid this kind of issues.

skvark commented 4 years ago

Summary:

@Breakthrough Pin your opencv-python version to avoid the issue. Otherwise there will be always a short period of time (10 - 24 hours) during new opencv-python releases when your builds will fail since all opencv-python wheels haven't yet been uploaded to PyPI.

@harlowja PyPI metadata cannot be updated (https://github.com/pypa/warehouse/issues/2170) so there's nothing I can do to fix the Python 2.7 issue. Since you are using Python 2.7, pin your opencv-python version to latest supported version which is 4.2.0.32.

Python 2.7 is considered legacy software and hasn't been supported here since its discontinuation in January 2020. Please update to Python 3: https://github.com/skvark/opencv-python/issues/152

Breakthrough commented 4 years ago

@skvark is it possible to defer uploading the new package until the wheels have been built? I was hoping to avoid falling into dependency hell, but it seems like with this release process, it's unavoidable now if the latest releases won't gracefully install after a period of time... So I might have to pin the version as you mentioned, was just trying to avoid that until absolutely necessary.

Is this something that could be resolved by infrastructure changes, or am I maybe missing something? Also, is there a way to specify different requirement files across Python 2.7 or 3, or should I just pin my package to 4.2.0.32 until I deprecate Python 2.7?

skvark commented 4 years ago

OpenCV is a very heavy package to build and I'm limited by the computing power of the free CI instances which Travis and Appveyor are providing. This issue would not exist if I had 40+ concurrent workers in the CI systems which is obviously not possible.

I'm not aware of any built-in feature in Travis which would allow deferring deployment. Appveyor stores the build artifacts, but still logic for detecting successful builds and uploading would have to be built separately. So webhooks / serverless stuff etc. which I would have to manage and host. Not really in the scope of this project. I do store the Travis build artifacts to Azure Storage, but that already costs me a couple of euros per month (we are talking here hundreds of gigabytes of build artifacts).

Version pinning is something which should be always done to ensure repeatable builds: https://pip.pypa.io/en/stable/user_guide/#pinned-version-numbers

Yeah, requirements.txt supports environment markers (I did not test the syntax, but you get the idea):

opencv-python==4.2.0.32 ; python_version < '3.0'
opencv-python ; python_version >= '3.5'

Breakthrough commented 4 years ago

@skvark Thank you for the explanation and the link. Will work on getting that setup for my package, much appreciated!

Also many thanks for maintaining this awesome package!

opencv / opencv-python