Pycurl 7.43.0.3 binary builds missing (windows & linux)

laura-surcel commented 5 years ago

Hello!

I am contacting you because I use PyCurl in a work related project and currently we always try to install latest version. However, since yesterday this doesn't work because the wheels for version 7.43.0.3 for windows are missing. We are using Python 2.7.8 (32-bits) on windows.

Is there an ETA for the upload of the new wheels on pypi?

Thank you, Laura Surcel

nelsonjchen commented 5 years ago

The fix I did for #567 with #568 should make Appveyor produce wheels with acceptable emails.

p commented 5 years ago

There are currently no official binary builds for 7.43.0.3 published. Contributions are welcome.

The next step for the potential contributor would be to verify that the current builds produced by appveyor meet all of the requirements to be uploaded to pypi.

nelsonjchen commented 5 years ago

The issue is that someone needs to make sure that these wheels can be uploaded to PyPI and ensure that they pass PyPI's validation as mentioned in #567. What I'm really baffled about is that #568 was changing something that has existed for a long time. How were the wheels in the past even generated to be uploaded? It might be a new validation on PyPI's part.

Anyway, to validate, I suppose someone needs to make a new project on PyPI, upload the generated wheels and report if PyPI accepts the wheels and/or it rejects it for some reason.

On Wed, Jun 19, 2019 at 2:48 AM laura-surcel notifications@github.com wrote:

Ok. One of my colleagues made a fork of your repo and got wheels generated with appveyor. We tested them with our product and no issues appeared. Are there other tests that should be run so they could be made public?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pycurl/pycurl/issues/569?email_source=notifications&email_token=AAABJ43Q2PJ3LHKRXHNRZFDP3H6HHA5CNFSM4HY7R6PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYBKMOI#issuecomment-503490105, or mute the thread https://github.com/notifications/unsubscribe-auth/AAABJ44JLKKXUHEO2PNIEWLP3H6HHANCNFSM4HY7R6PA .

p commented 5 years ago

Perhaps the wheels built in appveyor are built differently from the ones obtained via winbuild.py. On the other hand, I succeeded in uploading wheels that had seemingly garbage python versions (or that had file names that pypi split in weird places obtaining garbage python versions).

Another bit that would be helpful, if we are to upload appveyor builds to pypi, is a tool/feature added to setup.py to download all binaries for the most recent master build in appveyor.

nelsonjchen commented 5 years ago

New

Metadata-Version: 2.1
Name: pycurl
Version: 7.43.0.3
Summary: PycURL -- A Python Interface To The cURL library
Home-page: http://pycurl.io/
Author: Kjetil Jacobsen, Markus F.X.J. Oberhumer, Oleg Pudeyev
Author-email: kjetilja@gmail.com, markus@oberhumer.com, oleg@bsdpower.com
Maintainer: Oleg Pudeyev
Maintainer-email: oleg@bsdpower.com
License: LGPL/MIT
Keywords: curl,libcurl,urllib,wget,download,file transfer,http,www
Platform: All
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
Classifier: Topic :: Internet :: WWW/HTTP

Old

Metadata-Version: 2.0
Name: pycurl
Version: 7.43.0.2
Summary: PycURL -- A Python Interface To The cURL library
Home-page: http://pycurl.io/
Author: Oleg Pudeyev
Author-email: oleg@bsdpower.com
License: LGPL/MIT
Description-Content-Type: UNKNOWN
Keywords: curl,libcurl,urllib,wget,download,file transfer,http,www
Platform: All
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
Classifier: Topic :: Internet :: WWW/HTTP

I'm looking into how these differences came to be.

nelsonjchen commented 5 years ago

I have no idea why these differences happened.

Anyway, it does look like a post #568 build does upload. I don't have permission to upload to pycurl but I at least get a permission error and not an "invalid value" error.

%   twine upload --repository testpypi dist/*                                    06/30/19 - 11:31AM
Uploading distributions to https://test.pypi.org/legacy/
Uploading pycurl-7.43.0.3-cp27-cp27m-win32.whl
100%|███████████████████████████████████████████████████████████| 1.23M/1.23M [00:01<00:00, 747kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Client Error: 'kjetilja at gmail.com, markus at oberhumer.com, oleg at bsdpower.com' is an invalid value for Author-email. Error: Use a valid email address See https://packaging.python.org/specifications/core-metadata for url: https://test.pypi.org/legacy/
(twine)
nelson@nobynoby:~/code/pycurl-wheels
%   cd dist                                                                 :( 1 06/30/19 - 11:31AM
(twine)
nelson@nobynoby:~/code/pycurl-wheels/dist
%   cd ../                                                                       06/30/19 - 11:33AM
(twine)
nelson@nobynoby:~/code/pycurl-wheels
%   mv ~/Downloads/pycurl-7.43.0.3-cp27-cp27m-win32.whl dist                     06/30/19 - 11:33AM
(twine)
nelson@nobynoby:~/code/pycurl-wheels
%   twine upload --repository testpypi dist/*                                    06/30/19 - 11:33AM
Uploading distributions to https://test.pypi.org/legacy/
Uploading pycurl-7.43.0.3-cp27-cp27m-win32.whl
100%|███████████████████████████████████████████████████████████| 1.23M/1.23M [00:01<00:00, 725kB/s]
NOTE: Try --verbose to see response content.
HTTPError: 403 Client Error: The user 'nelsonjchen' isn't allowed to upload to project 'pycurl'. See https://test.pypi.org/help/#project-name for more information. for url: https://test.pypi.org/legacy/
(twine)
nelson@nobynoby:~/code/pycurl-wheels
%                                                                           :( 1 06/30/19 - 11:33AM

arossert commented 5 years ago

@nelsonjchen any news regarding when we will have the latest 7.43.0.3 version .whl available?

nelsonjchen commented 5 years ago

@arossert I don't know, I'm in the same boat as you. I can say that the blocker isn't present anymore. I don't know when @p will do another release.

arossert commented 5 years ago

@p can you please upload the latest version binaries using @nelsonjchen fix?

arossert commented 5 years ago

@nelsonjchen for windows I'm currently using the "Unofficial Windows Binaries for Python Extension Packages" by Christoph Gohlke: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pycurl

nelsonjchen commented 5 years ago

@arossert I don't think @p has a high opinion of those builds but it is probably stems more from that this project didn't build those wheels.

nelsonjchen commented 5 years ago

Actually, I think it's more that we can't see the steps exactly used to put those together like in some public CD system. They're a little mysterious.

matteius commented 5 years ago

Ok -- Please let me know what I can do here to take this over and get the right builds into pypi. I have spent 12 hours split across multiple sessions trying various things related to https://github.com/celery/kombu/issues/1035 and I have come to the following conclusions:

Building pylibcurl from source has way to many dependencies to expect downstream developers to ever have to do this just to use pylibcurl in their project.
Both building on my local Windows machine is challenging requiring many extra dependencies (which *nix based systems make slightly better by having cmake and gmake and possibly easier install of libcurl)
Curl offers asynchronous requests from Python that you just can't get from requests -- so we can't really convert off curl until there is some other viable thing that works better for us.
AppVeyor is a total PITA -- I used os.list commands to manually explore the file system and try to get at where the various dependencies to build from setup.py are and while I got pretty far I still never got it to fully build there (inside kombu's project) and it involved many ridiculous steps like prebuilding some header files and checking them into my own fork of pycurl because apparently setup.py just doesn't generate these.
Removing pycurl from kombu for now as the ticket author suggests may be the only options limits us from writing or executing any tests or integration tests (at least within Appveyor, but probably elsewhere) because we simply cannot install pycurl dependency.
Installing pycurl dependency anywhere, including downstream appveyor project build, would be ridiculously easy if prebuilt binaries existed.

So that being said -- I want to help make this happen. How can I help you build the binaries one time so that we all can re-use them pain free? What is the flow here -- it seems like above there is some talk of a discrepancy between what appveyor is building for you guys (I am amazed you have appveyor building this project at all) and some other build you are talking about. Let's make a plan -- best path forward is having binaries in pypi for the latest versions.

nelsonjchen commented 5 years ago

@matteius I usually debug Appveyor using RDP. https://www.appveyor.com/docs/how-to/rdp-to-build-worker/

It's actually something not available on Azure Pipelines/GitHub Actions 2.

matteius commented 5 years ago

Ok I re-read this thread and realized that @nelsonjchen has the fix and @p is the blocker because he hasn't replied to this thread yet, but looking at his profile he made some other github project commtis this week (other projects). Any chance we can get some attention on this @p or promote someone like @nelsonjchen to have enough permissions to help us out here?

p commented 5 years ago

I appreciate the offer to assist with windows builds. Here is additional information regarding them.

Windows does not have a package manager as part of the OS as Linux distributions do. Therefore on windows pycurl (or pycurl's distribution/release process) also builds libcurl and all of libcurl's dependencies, whereas on all other OSes pycurl just needs to build itself.
The time expenditure in windows builds is almost entirely allocated to configuring & building libcurl and libcurl's dependencies. Again unlike Unix-like operating systems, windows doesn't have the ./configure && make && make install setup. Pretty much every single one of libcurl's dependencies has its own build system, with different prerequisites, configuration systems, output hierarchies, etc. This is why winbuild.py requires cmake and perl and nasm, for instance.
Keep in mind that some of libcurl's dependencies need(ed) to be patched to use the correct flavor of msvcrt.
Appveyor does not build pycurl's dependencies, to keep the build times down. These are pre-built and pre-packaged. See https://github.com/pycurl/pycurl/blob/master/winbuild.py#L1160. As part of the release process these dependencies need to be rebuilt to their current versions. Therefore simply getting the current binary of pycurl off of appveyor isn't really ideal because it'll include 1 or 2 year old libcurl in it.
For an aspiring windows pycurl builder, I suggest getting winbuild.py running on their local machine. You'll probably need to scan through it a few times to orient yourself as to what it does, but hopefully once you've figured that out it's more or less touch & go. I figure there is at least 2 weeks of FTE effort in that file alone, so it's probably better to use it than to redo everything from scratch, but ultimately whoever does the work gets to say how the work is done and if you want to do it your way (or, say, use https://github.com/pycurl/pycurl/issues/584) that's also fine.
I think I couldn't figure out how to create an organization in appveyor, so tell me what I need to do to add someone else to the project or make it into an org.
Anyone who produced binary builds is welcome to create a PR in https://github.com/pycurl/downloads to have those builds added to the official channels.

I don't mind answering questions etc. but I don't presently have the bandwidth to do the actual builds on windows.

p commented 5 years ago

On another note, winbuild.py was written when I supported more pythons than I do today. I believe these days I just upload windows binaries for 2.7, 3.6 and 3.7. I'll merge a PR removing everything from winbuild.py having to do with older Pythons to make it smaller, if this helps.

arossert commented 4 years ago

@p is there any estimation on when we could expect this to be done?

p commented 4 years ago

@arossert Since there is not currently a volunteer to do the work, I would say there is no ETA.

matteius commented 4 years ago

I had relative ease in building pycurl on my new windows tablet book last night, and I took a different approach this time. I discovered technically the Windows tool chain does have a package manager which is Microsoft's vcpkg and this is what I used to build libcurl (the files libcurl.lib, libcurl.dll, etc.) but that requires Step1. 1.) Install Windows Visual Studio 2017 Community (make sure you check support for C/C++ build tools). To make things easier and not require cygwin, I copied the binaries for cl.exe etc. over to my Git mingw64 shell path binaries so that pycurl can build in future step3. 2.) git Clone https://github.com/Microsoft/vcpkg/ and follow initial steps there to build vcpkg, and the steps here to build libcurl: https://curl.haxx.se/docs/install.html [Building using vcpkg] 3.) git clone pycurl and run: python setup.py docstrings python setup.py install --curl-dir=/c/Users/matt/Projects/vcpkg/packages/curl_x86-windows which is where my curl package from vcpkg was built.

So it builds and installs on Windows -- I am less familiar with the process of building wheels files for distribution.

matteius commented 4 years ago

I'll say that the winbuild.py file is quite a mess and its complicated because it wants to build the wheels for multiple versions of python. I have a hunch that something like pyenv may be better suited here but I have not as much expertise with pyenv to make that happen immediately. However I went the next step of I can install pycurl locally, can I build the wheels using setup.py bdist_wheel and the answer was kind of, but not entirely.

EDIT: Apparently -k will keep the build artifact that I was complaining was removed. Well I think I built the wheel but I need to learn more about what that looks like packaged. I have:

But I do have the tmp and lib build artifacts which looks promising.

matt@DESKTOP-PFCBDC9 MINGW64 ~/Projects/pycurl (master)
$ python setup.py bdist_wheel --curl-dir=/c/Users/matt/Projects/vcpkg/packages/curl_x86-windows
Using curl directory: C:/Users/matt/Projects/vcpkg/packages/curl_x86-windows
Not using an SSL library
running bdist_wheel
running build
running build_py
running build_ext
installing to build\bdist.win32\wheel
running install
running install_lib
creating build\bdist.win32\wheel
creating build\bdist.win32\wheel\curl
copying build\lib.win32-3.8\curl\__init__.py -> build\bdist.win32\wheel\.\curl
copying build\lib.win32-3.8\pycurl.cp38-win32.pyd -> build\bdist.win32\wheel\.
running install_data
creating build\bdist.win32\wheel\pycurl-7.43.0.3.data
creating build\bdist.win32\wheel\pycurl-7.43.0.3.data\data
creating build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc
creating build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying AUTHORS -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying ChangeLog -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying COPYING-LGPL -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying COPYING-MIT -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying INSTALL.rst -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying README.rst -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
copying RELEASE-NOTES.rst -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl
creating build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\basicfirst.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\file_upload.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\linksys.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\opensocketexception.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\retriever-multi.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\retriever.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\sfquery.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\smtp.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\ssh_keyfunction.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
copying examples\xmlrpc_curl.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples
creating build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\file_upload_buffer.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\file_upload_real.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\file_upload_real_fancy.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\follow_redirect.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\form_post.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\get.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\get_python2.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\get_python2_https.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\get_python3.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\get_python3_https.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\put_buffer.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\put_file.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\response_headers.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\response_info.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
copying examples\quickstart\write_file.py -> build\bdist.win32\wheel\pycurl-7.43.0.3.data\data\doc\pycurl\examples\quickstart
running install_egg_info
running egg_info
writing pycurl.egg-info\PKG-INFO
writing dependency_links to pycurl.egg-info\dependency_links.txt
writing top-level names to pycurl.egg-info\top_level.txt
reading manifest file 'pycurl.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'src\Makefile'
writing manifest file 'pycurl.egg-info\SOURCES.txt'
Copying pycurl.egg-info to build\bdist.win32\wheel\.\pycurl-7.43.0.3-py3.8.egg-info
running install_scripts
adding license file "COPYING-LGPL" (matched pattern "COPYING*")
adding license file "COPYING-MIT" (matched pattern "COPYING*")
adding license file "AUTHORS" (matched pattern "AUTHORS*")
creating build\bdist.win32\wheel\pycurl-7.43.0.3.dist-info\WHEEL
creating 'dist\pycurl-7.43.0.3-cp38-cp38-win32.whl' and adding 'build\bdist.win32\wheel' to it
adding 'pycurl.cp38-win32.pyd'
adding 'curl/__init__.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/AUTHORS'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/COPYING-LGPL'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/COPYING-MIT'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/ChangeLog'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/INSTALL.rst'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/README.rst'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/RELEASE-NOTES.rst'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/basicfirst.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/file_upload.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/linksys.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/opensocketexception.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/retriever-multi.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/retriever.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/sfquery.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/smtp.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/ssh_keyfunction.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/xmlrpc_curl.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/file_upload_buffer.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/file_upload_real.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/file_upload_real_fancy.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/follow_redirect.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/form_post.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/get.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/get_python2.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/get_python2_https.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/get_python3.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/get_python3_https.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/put_buffer.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/put_file.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/response_headers.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/response_info.py'
adding 'pycurl-7.43.0.3.data/data/doc/pycurl/examples/quickstart/write_file.py'
adding 'pycurl-7.43.0.3.dist-info/AUTHORS'
adding 'pycurl-7.43.0.3.dist-info/COPYING-LGPL'
adding 'pycurl-7.43.0.3.dist-info/COPYING-MIT'
adding 'pycurl-7.43.0.3.dist-info/METADATA'
adding 'pycurl-7.43.0.3.dist-info/WHEEL'
adding 'pycurl-7.43.0.3.dist-info/top_level.txt'
adding 'pycurl-7.43.0.3.dist-info/RECORD'

matteius commented 4 years ago

Ok cool -- I just downloaded the latest wheel for pycurl from pypi and extracted it like a zip file and it contains the same file structure as the one I generated locally. One difference is the pycurl.cp38-win32.pyd I built locally is only 118 KB but the one I download from pypi is between 2.1 or 2.8 MB depending on the CPU arch.

matteius commented 4 years ago

@p I made a PR to the Downloads. It took a while to clone my fork of the Downloads so it can't be an optimal way to maintain the downloads/whl files. We need to have a longer term goal of how to manage those files and the build dependencies. I am playing around with options like pyenv-win to see if I can build multiple python versions on my machines architecture. Please let me know if this PR is helpful, and if you can account for the size of the WHL file being smaller or not: https://github.com/pycurl/downloads/pull/4

matteius commented 4 years ago

I got pyenv-win working and used it to launch various other python installers. Right now I have the latest python2.7 and 2.8 for both 32 bit and 64 bit architectures and I ran through the exercise of generating .whl files for each one.

The trick to building a 64 bit version is installing the right libcurl 64 bit version using vcpkg and to point at that. To build this way I copied the 64 bit version of the MS2017 VC toolset to git shell and also Set VCPKG_DEFAULT_TRIPLET=x64-windows in the Environment Variables settings in Windows.

matteius commented 4 years ago

@arossert Can you try the WHL files I have generated here: https://github.com/pycurl/downloads/pull/4

p commented 4 years ago

According to https://github.com/microsoft/vcpkg/blob/master/ports/curl/CONTROL ssl is optional, was libcurl built with it enabled?

It is also not clear to me if one can get a windows build with openssl. I am curious whether winssl supports/implements all of the libcurl callbacks that pycurl exposes.

arossert commented 4 years ago

@matteius thanks for all the effort. Sure, I will check it later today.

matteius commented 4 years ago

@p for whatever reason I had to manually copy the corresponding libcurl.dll and zlib1.dll files into the pycurl egg that was installed -- but once I did that I was able to do this SSL request of Google:

$ pyenv exec python -i
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
from io import BytesIO

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, b'https://www.google.com')
c.setopt(c.WRITEDATA, buffer)
c.perform()
c.close()

body = buffer.getvalue()
print(body)>>>
b'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world\'s information, including webpages, images, videos and more. Google has many special features to help you find exactly what you\'re looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script nonce="UWzbHXnSz2uvDDeWsgFSXg==">(function(){window.google={kEI:\'iv8NXuecD8245gKgqo-AAQ\',kEXPI:\'0,1353746,5663,731,223,4727,378,206,467,2488,249,10,168,545,338,175,364,925,194,35,3,278,4,60,315,427,208,10,593,99,1128776,143,1197751,398,38,329080,1294,12383,4855,22661,10031,15247,867,2236,15208,11240,363,3320,5505,8384,4858,1362,4323,4968,773,2249,2822,1923,3110,6204,1719,1808,1976,2044,5766,1,3142,5297,2974,873,37,1180,2975,4399,2032,1141,3912,3538,2094,621,2883,18,320,3172,975,1,369,2779,518,399,992,1285,8,2796,967,612,14,1279,2212,202,328,149,1103,840,517,318,1148,4,4,48,158,662,3438,260,52,1135,1,3,2063,606,1839,184,595,1138,43,144,377,686,1261,245,502,219,64,1,145,44,1009,93,328,1284,16,84,417,1760,666,44,1595,608,473,1339,29,1758,3093,134,773,1548,524,7,729,591,1574,3193,201,1276,69,3,6510,299,2533,257,582,60,299,681,1042,129,246,2084,2632,56,129,2,711,1228,1494,371,1274,108,977,269,26,1001,565,88,481,809,99,2,317,116,507,715,139,117,118,1160,366,127,97,529,130,991,258,265,361,1154,9,276,2,149,585,458,228,1257,519,14,52,355,202,1019,781,39,8,1,794,46,60,1,2,1,112,9,1,2,16,146,138,333,109,373,701,258,497,26,201,400,410,189,6,15,276,426,142,1,250,14,87,363,324,157,218,376,384,86,18,159,1324,157,5859137,3198,1802696,4194595,256,2801171,549,333,444,1,2,80,1,900,896,1,8,1,2,2551,1,748,141,59,736,563,1,4265,1,1,1,1,137,1,802,77,9,162,5,7,2,4,7,3,3,2,4,222,3365626,20598661\',authuser:0,kscs:\'c9c918f0_iv8NXuecD8245gKgqo-AAQ\',kGL:\'US\',kBL:\'v77x\'};google.sn=\'webhp\';google.kHL=\'en\';google.jsfs=\'Ffpdje\';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.time=function(){return(new Date).getTime()};google.log=function(a,b,e,c,g){if(a=google.logUrl(a,b,e,c,g)){b=new Image;var d=google.lc,f=google.li;d[f]=b;b.onerror=b.onload=b.onabort=function(){delete d[f]};google.vel&&google.vel.lu&&google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,e,c,g){var d="",f=google.ls||"";e||-1!=b.search("&ei=")||(d="&ei="+google.getEI(c),-1==b.search("&lei=")&&(c=google.getLEI(c))&&(d+="&lei="+c));c="";!e&&google.cshid&&-1==b.search("&cshid=")&&"slh"!=a&&(c="&cshid="+google.cshid);a=e||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+d+f+"&zx="+google.time()+c;/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};(function(){document.documentElement.addEventListener("submit",function(b){var a;if(a=b.target){var c=a.getAttribute("data-submitfalse");a="1"==c||"q"==c&&!a.elements.q.value?!0:!1}else a=!1;a&&(b.preventDefault(),b.stopPropagation())},!0);}).call(this);var a=window.location,b=a.href.indexOf("#");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf("#")&&a.replace("/search?"+c.replace(/(^|&)fp=[^&]*/g,"")+"&cad=h")};</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}\n</style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/nav_logo229.png) 0 -261px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}</style><script nonce="UWzbHXnSz2uvDDeWsgFSXg=="></script></head><body bgcolor="#fff"><script nonce="UWzbHXnSz2uvDDeWsgFSXg==">(function(){var src=\'/images/nav_logo229.png\';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}\nif (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}\n}\n})();</script><div id="mngb"> <div id=gbar><nobr><b class=gb1>Search</b> <a class=gb1 href="https://www.google.com/imghp?hl=en&tab=wi">Images</a> <a class=gb1 href="https://maps.google.com/maps?hl=en&tab=wl">Maps</a> <a class=gb1 href="https://play.google.com/?hl=en&tab=w8">Play</a> <a class=gb1 href="https://www.youtube.com/?gl=US&tab=w1">YouTube</a> <a class=gb1 href="https://news.google.com/nwshp?hl=en&tab=wn">News</a> <a class=gb1 href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="https://drive.google.com/?tab=wo">Drive</a> <a class=gb1 style="text-decoration:none" href="https://www.google.com/intl/en/about/products?tab=wh"><u>More</u> &raquo;</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href="http://www.google.com/history/optout?hl=en" class=gb4>Web History</a> | <a  href="/preferences?hl=en" class=gb4>Settings</a> | <a target=_top id=gb_70 href="https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/" class=gb4>Sign in</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div> </div><center><br clear="all" id="lgpd"><div id="lga"><img alt="Google" height="92" src="/images/branding/googlelogo/1x/googlelogo_white_background_color_272x92dp.png" style="padding:28px 0 14px" width="272" id="hplogo"><br><br></div><form action="/search" name="f"><table cellpadding="0" cellspacing="0"><tr valign="top"><td width="25%">&nbsp;</td><td align="center" nowrap=""><input name="ie" value="ISO-8859-1" type="hidden"><input value="en" name="hl" type="hidden"><input name="source" type="hidden" value="hp"><input name="biw" type="hidden"><input name="bih" type="hidden"><div class="ds" style="height:32px;margin:4px 0"><input class="lst" style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top" autocomplete="off" value="" title="Google Search" maxlength="2048" name="q" size="57"></div><br style="line-height:0"><span class="ds"><span class="lsbb"><input class="lsb" value="Google Search" name="btnG" type="submit"></span></span><span class="ds"><span class="lsbb"><input class="lsb" id="tsuid1" value="I\'m Feeling Lucky" name="btnI" type="submit"><script nonce="UWzbHXnSz2uvDDeWsgFSXg==">(function(){var id=\'tsuid1\';document.getElementById(id).onclick = function(){if (this.form.q.value){this.checked = 1;if (this.form.iflsig)this.form.iflsig.disabled = false;}\nelse top.location=\'/doodles/\';};})();</script><input value="AAP1E1EAAAAAXg4NmjpUe4obIXC3jBmcoMdxr2uP_G3M" name="iflsig" type="hidden"></span></span></td><td class="fl sblc" align="left" nowrap="" width="25%"><a href="/advanced_search?hl=en&amp;authuser=0">Advanced search</a><a href="/language_tools?hl=en&amp;authuser=0">Language tools</a></td></tr></table><input id="gbv" name="gbv" type="hidden" value="1"><script nonce="UWzbHXnSz2uvDDeWsgFSXg==">(function(){var a,b="1";if(document&&document.getElementById)if("undefined"!=typeof XMLHttpRequest)b="2";else if("undefined"!=typeof ActiveXObject){var c,d,e=["MSXML2.XMLHTTP.6.0","MSXML2.XMLHTTP.3.0","MSXML2.XMLHTTP","Microsoft.XMLHTTP"];for(c=0;d=e[c++];)try{new ActiveXObject(d),b="2"}catch(h){}}a=b;if("2"==a&&-1==location.search.indexOf("&gbv=2")){var f=google.gbvu,g=document.getElementById("gbv");g&&(g.value=a);f&&window.setTimeout(function(){location.href=f},0)};}).call(this);</script></form><div id="gac_scont"></div><div style="font-size:83%;min-height:3.5em"><br></div><span id="footer"><div style="font-size:10pt"><div style="margin:19px auto;text-align:center" id="fll"><a href="/intl/en/ads/">Advertising\xa0Programs</a><a href="/services/">Business Solutions</a><a href="/intl/en/about.html">About Google</a></div></div><p style="color:#767676;font-size:8pt">&copy; 2020 - <a href="/intl/en/policies/privacy/">Privacy</a> - <a href="/intl/en/policies/terms/">Terms</a></p></span></center><script nonce="UWzbHXnSz2uvDDeWsgFSXg==">(function(){window.google.cdo={height:0,width:0};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b){var c=window.document,d="CSS1Compat"==c.compatMode?c.documentElement:c.body;a=d.clientWidth;b=d.clientHeight}a&&b&&(a!=google.cdo.width||b!=google.cdo.height)&&google.log("","","/client_204?&atyp=i&biw="+a+"&bih="+b+"&ei="+google.kEI);}).call(this);})();(function(){var u=\'/xjs/_/js/k\\x3dxjs.hp.en_US.PXQ6TDUi4mU.O/m\\x3dsb_he,d/am\\x3dAAMCbAQ/d\\x3d1/rs\\x3dACT90oHRqO4y1Tn0VIAvndn1UG1WhY4Dyg\';\nsetTimeout(function(){var b=document;var a="SCRIPT";"application/xhtml+xml"===b.contentType&&(a=a.toLowerCase());a=b.createElement(a);a.src=u;google.timers&&google.timers.load&&google.tick&&google.tick("load","xjsls");document.body.appendChild(a)},0);})();(function(){window.google.xjsu=\'/xjs/_/js/k\\x3dxjs.hp.en_US.PXQ6TDUi4mU.O/m\\x3dsb_he,d/am\\x3dAAMCbAQ/d\\x3d1/rs\\x3dACT90oHRqO4y1Tn0VIAvndn1UG1WhY4Dyg\';})();function _DumpException(e){throw e;}\nfunction _F_installCss(c){}\n(function(){google.spjs=false;google.snet=true;google.em=[];google.emw=false;})();(function(){var pmc=\'{\\x22d\\x22:{},\\x22sb_he\\x22:{\\x22agen\\x22:true,\\x22cgen\\x22:true,\\x22client\\x22:\\x22heirloom-hp\\x22,\\x22dh\\x22:true,\\x22dhqt\\x22:true,\\x22ds\\x22:\\x22\\x22,\\x22ffql\\x22:\\x22en\\x22,\\x22fl\\x22:true,\\x22host\\x22:\\x22google.com\\x22,\\x22isbh\\x22:28,\\x22jsonp\\x22:true,\\x22msgs\\x22:{\\x22cibl\\x22:\\x22Clear Search\\x22,\\x22dym\\x22:\\x22Did you mean:\\x22,\\x22lcky\\x22:\\x22I\\\\u0026#39;m Feeling Lucky\\x22,\\x22lml\\x22:\\x22Learn more\\x22,\\x22oskt\\x22:\\x22Input tools\\x22,\\x22psrc\\x22:\\x22This search was removed from your \\\\u003Ca href\\x3d\\\\\\x22/history\\\\\\x22\\\\u003EWeb History\\\\u003C/a\\\\u003E\\x22,\\x22psrl\\x22:\\x22Remove\\x22,\\x22sbit\\x22:\\x22Search by image\\x22,\\x22srch\\x22:\\x22Google Search\\x22},\\x22ovr\\x22:{},\\x22pq\\x22:\\x22\\x22,\\x22refpd\\x22:true,\\x22rfs\\x22:[],\\x22sbpl\\x22:24,\\x22sbpr\\x22:24,\\x22scd\\x22:10,\\x22sce\\x22:5,\\x22stok\\x22:\\x22EnwRJaEYTG7Yd8z4AdVsCGo0sWM\\x22,\\x22uhde\\x22:false}}\';google.pmc=JSON.parse(pmc);})();</script>        </body></html>'
>>>

matteius commented 4 years ago

I think that is the size difference between the wheel files that already exist and the ones I built -- the ones I built don't include the .dll within the generated pyd. I am not sure how to get the .DLL file included on the build -- I have tried:

python setup.py bdist_wheel --curl-dir=/c/Users/matt/Projects/vcpkg/packages/curl_x64-windows --use-libcurl-dll

However the option use-libcurl-dll seems to do nothing. In either case I have to manually copy the DLL and lib to the pycurl egg once it has been installed. How have others gotten it to include in the pyd file?

p commented 4 years ago

I am not aware of whether there is a way to include arbitrary shared libraries in wheels, and if so how to attain that. For this reason I have always been building pycurl statically against all of its dependencies on windows.

However the option use-libcurl-dll seems to do nothing.

As stated in https://github.com/pycurl/pycurl/blob/master/setup.py#L922 this option tells setup.py to link dynamically against the libcurl dll rather than include libcurl statically. This doesn't package the libcurl dll that is linked against in such manner.

p commented 4 years ago

Windows binaries added to https://pypi.org/project/pycurl/#files.

treysis commented 4 years ago

Unfortunately, not for python2.7 which was the original question (yes, I know 2.7 is EOL by now).

pycurl / pycurl

Pycurl 7.43.0.3 binary builds missing (windows & linux) #569