Build wheels for Windows

[akaszynski]

This is a great module that I use on a regular basis. Could you please upload windows wheels to PyPi so that Windows users can install this easily?

If anyone needs them in the meantime, here's the binary wheels for 1.3.1 for Windows Python 3.5 and Python 3.6 wheels.zip

[djhoese]

We just had a satpy user run in to this issue on Windows as well. They didn't have the necessary compiler to build pykdtree. For the aggdraw project we have had @a-hurst trying out some wheel building tools for appveyor and travis. If it goes well for us maybe the lessons learned could be adopted by pykdtree.

https://github.com/pytroll/aggdraw/issues/38

[bwoodsend]

I've set up an AppVeyor project to build and test Windows and Mac wheels and stick them on PyPi. Currently it's just set to put them on TestPyPi. I just need your permission to switch it to actual PyPi then we can mark this as resolved. Would this be OK with you?

[djhoese]

@bwoodsend I would recommend making a pull request if you haven't already. Ideally we should make Windows, Mac, and Linux wheels so maybe your pull request can be that first step.

[akaszynski]

From my recent experience, I'm really in favor of using Azure devops for getting Windows, Linux, and MacOS built in one pipeline. If you'd like, I can build a pipeline for this project on azure.

[bwoodsend]

It's all on appveyor (using their UI) rather than in the repo itself so I don't actually have any changes to commit (other than this tiny update). It can export an appveyor.yml which I paste at the bottom. But I notice you've already got one for building with Conda which I didn't want to overwrite.

Ideally we should make Windows, Mac, and Linux wheels so maybe your pull request can be that first step.

PyPi is fussy about Linux wheels (see here). They have to be compiled using the manylinux docker image which AppVeyor doesn't have (I think). Linux normally comes with gcc so it's not such an issue to install from source? (It worked fine for me anyway).

@akaszynski If you're happy to do that then yes please. I assumed that, as this thread is a bit old and stale, you guys didn't have the time so I did it myself. But if you reckon you can do it reasonably painlessly on Azure...

version: 1.0.{build}
skip_non_tags: true
image:
- Visual Studio 2019
- macOS
shallow_clone: true
environment:
  TWINE_PASSWORD:
    secure: YFHOM630XJv9EA2eduDrTQ==
  TWINE_USERNAME: bwoodsend
build: off
test_script:
- pwsh: "If ($IsWindows) {\n    $PY_PATHS = @(\n           \"C:\\Python38\\python.exe\"; \"C:\\Python38-x64\\python.exe\";\n           \"C:\\Python37\\python.exe\"; \"C:\\Python37-x64\\python.exe\";\n           \"C:\\Python36\\python.exe\"; \"C:\\Python36-x64\\python.exe\";\n           \"C:\\Python35\\python.exe\"; \"C:\\Python35-x64\\python.exe\";\n                  )\n} ElseIf($IsLinux) {\n    # Turns out PyPi doesn't allow Linux wheels unless they are compiled on manylinux\n    # which appveyor doesn't have so no point using a Linux image. \n    # This will never be run.\n    $PY_PATHS = @(\"/home/appveyor/.localpython3.7.4/bin/python3\")\n} Else {\n    # Add other python versions when I learn where they are.\n    $PY_PATHS = @(\"/usr/local/opt/python/bin/python3.7\")\n}\n\n$env:PY_PATHS = $PY_PATHS\nforeach ($py in $PY_PATHS) {\n    $EXPRESSION = $py + \" -m pip install -U -q  --no-warn-script-location --user nose setuptools wheel\"\n    Invoke-Expression $EXPRESSION\n}\n\necho $PY_PATHS\nforeach ($py in $PY_PATHS) {\n    echo $py\n    ls\n    Invoke-Expression ($py + \" ./setup.py -q nosetests\")\n    Invoke-Expression ($py + \" ./setup.py -q sdist bdist_wheel\")\n}"
artifacts:
- path: ./dist/*.whl
  name: wheel
- path: ./dist/*.tar.gz
  name: source
deploy_script:
- pwsh: >-
    If ($IsWindows) {
      # Global variables don't seem to be global on windows.
      # PY_PATHS needs to be redifined.
      $PY_PATHS = @("C:\Python37-x64\python.exe")
    }

    # Pick a version of python to use twine to upload.

    foreach ($py in $PY_PATHS) {
        Invoke-Expression ($py + " -m pip install -U -q  --no-warn-script-location --user twine")

        # Login details are already in environment variable.
        Invoke-Expression ($py + " -m twine upload --skip-existing --repository-url https://test.pypi.org/legacy/ dist/* --non-interactive")

        break
    }

[djhoese]

@akaszynski That would be really nice. I have some experience doing it on Azure for the VisPy project using cibuildwheel (https://github.com/joerick/cibuildwheel). Although I haven't tried it I know there is also https://github.com/OpenAstronomy/azure-pipelines-templates/. How would you do it @akaszynski?

[akaszynski]

Building on Azure is a big of black magic. The trick for linux wheels is using a docker image. My pyvista/tetgen repo describes this process: https://github.com/pyvista/tetgen/blob/master/azure-pipelines.yml

The trick is here:

    - script: |
        set -ex
        docker run -e PLAT=$(plat) -e package_name=$(package_name) --rm -v `pwd`:/io $(image) /io/.ci/build_wheels.sh
        ls -lh wheelhouse/
        mkdir -p dist
        # cheat by renaming file to manylinux1 (even though it's really 2010)
        cd wheelhouse
        find . -type f -name '$(package_name)-*' | while read FILE ; do
          NEWFILE="$(echo ${FILE} |sed -e 's/\linux/manylinux1/')"
          cp "${FILE}" ../dist/"${NEWFILE}" ;
        done 
        cd ..
      displayName: Build wheels

What I'm doing here is using the quay.io/pypa/manylinux2010_x86_64 docker image to create these wheels. We might be able to use an older manylinux docker image, so we don't have to actually rename the wheel at the end. I'd have to experiment with this.

I'll move ahead with making a PR for building on Azure.

[akaszynski]

Also, I'll need to be added as an admin to the pykdtree pypi repo so I can push there. My username on PyPi is akaszynski. Please let me know when this is done so I can test it out.

[djhoese]

@akaszynski I am not an admin but work closely with @mraspaud who is. You should be able to make a PR with a config and @mraspaud can enable Azure access for this repository without making you an admin on the project.

I think maybe we should look at using cibuildwheel. The config ends up looking much simpler:

https://github.com/vispy/vispy/blob/master/azure-pipelines.yml

[akaszynski]

I've used cibuildwheel in the past and didn't feel it was functional enough, but as that was two years ago I'm willing on trying again, especially since vispy has figured it out. I'll work the PR, and there's no need to make me an admin.

[akaszynski]

Ah, nevermind, I see that you've added a PR. Cool! Glad we're building wheels for pykdtree!