Move building of wheels to Github action

[jonathf]

Currently built manualy on CircleCI. To be able to support MacOS and Win, we ned to move to Azure Pipeline.

[nilsnolde]

or to github actions? I could help there a little, I already had a working setup for linux & mac wheels for a https://github.com/valhalla/valhalla wrapper with cibuildwheel. I can share the config later if you're interested.

[jonathf]

Cosidering that I haven't started on this task and you already have working builds with github action, I have no issues with switching.

Yes, please share the config.

[nilsnolde]

the code is closed by now, below is the config. a few comments:

like I said, it was a cmake-based build, so likely your usage of cibuildwheel will change. I forked the manylinux image and pre-installed all dependencies so it's easier on CI (valhalla used to have big ones..). for windows I remember the below didn't work, somehow there were problems with vcpkg (it's longer ago and the logs are gone). maybe there are better ways these days to handle the windows build chain. although it's mostly header libs here in vroom, so I guess it's enough to get the sources from somewhere on the web. the few linked dependencies here are optional and I couldn't easily build glpk for windows so plan mode is harder on win too.

actions.yml

name: manylinux_2_24 & Windows 2019
on:
  push:
    branches:
      - "release_branch"
    tags:
      - "*"
    paths-ignore:
      - '**.md'
      - '.circleci/*'
      - '.azure-pipelines.yml'
      - 'run-route-scripts/*'
      - 'test/*'
      - 'test_requests/*'
      - '.vcpkg_deps.txt'

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-20.04, macos-10.15, windows-2019]
    env:
      CIBW_ARCHS: x86_64
      VCPKG_DIR: ${{ github.workspace }}\vcpkg
      VCPKG_REF: 'f4bd6423'

    steps:
      - name: Checkout 
        uses: actions/checkout@v2  
        with:
          submodules: recursive

      - name: Install Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.x'

      - name: Install cibuildwheel
        run: python -m pip install cibuildwheel==2.0.0

      - name: Build Linux wheels
        if: matrix.os == 'ubuntu-20.04'
        run: |
          python -m cibuildwheel --output-dir wheelhouse
        env:
          CIBW_BUILD: cp37-manylinux_x86_64 cp38-manylinux_x86_64 cp39-manylinux_x86_64
          CIBW_MANYLINUX_X86_64_IMAGE: ghcr.io/gis-ops/manylinux:valhalla_py
          CIBW_TEST_COMMAND: cd {project}/test/bindings/python && python -m unittest test_utrecht.py -v
          CIBW_BEFORE_BUILD: 'python -m pip install --upgrade pip'

      - name: Build Mac wheels
        if: matrix.os == 'macos-10.15'
        run: |
          brew install ninja cmake protobuf cmake ccache libtool boost libspatialite pkg-config luajit curl wget czmq lz4 spatialite-tools unzip
          pip3 install requests
          git clone https://github.com/kevinkreiser/prime_server --recurse-submodules && cmake -B build . && cmake --build build && make -C build install
          python -m cibuildwheel --output-dir wheelhouse
        env:
          CIBW_BUILD:   cp37-macosx_x86_64 cp38-macosx_x86_64 cp39-macosx_x86_64
          CIBW_TEST_COMMAND: cd {project}/test/bindings/python && python -m unittest test_utrecht.py -v
          CIBW_BEFORE_BUILD: 'python -m pip install --upgrade pip'

      - name: Install vcpkg
        if: matrix.os == 'windows-2019'
        run: |
          git clone https://github.com/microsoft/vcpkg ${{ env.VCPKG_DIR }}
          git -C ${{ env.VCPKG_DIR }} checkout ${{ env.VCPKG_REF }}
          echo.set(VCPKG_BUILD_TYPE release)>> ${{ env.VCPKG_DIR }}\triplets\x64-windows.cmake
          call ${{ env.VCPKG_DIR }}\bootstrap-vcpkg.bat
          ${{ env.VCPKG_DIR }}\vcpkg.exe version
        shell: cmd

      - name: Install Win deps
        if: matrix.os == 'windows-2019'
        uses: lukka/run-vcpkg@v7
        with:
          setupOnly: false
          vcpkgDirectory: ${{ env.VCPKG_DIR }}
          appendedCacheKey: ${{ hashFiles('./.vcpkg_deps.txt') }}-${{ env.VCPKG_REF }}
          vcpkgTriplet: x64-windows
          vcpkgArguments: '@${{ github.workspace }}/.vcpkg_deps.txt'
          doNotUpdateVcpkg: true
          useShell: cmd

      - name: Build Windows wheels
        if: matrix.os == 'windows-2019'
        run: |
          dir ${{ env.VCPKG_DIR }}\scripts\buildsystems
          dir ${{ env.VCPKG_DIR }}\installed\x64-windows\include
          dir test\bindings\python
          python -m cibuildwheel --output-dir wheelhouse
        shell: cmd
        env:
          CIBW_BUILD: cp37-win_amd64 cp38-win_amd64 cp39-win_amd64
          CIBW_TEST_COMMAND: ''
          CIBW_ARCHS_WINDOWS: AMD64
          CMAKE_TOOLCHAIN_FILE: ${{ env.VCPKG_DIR }}\scripts\buildsystems\vcpkg.cmake
          LUA_INCLUDE_DIR: ${{ env.VCPKG_DIR }}\installed\x64-windows\include\luajit
          LUA_LIBRARIES: ${{ env.VCPKG_DIR }}\installed\x64-windows\lib\lua51.lib

      - name: Upload artifacts
        uses: actions/upload-artifact@v2
        with:
          path: ./wheelhouse/*.whl

  upload_pypi:
    needs: [build_wheels]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: artifact
          path: dist

      - uses: pypa/gh-action-pypi-publish@v1.4.2
        with:
          user: ${{ secrets.TWINE_USERNAME }}
          password: ${{ secrets.TWINE_PASSWORD }}
          repository_url: https://test.pypi.org/legacy/

[nilsnolde]

while porting your flow to our valhalla bindings (thanks again, I could finally get rid of the cmake abomination;)), I stumbled across this one: https://github.com/pybind/python_example/tree/1f4f73582cbfc2a0690b3930680abeab39820c03/.github/workflows. pretty slim

[jonathf]

This job is now complete.