Environment variables not being recognized

[MaxHalford]

Hello. First of all I would like to convey my respect for the project, it's great. However, I've been banging my head for the last hours trying to figure how the environment variables affect the wheels. My understanding is that we have to define the environment variables with the env directive, as so:

language: python

python:
  - 3.6
  - 3.7

cache:
  apt: true
  directories:
    - $HOME/.cache/pip
    - $HOME/downloads

env:
  global:
    - CIBW_BUILD="cp36-* cp37-*"
    - CIBW_BEFORE_BUILD="pip install cython"

matrix:
  include:
    - sudo: required
      services:
        - docker
      env: PIP=pip
    - os: osx
      language: generic
      env: PIP=pip2

install: skip
script: ./ci/test.sh ${TRAVIS_PYTHON_VERSION}

jobs:
  include:

  - stage: pypi
    install: pip install cibuildwheel==0.12.0
    script: cibuildwheel --output-dir dist
    deploy:
      provider: pypi
      user: MaxHalford
      password:
        secure: ""
      on:
        all_branches: true
      skip_existing: true
    if: tag IS present

I am thus specifying that I only want to build wheels for Python 3.6 and 3.7. What's more I want to install cython before building the wheels. The issue is that the CIBW_BUILD="cp36-* cp37-*" and CIBW_BEFORE_BUILD="pip install cython" variables don't seem to be taken into account. I looked at another project, namely tgcrypto, and this seems to be exactly what they're doing. Here is what I get when I run this on Travis:

cibuildwheel version 0.12.0

Build options:
  platform: 'linux'
  before_build: None
  build_selector: BuildSelector('*' - '')
  build_verbosity: 0
  environment: ParsedEnvironment([])
  manylinux1_images: {'x86_64': None, 'i686': None}
  output_dir: 'dist'
  project_dir: '.'
  test_command: None
  test_extras: ''
  test_requires: []

Here we go!

Docker version 18.06.0-ce, build 0ffa825
docker command: docker create --env CIBUILDWHEEL --name cibuildwheel-f2c572e9-7b6f-4be9-8165-af8579edd5d4 -i -v /:/host quay.io/pypa/manylinux1_x86_64 /bin/bash
Unable to find image 'quay.io/pypa/manylinux1_x86_64:latest' locally
latest: Pulling from pypa/manylinux1_x86_64
7d0d9526f38a: Pulling fs layer
6b63be20283e: Pulling fs layer
0ddc3c810aa4: Pulling fs layer
b2001bd6db20: Pulling fs layer
b2001bd6db20: Waiting
6b63be20283e: Verifying Checksum
6b63be20283e: Download complete
0ddc3c810aa4: Verifying Checksum
0ddc3c810aa4: Download complete
7d0d9526f38a: Verifying Checksum
7d0d9526f38a: Download complete
7d0d9526f38a: Pull complete
6b63be20283e: Pull complete
0ddc3c810aa4: Pull complete
b2001bd6db20: Pull complete
Digest: sha256:b5c4a09d232fda6cf60c4c28dd27eae3f845007aea3364cf2f5561d8c8e0ab68
Status: Downloaded newer image for quay.io/pypa/manylinux1_x86_64:latest
46f57d03a5bb5d4350f1724facab8ba4eeebecd3b647ebe2ff6b1ac651b2fcf7
docker command: docker cp /home/travis/build/creme-ml/creme/. cibuildwheel-f2c572e9-7b6f-4be9-8165-af8579edd5d4:/project
docker command: docker start -i -a cibuildwheel-f2c572e9-7b6f-4be9-8165-af8579edd5d4
+ mkdir /output
+ cd /project
+ for PYBIN in /opt/python/cp27-cp27m/bin /opt/python/cp27-cp27mu/bin /opt/python/cp34-cp34m/bin /opt/python/cp35-cp35m/bin /opt/python/cp36-cp36m/bin /opt/python/cp37-cp37m/bin
+ rm -rf /tmp/built_wheel
+ rm -rf /tmp/delocated_wheel
+ mkdir /tmp/built_wheel
+ mkdir /tmp/delocated_wheel
+ '[' '!' -z '' ']'
+ PATH=/opt/python/cp27-cp27m/bin:/opt/rh/devtoolset-2/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ /opt/python/cp27-cp27m/bin/pip wheel . -w /tmp/built_wheel --no-deps
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Processing /project
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-nm2fsj/setup.py", line 150, in <module>
        ext_modules=cythonize([Extension('*', sources=['**/*.pyx'], libraries=['m'])])
      File "/tmp/pip-req-build-nm2fsj/setup.py", line 19, in cythonize
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-nm2fsj/
WARNING: You are using pip version 19.1.1, however version 19.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
docker command: docker rm --force -v cibuildwheel-f2c572e9-7b6f-4be9-8165-af8579edd5d4
cibuildwheel-f2c572e9-7b6f-4be9-8165-af8579edd5d4
The command "cibuildwheel --output-dir dist" exited with 1.
cache.2
store build cache
Done. Your build exited with 1.

It's attempting wheels for Python 2.7, 3.4, and 3.5 but I told it not too. Moreover it's not installing cython as I asked it too, and thus failing. Any idea what's going on?

[MaxHalford]

I just tried adding the environment variables via the matrix field in my .travis.yml file but it didn't work.

[YannickJadoul]

Sorry we had missed this!

But so indeed, cibuildwheel is not picking up your environment variables: see the line build_selector: BuildSelector('*' - '').

This is on Linux, right? Could you try running env before running cibuildwheel? That should print all your environment variables. E.g.:

script: |
  env
  cibuildwheel --output-dir dist

[MaxHalford]

No worries! Although this is a bit pressing, as I am trying to deploy the new version of creme.

Here is the output:

PERLBREW_SHELLRC_VERSION=0.86
MANPATH=/home/travis/.nvm/versions/node/v8.12.0/sare/man:/home/travis/.rvm/rubies/ruby-2.5.3/share/man:/usr/local/cmake-3.12.4/man:/usr/loca/clang-7.0.0/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/home/travis/.rvm/man
TRAVIS_ARCH=amd64
XDG_SESSION_ID=2
TRAVIS_FILTERED=redirect_io
MYSQL_UNIX_PORT=/var/run/mysqld/mysqld.sock
rvm_bin_path=/home/travis/.rvm/bi
HAS_JOSH_K_SEAL_OF_APPROVAL=true
PYENV_ROOT=/opt/pyenv
NVM_CD_FLAGS=
GEM_HOME=/home/travis/.rvm/gems/ruby-2.5.3
TRAVIS_STACK_JOB_BOARD_REGISTER=/.job-board-register.yml
TRAVIS_TEST_RESULT
TRAVIS_STACK_LANGUAGES=__sardonyx_ c c++ clojure cplusplus cpp default generic go groovy java node_js php pure_java python ruby scala
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
IRBRC=/home/travis/.rvm/rubies/ruby-2.5.3/.irbrc
SSH_CLIENT=10.16.4.12 58100 22
TRAVIS_COMMIT=9b94b7bada2e573260d8389d7def3beb5ec4824a
TRAVIS_OS_NAME=linux
TRAVIS_APT_PROXY=http://apt.cache.travis-ci.com
TRAVIS_JOB_NAME=
CASHER_DIR=/home/travis/.casher
TRAVIS_UID=2000
TRAVIS_INTERNAL_RUBY_REGEX=^ruby-(2\.[0-4]\.[0-9]|1\.9\.3)
OLDPWD=/home/travis/build
MY_RUBY_HOME=/home/travis/.rvm/rubies/ruby-2.5.3
TRAVIS_ROOT=/
SSH_TTY=/dev/pts/0
TRAVIS_TIMER_ID=04ea33d8
LC_ALL=en_US.UTF-8
ANSI_GREEN=\033[32;1m
NVM_DIR=/home/travis/.nvm
HISTFILESIZE=2000
USER=travis
_system_type=Linux
TRAVIS_LANGUAGE=python
TRAVIS_INFRA=unknown
ANSI_RESET=\033[0m
rvm_path=/home/travis/.rvm
TRAVIS_DIST=xenial
TRAVIS=true
TRAVIS_REPO_SLUG=creme-ml/creme
ANSI_YELLOW=\033[33;1m
HAS_ANTARES_THREE_LITTLE_FRONZIES_BADGE=true
PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs4 --with-wide-uniode --enable-shared --enable-ipv6 --enable-loadable-sqlite-extensions --with-computed-gotos
VIRTUAL_ENV=/home/travis/virtualenv/python3.6.7
TRAVIS_BUILD_STAGE_NAME=Pypi
TRAVIS_COMMIT_MESSAGE=
Run ENV
TRAVIS_PULL_REQUEST=false
PAGER=cat
RACK_ENV=test
TRAVIS_CMD=env
PERLBREW_ROOT=/home/travis/perl5/perlbrew
CODECOV_TOKEN=[secure]
TRAVIS_STACK_TIMESTAMP=2019-03-25 16:16:56 UTC
rvm_prefix=/home/travis
PIP_PROGRESS_BAR=off
TRAVIS_PYTHON_VERSION=3.6
PYTHON_CFLAGS=-g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security
PATH=/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/.rvm/bin:/home/travis/virtualenv/python3.6.7/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:sr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin
MAIL=/var/mail/travis
TRAVIS_PULL_REQUEST_SHA=
TRAVIS_OSX_IMAGE=
TRAVIS_JOB_WEB_URL=https://travis-ci.org/creme-ml/creme/jobs/601734467
TRAVIS_TMPDIR=/tmp/tmp.KgcPSEaQue
TRAVIS_BUILD_WEB_URL=https://travis-ci.org/creme-ml/creme/builds/601734464
APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1
PWD=/home/travis/build/creme-ml/creme
JAVA_HOME=/usr/local/lib/jvm/openjdk11
CONTINUOUS_INTEGRATION=true
LANG=en_US.UTF-8
TRAVIS_PRE_CHEF_BOOTSTRAP_TIME=2019-03-25T16:16:42
MERB_ENV=test
TZ=UTC
_system_arch=x86_64
PERLBREW_HOME=/home/travis/.perlbrew
PS1=(python3.6.7) ${debian_chroot:+($debian_chroot)}\u@\h:\w\$ 
TRAVIS_ENABLE_INFRA_DETECTION=true
_system_version=16.04
TRAVIS_SUDO=true
TRAVIS_TAG=
TRAVIS_ALLOW_FAILURE=false
RBENV_SHELL=bash
HISTCONTROL=ignoredups:ignorespace
TRAVIS_HOME=/home/travis
TRAVIS_INIT=systemd
rvm_version=1.29.7 (latest)
TRAVIS_JOB_NUMBER=513.3
TRAVIS_EVENT_TYPE=push
PYENV_SHELL=bash
PS4=+
HOME=/home/travis
SHLVL=2
GOROOT=/home/travis/.gimme/versions/go1.11.1.linux.amd64
LANGUAGE=en_US.UTF-8
ANSI_CLEAR=\033[0K
RAILS_ENV=test
TRAVIS_TIMER_START_TIME=1571824302892858293
CI=true
TRAVIS_BUILD_ID=601734464
LOGNAME=travis
PIP_DISABLE_PIP_VERSION_CHECK=1
TRAVIS_STACK_FEATURES=basic couchdb disabled-ipv6 docker docker-compose elasticsearch firefox go-toolchain google-chrome jdk memcached mongodb mysql nodejs_interpreter perl_interpreter perlbrew phantomjs postgresql python_interpreter redis ruby_interpreter sqlite xserver
TRAVIS_PULL_REQUEST_SLUG=
COMPOSER_NO_INTERACTION=1
GEM_PATH=/home/travis/.rvm/gems/ruby-2.5.3:/home/travis/.rvm/gems/ruby-2.5.3@global
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
SSH_CONNECTION=10.16.4.12 58100 10.20.255.1 22
LC_CTYPE=en_US.UTF-8
TRAVIS_SECURE_ENV_VARS=true
DEBIAN_FRONTEND=noninteractive
NVM_BIN=/home/travis/.nvm/versions/node/v8.12.0/bin
GOPATH=/home/travis/gopath
TRAVIS_STACK_NODE_ATTRIBUTES=/.node-attributes.yml
TRAVIS_STACK_NAME=sardonyx
TRAVIS_APP_HOST=build.travis-ci.org
GIT_ASKPASS=echo
TRAVIS_BRANCH=travis-wheels
XDG_RUNTIME_DIR=/run/user/2000
TRAVIS_COMMIT_RANGE=5472b4ddf73b...9b94b7bada2e
JRUBY_OPTS= --client -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 -J-Xss2m -Xcompile.invokedynamic=false
TRAVIS_PULL_REQUEST_BRANCH=
TRAVIS_JOB_ID=601734467
ANSI_RED=\033[31;1m
RUBY_VERSION=ruby-2.5.3
_system_name=Ubuntu
TRAVIS_BUILD_NUMBER=513
TRAVIS_BUILD_DIR=/home/travis/build/creme-ml/creme
GH_TOKEN=[secure]

It seems the variables are not being set... So this is probably a Travis issue. By the way the Travis file is stored in this branch.

[YannickJadoul]

Huh, that's weird. But cibuildwheel not picking them up would also be weird.

Probably it's Travis, then, yes. Is the "stages" functionality still in beta?

[MaxHalford]

I just tried to explicitly set the environment variables in the script step, but they still didn't appear in the output of the env command, I guess it still is in beta...

[YannickJadoul]

I just tried to explicitly set the environment variables in the script step, but they still didn't appear in the output of the env command

Wait, really? That's really weird! What about export CIBW_BUILD=...? Otherwise, CIBW_BUILD=... cibuildwheel ... (setting them in one line, before calling cibuildwheel) should definitely work, no?

[MaxHalford]

Okay I just managed to fix it. When using stages, you actually have to set any environment variables via the env field, as so:

- stage: pypi
    env:
      - CIBW_BUILD="cp36-* cp37-*"
      - CIBW_BEFORE_BUILD="pip install cython"
    install: pip install cibuildwheel==0.12.0
    script: cibuildwheel --output-dir dist

I must have skipped this when initially reading the docs. Thanks for your time and sorry for wasting it!

[MaxHalford]

It works really great! Now I just have to figure how to also build wheels for MacOS, as the matrix expansion doesn't apply to stages.

[YannickJadoul]

Ha, great to hear! Thanks for reporting back; it's good to know if other people run into the same issues.

It's quite silly that these matrix expansions don't apply to stages, though? Then again, you only need 2 jobs, one for Linux and one for macOS, and cibuildwheel will handle the rest ;)

[MaxHalford]

No worries!

Well I'm not exactly sure how that pans out. I need this to be a single stage because I then want to upload all the wheels to PyPI. My understanding is that to build MacOS wheels I need to use the osx environment on Travis. I'll see what I manage to do :).

Again, congrats on making such a great tool.

[YannickJadoul]

@MaxHalford It's quite complex, but I have the same (with stages) in my project. You can have a look at the .travis.yml file, if you want: https://github.com/YannickJadoul/Parselmouth/blob/515b635fb50a9161e0d58944a9feee691a2ddf9b/.travis.yml#L80-L173