Add CUDA build

[pearu]

This PR requires https://github.com/conda-forge/arrow-cpp-feedstock/pull/125

Checklist

• [x] Used a fork of the feedstock to propose changes
• [x] Bumped the build number (if the version is unchanged)
• [ ] Reset the build number to 0 (if the version changed)
• [x] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• [x] Ensured the license file is being packaged.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• Failed to even lint the recipe, probably because of a conda-smithy bug :cry:. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• The recipe must have some tests.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[pearu]

@isuruf @jakirkham @xhochy , this PR has two issues that I have failed to fix:

• osx_python3.7.____cpython fails with conflicts while py3.6 and py3.8 cases are ok
• %PYTHON% is undefined in bld.bat

Any help is appreciated!

[isuruf]

For the 2nd, please read conda-build docs on activate_in_script

[xhochy]

osx_python3.7.____cpython stalled on the arrow-cpp side. I pushed an empty commit to restart CI as it was not possible to re-trigger the Job through the UI.

[pearu]

@xhochy @jakirkham and others. This PR provides pyarrow conda package with CUDA enabled. Please review.

[h-vetinari]

@pearu If you're comfortable with making me a collaborator on your fork of https://github.com/conda-forge/pyarrow-feedstock, I'm happy to help on this recipe as soon as conda-forge/conda-forge-ci-setup-feedstock#93 & conda-forge/docker-images#136 get merged (conda-forge/docker-images#135 is already in).

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• The build section contained an unexpected subsection name. activate_in_script is not a valid subsection name.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

@conda-forge-admin, please rerender

[github-actions[bot]]

Hi! This is the friendly automated conda-forge-webservice. I tried to rerender for you, but it looks like I wasn't able to push to the pearu/enable-cuda branch of Quansight/pyarrow-feedstock. Did you check the "Allow edits from maintainers" box?

[h-vetinari]

@conda-forge-admin, please rerender

[github-actions[bot]]

Hi! This is the friendly automated conda-forge-webservice. I tried to rerender for you, but it looks like I wasn't able to push to the pearu/enable-cuda branch of Quansight/pyarrow-feedstock. Did you check the "Allow edits from maintainers" box?

[h-vetinari]

So, I was wrong about the usefulness about building with cuda 10.x, as @isuruf noted in another PR on the arrow-cpp-feedstock. To wit (IIUC): since arrow only uses the cuda driver (and not the toolkit), building for one cuda version is compatible with all.

I reverted the corresponding commit.

[h-vetinari]

Nothing I touched should have affected the osx/gandiva builds, but there's an error

import: 'pyarrow.gandiva'
Traceback (most recent call last):
  File "/Users/runner/miniforge3/conda-bld/pyarrow_1589551487060/test_tmp/run_test.py", line 11, in <module>
    import pyarrow.gandiva
ImportError: dlopen([...snip...]/lib/python3.8/site-packages/pyarrow/gandiva.cpython-38-darwin.so, 2): Library not loaded: @rpath/libre2.6.dylib
  Referenced from: [...snip...]/lib/libgandiva.16.dylib
  Reason: image not found

Not sure where this libre2.6.dylib should be coming from?

Same error on linux64 (but not on aarch/ppc):

import: 'pyarrow.gandiva'
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/pyarrow_1589551467439/test_tmp/run_test.py", line 11, in <module>
    import pyarrow.gandiva
ImportError: libre2.so.6: cannot open shared object file: No such file or directory

[pearu]

Nothing I touched should have affected the osx/gandiva builds, but there's an error

import: 'pyarrow.gandiva'

CUDA enabled arrow-cpp 0.16 disables gandiva for ppc64le and aarch64, see https://github.com/conda-forge/arrow-cpp-feedstock/blob/0.16.x/recipe/build-arrow.sh#L11

CUDA enabled arrow-cpp 0.17, however, will enable gandiva support, see https://github.com/conda-forge/arrow-cpp-feedstock/pull/137 .

However, for this PR, that is arrow-cpp 0.16 based, the gandiva support must be disabled.

[pearu]

Same error on linux64 (but not on aarch/ppc):

import: 'pyarrow.gandiva'
...
ImportError: libre2.so.6: cannot open shared object file: No such file or directory

The latest re2 package (v 2020.05) provides libre2.so.7 but arrow-cpp 0.16 was built against re2 package v 2020.04 that provides libre2.so.6.

[pearu]

This PR is blocked by https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/93 .

[xhochy]

Same error on linux64 (but not on aarch/ppc):

import: 'pyarrow.gandiva'
...
ImportError: libre2.so.6: cannot open shared object file: No such file or directory

The latest re2 package (v 2020.05) provides libre2.so.7 but arrow-cpp 0.16 was built against re2 package v 2020.04 that provides libre2.so.6.

Made a hotfix PR: https://github.com/conda-forge/cf-mark-broken/pull/53

[h-vetinari]

@conda-forge-admin, please restart ci

[h-vetinari]

@pearu @xhochy The GPU builds are green now 🥳 Just waiting for aarch/ppc to finish.

[pearu]

@h-vetinari @xhochy @isuruf , are there any issues left that prevent landing this PR?

[xhochy]

Ignore the suggestions, if CI passes again, we can merge.

[github-actions[bot]]

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

• linter: passed
• travis: passed
• azure: passed

Thus the PR was passing and merged! Have a great day!