[CI] Reduce the number of Azure Pipeline jobs

[pitrou]

Describe the enhancement requested

Crossbow execution resources on Azure Pipelines are scarce, while they are quite ample on Github Actions. It would therefore be probably beneficial to move most Azure Pipeline jobs to Github Actions.

Component(s)

Continuous Integration

[pitrou]

@assignUser @raulcd @kou @nealrichardson

[kou]

Here is a list of jobs (and their template files) that use Azure Pipelines:

$ grep -B1 -A1 'ci: azure' dev/tasks/tasks.yml | grep -v 'ci: azure' | grep -v '^--'
  conda-clean:
    template: conda-recipes/azure.clean.yml
  conda-linux-x64-cpu-r41:
    template: conda-recipes/azure.linux.yml
  conda-linux-x64-cpu-r42:
    template: conda-recipes/azure.linux.yml
  conda-linux-x64-cpu-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-x64-cuda-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cpu-r41:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cpu-r42:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cpu-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-aarch64-cuda-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-ppc64le-cpu-py3:
    template: conda-recipes/azure.linux.yml
  conda-linux-ppc64le-cuda-py3:
    template: conda-recipes/azure.linux.yml
  conda-osx-x64-cpu-r41:
    template: conda-recipes/azure.osx.yml
  conda-osx-x64-cpu-r42:
    template: conda-recipes/azure.osx.yml
  conda-osx-x64-cpu-py3:
    template: conda-recipes/azure.osx.yml
  conda-osx-arm64-cpu-r41:
    template: conda-recipes/azure.osx.yml
  conda-osx-arm64-cpu-r42:
    template: conda-recipes/azure.osx.yml
  conda-osx-arm64-cpu-py3:
    template: conda-recipes/azure.osx.yml
  conda-win-x64-cpu-r41:
    template: conda-recipes/azure.win.yml
  conda-win-x64-cpu-py3:
    template: conda-recipes/azure.win.yml
  conda-win-x64-cuda-py3:
    template: conda-recipes/azure.win.yml
  test-conda-cpp-valgrind:
    template: docker-tests/azure.linux.yml
  test-debian-11-python-3:
    template: docker-tests/azure.linux.yml
  test-ubuntu-20.04-python-3:
    template: docker-tests/azure.linux.yml
  test-fedora-35-python-3:
    template: docker-tests/azure.linux.yml
  test-r-linux-valgrind:
    template: docker-tests/azure.linux.yml
  test-r-depsource-bundled:
    template: r/azure.linux.yml
  test-r-offline-minimal:
    template: r/azure.linux.yml
  test-r-rhub-debian-gcc-release-custom-ccache:
    template: r/azure.linux.yml
  test-r-{{ r_org }}-{{ r_image }}-{{ r_tag }}:
    template: r/azure.linux.yml
  test-r-rhub-debian-gcc-devel-lto-latest:
    template: r/azure.linux.yml
  test-r-rstudio-r-base-4.2-centos7-devtoolset-8:
    template: r/azure.linux.yml
  test-r-minimal-build:
    template: r/azure.linux.yml
  test-ubuntu-r-sanitizer:
    template: docker-tests/azure.linux.yml
  test-fedora-r-clang-sanitizer:
    template: docker-tests/azure.linux.yml
  test-debian-11-go-{{ go_version }}:
    template: docker-tests/azure.linux.yml

Only template files:

$ grep -B1 -A1 'ci: azure' dev/tasks/tasks.yml | grep -v 'ci: azure' | grep -v '^--' | grep template | sort | uniq
    template: conda-recipes/azure.clean.yml
    template: conda-recipes/azure.linux.yml
    template: conda-recipes/azure.osx.yml
    template: conda-recipes/azure.win.yml
    template: docker-tests/azure.linux.yml
    template: r/azure.linux.yml

• conda-recipes/*: We need to rewrite all files to use GitHub Actions instead of Azure Pipelines...
• docker-tests/azure.linux.yml: We can just switch to docker-tests/github.linux.yml and remove docker-tests/azure.linux.yml
• r/azure.linux.yml: We need to rewrite this to use GitHub Actions instead of Azure Pipelines.

[assignUser]

fyi: I removed the 10 r-arrow conda builds: #39243