[Archery][CI][Integration] Integration workflow is no longer running Rust integration tests

[paleolimbot]

Describe the bug, including details regarding any error messages, version, and platform.

Discovered while trying to modify the integration workflow to get nanoarrow's integration tests to run (#39302). Despite issuing the archery docker run invocation:

https://github.com/apache/arrow/blob/22f88fa4a8f5ac7250f1845aace5a78d20006ef2/.github/workflows/integration.yml#L98-L100

We get:

DEBUG:archery:Executing `['docker', 'run', '--rm', '-e', 'ARCHERY_DEFAULT_BRANCH=main', '-e', 'ARCHERY_INTEGRATION_WITH_RUST=1', '-e', 'ARCHERY_INTEGRATION_WITH_RUST=0', '-e', 'ARROW_CPP_EXE_PATH=/build/cpp/debug', '-e', 'ARROW_RUST_EXE_PATH=/build/rust/debug', '-e', 'CCACHE_COMPILERCHECK=content', '-e', 'CCACHE_COMPRESS=1', '-e', 'CCACHE_COMPRESSLEVEL=6', '-e', 'CCACHE_DIR=/ccache', '-e', 'CCACHE_MAXSIZE=1G', '-e', 'GITHUB_ACTIONS=true', '-v', '/home/runner/work/arrow/arrow:/arrow', '-v', '/home/runner/work/arrow/arrow/.docker/conda-ccache:/ccache', 'apache/arrow-dev:amd64-conda-integration', '/arrow/ci/scripts/integration_arrow_build.sh /arrow /build && /arrow/ci/scripts/integration_arrow.sh /arrow /build']`

https://github.com/apache/arrow/actions/runs/8907059610/job/24460263757#step:8:345

...which results in

 + /arrow/ci/scripts/rust_build.sh /arrow /build
=====================================================================
Not building the Rust implementation.
=====================================================================

The place where this command appears to be calculated is below. It seems that it's doubling the parameters that are included in the docker-compose.yml file (overriding the values in docker-compose.yml seems to be the only way I was able to get Rust to build).

https://github.com/apache/arrow/blob/22f88fa4a8f5ac7250f1845aace5a78d20006ef2/dev/archery/archery/docker/core.py#L59-L169

Component(s)

Archery, Continuous Integration, Integration

[paleolimbot]

They're also failing to build: https://github.com/apache/arrow/actions/runs/9007030425/job/24745896123?pr=39302#step:9:516

[tustvold]

I've filed https://github.com/apache/arrow/issues/41637 with a suggested fix.

In the meantime we have downgraded the Rust toolchain version to 1.77 in arrow-rs' CI to workaround this issue - https://github.com/apache/arrow-rs/pull/5761

[alamb]

Also possibly of interest is https://github.com/apache/arrow-rs/pull/5761 which may have started running the nanoarrow tests in arrow-rs