Error in Azure Synapse - Sparkpool while adding zen-engine 0.22.0 in requirement.txt

[karkibipin]

Issue: Error Adding zen-engine to Requirements in Azure Synapse Spark Pool

Environment:

• PySpark version: 3.3.1.5.2.20240522.3
• Python version: 3.10.6 (packaged by conda-forge) [GCC 10.4.0]

Description:

I am encountering an error when adding zen-engine in the requirements.txt file for a Spark pool in Azure Synapse. The error suggests an issue with building the zen-engine==0.22.0 package during the installation process, specifically pointing to a failure in preparing metadata from the pyproject.toml file. Also, why a vanilla python install via pip-install works on all the env's, we've tried (w11/w10/mint/etc.) but the synapse environment is complaining about external dependencies

Error Message from requirements.txt:

Collecting zen-engine==0.22.0 (from -r /usr/lib/library-manager/bin/lmjob/sandbox/condaenv.gcir2h2z.requirements.txt (line 35)) Downloading zen_engine-0.22.0.tar.gz (124 kB) 124.8/124.8 kB 19.4 MB/s eta 0:00:00 Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error'

Error Message from Notebook in Spark Pool Session:

Collecting zen-engine Using cached zen_engine-0.23.0.tar.gz (125 kB) Installing build dependencies... done Getting requirements to build wheel... done Preparing metadata (pyproject.toml)... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Checking for Rust toolchain.... Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

Steps to Reproduce:

1. Add zen-engine==0.22.0 to the requirements.txt file.
2. Deploy the Spark pool in Azure Synapse.
3. Observe the installation failure during the deployment process.

Additional Information:

The error log indicates the need for Rust and Cargo, which are required to compile extensions for the zen-engine package. This suggests that the build environment in the Spark pool may not have Rust and Cargo installed.

Thank you!

[ivanmiletic]

We didn't work with Azure Synapse - Sparkpool.

Zen Engine is a native library and the core is written in Rust, we are compiling it to different architectures and on the first look it seems there isn't a build/package for that specific service / arch and that the system is trying to build it from source, otherwise it wouldnt need a rust/cargo.

[karkibipin]

Thanks for the feedback. I created a wheel and uploaded in synapse and its working.