search

PyO3 / pyo3

Rust bindings for the Python interpreter
https://pyo3.rs
Apache License 2.0
12.06k stars 744 forks source link

Move env! to env::var_os for buck determinism #4578

Open capickett opened 1 week ago

capickett commented 1 week ago

When building pyo3 with buck2 and remote execution, we're especially sensitive to deterministic builds and being able to compile different rust libraries on different hosts in parallel.

The issue with resolve_cross_compile_config_path was that we were embedding the full path of OUT_DIR into the compiled rlib and we are unable to guarantee the path is identical across all invocations of rustc. The other instances of env!("OUT_DIR") are no issue as they are used as part of the larger expression include_str!(concat!(env!("OUT_DIR"), "/pyo3-build-config-file.txt")); and we are able to guarantee that the contents of those files are deterministic.

By switching to env::var_os("OUT_DIR"), we are able to avoid embedding absolute paths into the compiled rust library and maintain backwards compatibility with the rest of the cargo ecosystem.

capickett commented 1 week ago

Looks like I need to update a changelog https://github.com/PyO3/pyo3/actions/runs/11042056229/job/30673632736?pr=4578

laudv commented 6 days ago

Related: https://github.com/PyO3/pyo3/pull/4555

davidhewitt commented 6 days ago

Thanks very much for the PR! I think you're right that there's a problem here, but I worry that this isn't the complete solution. I'll write up my observations in an issue now...

davidhewitt commented 6 days ago

See #4579