MarcoGorelli
commented
3 weeks ago thanks @langslike for the report! i'll take a look
Open langslike opened 3 weeks ago
MarcoGorelli
commented
3 weeks ago thanks @langslike for the report! i'll take a look
MarcoGorelli
commented
3 weeks ago i can't reproduce this - could you show the exact commands you ran please?
langslike
commented
3 weeks ago the command is : maturin develop. No more than this. I have tried uninstall the cookiecutter, polars and maturin and reinstall , then create the plugin again, then compile, still the same error. I can compile and run the crate from maturin tutorial : https://www.maturin.rs/tutorial I tried to use older version of polars, modify the cargo.toml and then removed the cargo.lock, but still not working: polars = { version = "0.41.0", default-features = false } polars-arrow = { version = "0.41.0", default-features = false } Is there any way I can force maturin to use specified version of polars-arrow to compile?
It sames that the problem is I cannot compile polars-arrow 0.42.0(the reason is unknown), After I downgrade polars to 0.41.3, pyo3-polars to 0.15.0, pyo3 to 0.21.0,command 'cargo build' could walk through the dependency compile stage, but failed eventually on Compiling minimal_plugin because older pyo3 do not have some features we need:
Blocking waiting for file lock on package cache
Updating rsproxy-sparse index
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Compiling psm v0.1.21
Compiling stacker v0.1.16
Compiling pyo3-build-config v0.21.2
Compiling polars-utils v0.41.3
Compiling pyo3-macros-backend v0.21.2
Compiling pyo3-ffi v0.21.2
Compiling pyo3 v0.21.2
Compiling polars-arrow v0.41.3
Compiling pyo3-macros v0.21.2
Compiling polars-compute v0.41.3
Compiling polars-row v0.41.3
Compiling polars-core v0.41.3
Compiling polars-parquet v0.41.3
Compiling polars-io v0.41.3
Compiling polars-ops v0.41.3
Compiling polars v0.41.3
Compiling pyo3-polars v0.15.0
Compiling minimal_plugin v0.1.0 (D:\Project\playground\quant\polars_plugin\minimal_plugin)
error[E0432]: unresolved import pyo3_polars::derive
--> src\expressions.rs:3:18 |
3 | use pyo3_polars::derive::polars_expr; | ^^^^^^ could not find derive in pyo3_polars |
|---|
note: found an item that was configured out
--> C:\Users\roar.cargo\registry\src\rsproxy.cn-0dccff568467c15b\pyo3-polars-0.15.0\src/lib.rs:45:9
|
45 | pub mod derive;
| ^^^^^^
= note: the item is gated behind the derive feature
error[E0432]: unresolved import pyo3_polars::PolarsAllocator
--> src\lib.rs:4:5 |
4 | use pyo3_polars::PolarsAllocator; | ^^^^^^^^^^^^^--------------- |
|---|---|---|---|
help: a similar name exists in the module: PolarsIterator |
|||
no PolarsAllocator in the root |
error[E0599]: no method named apply_into_string_amortized found for reference &ChunkedArray<StringType> in the current scope
--> src\expressions.rs:9:33 |
9 | let out: StringChunked = ca.apply_into_string_amortized( | value: &str, output: &mut String | { | ---^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|---|
help: there is a method apply_to_slice with a similar name, but with different arguments
--> C:\Users\roar.cargo\registry\src\rsproxy.cn-0dccff568467c15b\polars-core-0.41.3\src\chunked_array\ops\mod.rs:229:5
|
229 | / fn apply_to_slice<F, S>(&'a self, f: F, slice: &mut [S])
230 | | // (value of chunkedarray, value of slice) -> value of slice
231 | | where
232 | | F: Fn(Option
warning: unused import: PyModuleMethods
--> src\lib.rs:2:29 |
2 | use pyo3::types::{PyModule, PyModuleMethods}; | ^^^^^^^^^^^^^^^ |
|---|
= note: #[warn(unused_imports)] on by default
Some errors have detailed explanations: E0432, E0599.
For more information about an error, try rustc --explain E0432.
warning: minimal_plugin (lib) generated 1 warning
error: could not compile minimal_plugin (lib) due to 3 previous errors; 1 warning emitted
MarcoGorelli
commented
3 weeks ago Sorry I mean, could you please show all the steps you took in full please?
For example, here's what I've run that all works fine:
marcogorelli@DESKTOP-U8OKFP3:~$ cookiecutter https://github.com/MarcoGorelli/cookiecutter-polars-plugins
You've downloaded /home/marcogorelli/.cookiecutters/cookiecutter-polars-plugins before. Is it okay to delete and re-download it? [y/n] (y): y
[1/3] plugin_name (My Plugin):
[2/3] project_slug (my_plugin):
[3/3] author ():
marcogorelli@DESKTOP-U8OKFP3:~$ cd my_plugin/
marcogorelli@DESKTOP-U8OKFP3:~/my_plugin$ uv venv -p 3.12
Using Python 3.12.4
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
marcogorelli@DESKTOP-U8OKFP3:~/my_plugin$ . .venv/bin/activate
(my_plugin) marcogorelli@DESKTOP-U8OKFP3:~/my_plugin$ uv pip install -U polars maturin pip
Resolved 3 packages in 42ms
Prepared 3 packages in 0.40ms
Installed 3 packages in 8ms
+ maturin==1.7.1
+ pip==24.2
+ polars==1.5.0
(my_plugin) marcogorelli@DESKTOP-U8OKFP3:~/my_plugin$ maturin develop(my_plugin) marcogorelli@DESKTOP-U8OKFP3:~/my_plugin$ python run.py
shape: (5, 2)
┌─────────┬───────────┐
│ english ┆ pig_latin │
│ --- ┆ --- │
│ str ┆ str │
╞═════════╪═══════════╡
│ this ┆ histay │
│ is ┆ siay │
│ not ┆ otnay │
│ pig ┆ igpay │
│ latin ┆ atinlay │
└─────────┴───────────┘PS D:\Project\playground\quant\polars_plugin> cookiecutter https://github.com/MarcoGorelli/cookiecutter-polars-plugins [1/3] plugin_name (My Plugin): minimal [2/3] project_slug (minimal): [3/3] author (): la
PS D:\Project\playground\quant\polars_plugin\minimal> python -m venv env
PS D:\Project\playground\quant\polars_plugin\minimal> env/Scripts/activate
(env) PS D:\Project\playground\quant\polars_plugin\minimal> python -m pip install --upgrade pip
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: pip in d:\project\playground\quant\polars_plugin\minimal\env\lib\site-packages (22.0.4)
Collecting pip
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/d4/55/90db48d85f7689ec6f81c0db0622d704306c5284850383c090e6c7195a5c/pip-24.2-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
Successfully uninstalled pip-22.0.4
Successfully installed pip-24.2
(env) PS D:\Project\playground\quant\polars_plugin\minimal> pip install -U polars maturin
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting polars
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/df/75/7c9774d96ada8fae71124b536c4b00d99d4ae81856c53e6ea6f98f28a368/polars-1.5.0-cp38-abi3-win_amd64.whl (31.5 MB)
---------------------------------------- 31.5/31.5 MB 5.4 MB/s eta 0:00:00
Collecting maturin
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a1/5b/512efa939f747f1a1277f981ca1de332f01bb187d193cb8d67f816c38735/maturin-1.7.1-py3-none-win_amd64.whl (7.3 MB)
---------------------------------------- 7.3/7.3 MB 5.7 MB/s eta 0:00:00
Collecting tomli>=1.1.0 (from maturin)
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl (12 kB)
Installing collected packages: tomli, polars, maturin
Successfully installed maturin-1.7.1 polars-1.5.0 tomli-2.0.1
(env) PS D:\Project\playground\quant\polars_plugin\minimal> pip install -U polars maturin
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting polars
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/df/75/7c9774d96ada8fae71124b536c4b00d99d4ae81856c53e6ea6f98f28a368/polars-1.5.0-cp38-abi3-win_amd64.whl (31.5 MB)
Collecting maturin
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a1/5b/512efa939f747f1a1277f981ca1de332f01bb187d193cb8d67f816c38735/maturin-1.7.1-py3-none-win_amd64.whl (7.3 MB)
Collecting tomli>=1.1.0 (from maturin)
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl (12 kB)
Installing collected packages: tomli, polars, maturin
Successfully installed maturin-1.7.1 polars-1.5.0 tomli-2.0.1
(env) PS D:\Project\playground\quant\polars_plugin\minimal> maturin develop
Updating rsproxy-sparse index
Locking 143 packages to latest compatible versions
Adding atoi_simd v0.15.6 (latest: v0.16.0)
Adding iter-read v0.3.1 (latest: v1.1.0)
Adding memmap2 v0.7.1 (latest: v0.9.4)
Adding planus v0.3.1 (latest: v0.4.0)
Adding syn v1.0.109 (latest: v2.0.76)
Adding wasi v0.11.0+wasi-snapshot-preview1 (latest: v0.13.2+wasi-0.2.1)
Adding windows-sys v0.36.1 (latest: v0.59.0)
Adding windows-sys v0.52.0 (latest: v0.59.0)
Adding windows_aarch64_msvc v0.36.1 (latest: v0.52.6)
Adding windows_i686_gnu v0.36.1 (latest: v0.52.6)
Adding windows_i686_msvc v0.36.1 (latest: v0.52.6)
Adding windows_x86_64_gnu v0.36.1 (latest: v0.52.6)
Adding windows_x86_64_msvc v0.36.1 (latest: v0.52.6)
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
🐍 Using CPython 3.9 at D:\Project\playground\quant\polars_plugin\minimal\env\Scripts\python.exe to generate to link bindings (With abi3, an interpreter is only required on windows)
Compiling proc-macro2 v1.0.86
Compiling unicode-ident v1.0.12
Compiling version_check v0.9.5
Compiling autocfg v1.3.0
Compiling cfg-if v1.0.0
Compiling once_cell v1.19.0
Compiling libm v0.2.8
Compiling crossbeam-utils v0.8.20
Compiling getrandom v0.2.15
Compiling shlex v1.3.0
Compiling rayon-core v1.12.1
Compiling cc v1.1.14
Compiling libc v0.2.158
Compiling ahash v0.8.11
Compiling windows_x86_64_msvc v0.52.6
Compiling either v1.13.0
Compiling windows_x86_64_msvc v0.36.1
Compiling thiserror v1.0.63
Compiling allocator-api2 v0.2.18
Compiling windows-sys v0.36.1
Compiling windows-targets v0.52.6
Compiling quote v1.0.37
Compiling crossbeam-epoch v0.9.18
Compiling syn v2.0.76
Compiling crossbeam-deque v0.8.5
Compiling smartstring v1.0.1
Compiling syn v1.0.109
Compiling simdutf8 v0.1.4
Compiling target-features v0.1.6
Compiling equivalent v1.0.1
Compiling lock_api v0.4.12
Compiling polars-utils v0.42.0
Compiling rayon v1.10.0
Compiling parking_lot_core v0.9.10
Compiling bitflags v2.6.0
Compiling static_assertions v1.1.0
Compiling raw-cpuid v11.1.0
Compiling psm v0.1.21
Compiling stacker v0.1.16
Compiling scopeguard v1.2.0
Compiling memmap2 v0.7.1
Compiling bytes v1.7.1
Compiling smallvec v1.13.2
Compiling byteorder v1.5.0
Compiling parking_lot v0.12.3
Compiling num-traits v0.2.19
Compiling dyn-clone v1.0.17
Compiling itoa v1.0.11
Compiling ethnum v1.5.0
Compiling atoi_simd v0.15.6
Compiling streaming-iterator v0.1.9
Compiling fast-float v0.2.0
Compiling ryu v1.0.18
Compiling strength_reduce v0.2.4
Compiling rand_core v0.6.4
Compiling serde v1.0.209
Compiling polars-compute v0.42.0
Compiling target-lexicon v0.12.16
Compiling array-init-cursor v0.2.0
Compiling polars-core v0.42.0
Compiling planus v0.3.1
Compiling xxhash-rust v0.8.12
Compiling polars-arrow v0.42.0
Compiling chrono v0.4.38
Compiling memchr v2.7.4
Compiling pyo3-build-config v0.22.2
Compiling regex-syntax v0.8.4
Compiling aho-corasick v1.1.3
Compiling zerocopy-derive v0.7.35
Compiling thiserror-impl v1.0.63
Compiling bytemuck_derive v1.7.1
Compiling serde_derive v1.0.209
Compiling heck v0.5.0
Compiling regex-automata v0.4.7
Compiling zerocopy v0.7.35
Compiling bytemuck v1.17.0
Compiling polars-error v0.42.0
Compiling multiversion-macros v0.7.4
Compiling ppv-lite86 v0.2.20
Compiling windows-sys v0.52.0
Compiling hashbrown v0.14.5
Compiling rand_chacha v0.3.1
Compiling multiversion v0.7.4
Compiling indexmap v2.4.0
Compiling rand v0.8.5
Compiling regex v1.10.6
Compiling polars-ops v0.42.0
Compiling rustversion v1.0.17
Compiling home v0.5.9
Compiling percent-encoding v2.3.1
Compiling glob v0.3.1
Compiling rand_distr v0.4.3
Compiling pyo3-ffi v0.22.2
Compiling pyo3-macros-backend v0.22.2
Compiling recursive-proc-macro-impl v0.1.1
Compiling polars-plan v0.42.0
Compiling argminmax v0.6.2
Compiling strum_macros v0.26.4
Compiling memoffset v0.9.1
Compiling recursive v0.1.1
Compiling fallible-streaming-iterator v0.1.9
Compiling streaming-decompression v0.1.2
Compiling pyo3 v0.22.2
Compiling num-integer v0.1.46
Compiling polars v0.42.0
Compiling base64 v0.22.1
Compiling parquet-format-safe v0.2.4
Compiling polars-arrow-format v0.1.0
Compiling num-bigint v0.4.6
Compiling unindent v0.2.3
Compiling indoc v2.0.5
Compiling iter-read v0.3.1
Compiling serde-pickle v1.1.1
Compiling pyo3-macros v0.22.2
Compiling polars-row v0.42.0
Compiling polars-parquet v0.42.0
Compiling polars-io v0.42.0
Compiling polars-ffi v0.42.0
Compiling pyo3-polars-derive v0.10.0
Compiling pyo3-polars v0.16.1
error[E0308]: mismatched types
--> C:\Users\roar.cargo\registry\src\rsproxy.cn-0dccff568467c15b\pyo3-polars-0.16.1\src\alloc.rs:83:78 |
83 | unsafe { libc::write(2, msg.as_ptr() as *const libc::c_void, msg.len()) }; | ----------- ^^^^^^^^^ expected u32, found usize |
|---|---|---|---|
| arguments to this function are incorrect | |||
| note: function defined here --> C:\Users\roar.cargo\registry\src\rsproxy.cn-0dccff568467c15b\libc-0.2.158\src\windows\mod.rs:507:12 | 507 | pub fn write(fd: ::c_int, buf: *const ::c_void, count: ::c_uint) -> ::c_int; | ^^^^^
help: you can convert a usize to a u32 and panic if the converted value doesn't fit |
|---|---|---|---|
| 83 | unsafe { libc::write(2, msg.as_ptr() as *const libc::c_void, msg.len().try_into().unwrap()) }; | ||
| ++++++++++++++++++++ |
For more information about this error, try rustc --explain E0308.
error: could not compile pyo3-polars (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit code: 101": "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "D:\\Project\\playground\\quant\\polars_plugin\\minimal\\Cargo.toml" "--lib"
| (env) PS D:\Project\playground\quant\polars_plugin\minimal> maturin develop 🍹 Building a mixed python/rust project 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8 🐍 Using CPython 3.11 at D:\Project\playground\quant\polars_plugin\minimal\env\Scripts\python.exe to generate to link bindings (With abi3, an interpreter is only required on windows) Compiling pyo3-build-config v0.22.2 Compiling pyo3-macros-backend v0.22.2 Compiling pyo3-ffi v0.22.2 Compiling pyo3 v0.22.2 Compiling pyo3-macros v0.22.2 Compiling pyo3-polars v0.16.1 error[E0308]: mismatched types --> C:\Users\langslike.cargo\registry\src\rsproxy.cn-0dccff568467c15b\pyo3-polars-0.16.1\src\alloc.rs:83:78 | 83 | unsafe { libc::write(2, msg.as_ptr() as *const libc::c_void, msg.len()) }; | ----------- ^^^^^^^^^ expected u32, found usize |
|---|---|---|---|
| arguments to this function are incorrect | |||
| note: function defined here --> C:\Users\langslike.cargo\registry\src\rsproxy.cn-0dccff568467c15b\libc-0.2.158\src\windows\mod.rs:507:12 | 507 | pub fn write(fd: ::c_int, buf: *const ::c_void, count: ::c_uint) -> ::c_int; | ^^^^^
help: you can convert a usize to a u32 and panic if the converted value doesn't fit |
|---|---|---|---|
| 83 | unsafe { libc::write(2, msg.as_ptr() as *const libc::c_void, msg.len().try_into().unwrap()) }; | ||
| ++++++++++++++++++++ |
For more information about this error, try rustc --explain E0308.
error: could not compile pyo3-polars (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit code: 101": "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "D:\\Project\\playground\\quant\\polars_plugin\\minimal\\Cargo.toml" "--lib"
I used this version of minimal plugin, and the compile and run is sucessful: https://github.com/MarcoGorelli/cookiecutter-polars-plugins/tree/3ad0a5dbc342c3f5f876e2e65a80d104c3153507
sauerburger
commented
1 week ago I also ran into that issue.
MarcoGorelli
commented
1 week ago thanks @sauerburger - also on windows?
sauerburger
commented
1 week ago It's on Ubuntu 20.04 with
Actually, I was too quick to say I have the same problem. The full error message is different, see below.
I was able to compile with the older version of the template (3ad0a5d. The older version uses polars-arrow v0.41.3 which seems to solve the problem for me.
Compiling polars-arrow v0.42.0
error[E0658]: inline-const is experimental
--> /home/frank/.cargo/registry/src/index.crates.io-6f17d22bba15001f/polars-arrow-0.42.0/src/array/static_array.rs:70:9
|
70 | no_call_const!()
| ^^^^^^^^^^^^^^^^
|
= note: see issue #76001 <https://github.com/rust-lang/rust/issues/76001> for more information
= note: this error originates in the macro `no_call_const` (in Nightly builds, run with -Z macro-backtrace for more info)i think you need the nightly rust for that - you might be missing https://github.com/MarcoGorelli/polars-plugins-tutorial/blob/main/rust-toolchain.toml ?
[buffer::immutable::Buffer<u8>]: Defaultis not satisfied --> C:\Users\roar.cargo\registry\src\rsproxy.cn-8f6827c7555bfaf8\polars-arrow-0.42.0\src\compute\cast\binary_to.rs:205:58Defaultis not implemented for[buffer::immutable::Buffer<u8>], which is required bystd::sync::Arc<_, _>: DefaultCompiling fallible-streaming-iterator v0.1.9 Compiling pyo3 v0.22.2 Compiling streaming-decompression v0.1.2 Compiling num-integer v0.1.46 Compiling base64 v0.22.1 Compiling parquet-format-safe v0.2.4 Compiling num-bigint v0.4.6 For more information about this error, try
rustc --explain E0277. error: could not compilepolars-arrow(lib) due to 1 previous error warning: build failed, waiting for other jobs to finish... error: could not compilepolars-arrow(lib) due to 1 previous error 💥 maturin failed Caused by: Failed to build a native library through cargo Caused by: Cargo build finished with "exit code: 101":"cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "D:\\Project\\playground\\quant\\polars_plugin\\ops\\Cargo.toml" "--lib"my env: windows10, vs2022, py39, polars 0.20.31, maturin 1.7.0, rust 1.79.0. I can compile my own rust program that uses polars as dependency sucessfully.