search

pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
30.65k stars 1.99k forks source link

Fail to compile polars-core #3334

Closed mathiaHT closed 2 years ago

mathiaHT commented 2 years ago

What language are you using?

python

Have you tried latest version of polars?

If the problem was resolved, please update polars. :)

What version of polars are you using?

0.13.30

What operating system are you using polars on?

amazon linux 2

What language version are you using

python

Describe your bug.

Fail to compile polars-core in amazon linux with rust 1.60.0

What are the steps to reproduce the behavior?

FROM public.ecr.aws/lambda/python:3.9

WORKDIR /app

RUN yum update -y && \
    yum install -y \
        cmake3 \
        curl \
        Cython \
        findutils \
        gcc \
        gcc-c++ \
        git \
        gzip \
        jq \
        libxml2 \
        libxslt \
        make \
        man \
        man-pages \
        openssl-devel \
        rsync \
        tar \
        wget \
        which \
        zip && \
    ln -s /usr/bin/cmake3 /usr/bin/cmake

ENV RUSTUP_HOME=/rustup \
    CARGO_HOME=/cargo \
    PATH=/cargo/bin:$PATH
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
        | sh -s -- \
            -y \
            --profile minimal \
            --default-toolchain 1.60.0

RUN pip3 install polars

What is the actual behavior?

      error: could not compile `arrow2`

      Caused by:
        process didn't exit successfully: `rustc --crate-name arrow2 --edition=2021 /cargo/git/checkouts/arrow2-8a2ad61d97265680/d035964/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 --cfg 'feature="arrow-format"' --cfg 'feature="avro-schema"' --cfg 'feature="base64"' --cfg 'feature="compute_aggregate"' --cfg 'feature="compute_arithmetics"' --cfg 'feature="compute_boolean"' --cfg 'feature="compute_boolean_kleene"' --cfg 'feature="compute_cast"' --cfg 'feature="compute_comparison"' --cfg 'feature="compute_concatenate"' --cfg 'feature="compute_filter"' --cfg 'feature="compute_if_then_else"' --cfg 'feature="compute_substring"' --cfg 'feature="compute_take"' --cfg 'feature="crc"' --cfg 'feature="csv-core"' --cfg 'feature="fallible-streaming-iterator"' --cfg 'feature="futures"' --cfg 'feature="indexmap"' --cfg 'feature="io_avro"' --cfg 'feature="io_avro_compression"' --cfg 'feature="io_csv_write"' --cfg 'feature="io_ipc"' --cfg 'feature="io_ipc_compression"' --cfg 'feature="io_json"' --cfg 'feature="io_parquet"' --cfg 'feature="io_parquet_compression"' --cfg 'feature="lexical-core"' --cfg 'feature="libflate"' --cfg 'feature="lz4"' --cfg 'feature="multiversion"' --cfg 'feature="parquet2"' --cfg 'feature="serde"' --cfg 'feature="serde_json"' --cfg 'feature="simd"' --cfg 'feature="snap"' --cfg 'feature="streaming-iterator"' --cfg 'feature="strength_reduce"' --cfg 'feature="zstd"' -C metadata=20c37f9973a5b6e9 -C extra-filename=-20c37f9973a5b6e9 --out-dir /tmp/pip-req-build-qqa4d5y7/target/release/deps -C strip=symbols -L dependency=/tmp/pip-req-build-qqa4d5y7/target/release/deps --extern arrow_format=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libarrow_format-815a5f2e14037358.rmeta --extern avro_schema=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libavro_schema-d36afe0262fe0f11.rmeta --extern base64=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libbase64-ab9a92a7bd4e4124.rmeta --extern bytemuck=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libbytemuck-63aa3784f8ec95fe.rmeta --extern chrono=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libchrono-c482ce4a6c2b04f4.rmeta --extern crc=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libcrc-aa4471382dbd7596.rmeta --extern csv_core=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libcsv_core-e9b1b215f5ba0b67.rmeta --extern either=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libeither-688e22c09e09f02b.rmeta --extern fallible_streaming_iterator=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libfallible_streaming_iterator-65561399ed91de2e.rmeta --extern futures=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libfutures-ac531d7457c4ecc2.rmeta --extern hash_hasher=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libhash_hasher-06ec54ec1611c87f.rmeta --extern indexmap=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libindexmap-cefa4272e40ed50d.rmeta --extern lexical_core=/tmp/pip-req-build-qqa4d5y7/target/release/deps/liblexical_core-c649ae3dc9b5832c.rmeta --extern libflate=/tmp/pip-req-build-qqa4d5y7/target/release/deps/liblibflate-c5e36efde048639e.rmeta --extern lz4=/tmp/pip-req-build-qqa4d5y7/target/release/deps/liblz4-abfd7516d3ce1828.rmeta --extern multiversion=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libmultiversion-4fbd8f8e9b1ce6ad.rmeta --extern num_traits=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libnum_traits-08803870504b52f6.rmeta --extern parquet2=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libparquet2-60e6beeab6537ec2.rmeta --extern serde=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libserde-4e38b4dd6b106d40.rmeta --extern serde_json=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libserde_json-a870b149a35cadee.rmeta --extern simdutf8=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libsimdutf8-fc5280b53ccddae3.rmeta --extern snap=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libsnap-13bd712764f3bfe9.rmeta --extern streaming_iterator=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libstreaming_iterator-016b9c42ba4ba749.rmeta --extern strength_reduce=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libstrength_reduce-311bd15322ec0835.rmeta --extern zstd=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libzstd-21500c9890567c5e.rmeta --cap-lints allow -Dwarnings -L native=/tmp/pip-req-build-qqa4d5y7/target/release/build/lz4-sys-5bd3fb9c0170aa80/out -L native=/tmp/pip-req-build-qqa4d5y7/target/release/build/libz-sys-72ed527f5ceb0949/out/lib -L native=/tmp/pip-req-build-qqa4d5y7/target/release/build/zstd-sys-946542f627153289/out` (signal: 9, SIGKILL: kill)
      warning: build failed, waiting for other jobs to finish...
      error: could not compile `polars-core`; 3 warnings emitted

      Caused by:
        process didn't exit successfully: `rustc --crate-name polars_core --edition=2021 local_dependencies/polars-core/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 --cfg 'feature="abs"' --cfg 'feature="asof_join"' --cfg 'feature="base64"' --cfg 'feature="chrono"' --cfg 'feature="chunked_ids"' --cfg 'feature="comfy-table"' --cfg 'feature="concat_str"' --cfg 'feature="cross_join"' --cfg 'feature="cum_agg"' --cfg 'feature="dataframe_arithmetic"' --cfg 'feature="default"' --cfg 'feature="diagonal_concat"' --cfg 'feature="diff"' --cfg 'feature="docs"' --cfg 'feature="dot_product"' --cfg 'feature="dtype-categorical"' --cfg 'feature="dtype-date"' --cfg 'feature="dtype-datetime"' --cfg 'feature="dtype-duration"' --cfg 'feature="dtype-i16"' --cfg 'feature="dtype-i8"' --cfg 'feature="dtype-struct"' --cfg 'feature="dtype-time"' --cfg 'feature="dtype-u16"' --cfg 'feature="dtype-u8"' --cfg 'feature="dynamic_groupby"' --cfg 'feature="ewma"' --cfg 'feature="extract_jsonpath"' --cfg 'feature="fmt"' --cfg 'feature="hex"' --cfg 'feature="horizontal_concat"' --cfg 'feature="interpolate"' --cfg 'feature="is_first"' --cfg 'feature="is_in"' --cfg 'feature="jsonpath_lib"' --cfg 'feature="lazy"' --cfg 'feature="list"' --cfg 'feature="log"' --cfg 'feature="mode"' --cfg 'feature="moment"' --cfg 'feature="ndarray"' --cfg 'feature="object"' --cfg 'feature="parquet"' --cfg 'feature="partition_by"' --cfg 'feature="pct_change"' --cfg 'feature="performant"' --cfg 'feature="private"' --cfg 'feature="product"' --cfg 'feature="rand"' --cfg 'feature="rand_distr"' --cfg 'feature="random"' --cfg 'feature="rank"' --cfg 'feature="regex"' --cfg 'feature="reinterpret"' --cfg 'feature="repeat_by"' --cfg 'feature="rolling_window"' --cfg 'feature="round_series"' --cfg 'feature="row_hash"' --cfg 'feature="rows"' --cfg 'feature="semi_anti_join"' --cfg 'feature="serde"' --cfg 'feature="serde-lazy"' --cfg 'feature="serde_json"' --cfg 'feature="simd"' --cfg 'feature="sort_multiple"' --cfg 'feature="string_encoding"' --cfg 'feature="strings"' --cfg 'feature="temporal"' --cfg 'feature="unique_counts"' --cfg 'feature="zip_with"' -C metadata=b9d962a477c80633 -C extra-filename=-b9d962a477c80633 --out-dir /tmp/pip-req-build-qqa4d5y7/target/release/deps -C strip=symbols -L dependency=/tmp/pip-req-build-qqa4d5y7/target/release/deps --extern ahash=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libahash-2a9d9d7255c8da3e.rmeta --extern anyhow=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libanyhow-d737f895e9b12609.rmeta --extern arrow=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libarrow2-20c37f9973a5b6e9.rmeta --extern base64=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libbase64-ab9a92a7bd4e4124.rmeta --extern chrono=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libchrono-c482ce4a6c2b04f4.rmeta --extern comfy_table=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libcomfy_table-b95917098281c91c.rmeta --extern hashbrown=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libhashbrown-045daac952194173.rmeta --extern hex=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libhex-9cfbfef41c9d0dc5.rmeta --extern indexmap=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libindexmap-cefa4272e40ed50d.rmeta --extern jsonpath_lib=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libjsonpath_lib-635b8b8c56a4b9c8.rlib --extern ndarray=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libndarray-e35518ac6c6ee00d.rmeta --extern num=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libnum-8c719317f2d112ea.rmeta --extern once_cell=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libonce_cell-ab9649924ff484bb.rmeta --extern polars_arrow=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libpolars_arrow-e79aea7bff8d2c71.rmeta --extern polars_utils=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libpolars_utils-ed39acbf33c5c8e7.rmeta --extern rand=/tmp/pip-req-build-qqa4d5y7/target/release/deps/librand-3bd71f479a1acce6.rmeta --extern rand_distr=/tmp/pip-req-build-qqa4d5y7/target/release/deps/librand_distr-1fd90ac004b6dca2.rmeta --extern rayon=/tmp/pip-req-build-qqa4d5y7/target/release/deps/librayon-8cd4668f760fd3ac.rmeta --extern regex=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libregex-ffad26b9b8579ff0.rmeta --extern serde=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libserde-4e38b4dd6b106d40.rmeta --extern serde_json=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libserde_json-a870b149a35cadee.rmeta --extern thiserror=/tmp/pip-req-build-qqa4d5y7/target/release/deps/libthiserror-b8f8231b61af5740.rmeta -Dwarnings -L native=/tmp/pip-req-build-qqa4d5y7/target/release/build/lz4-sys-5bd3fb9c0170aa80/out -L native=/tmp/pip-req-build-qqa4d5y7/target/release/build/libz-sys-72ed527f5ceb0949/out/lib -L native=/tmp/pip-req-build-qqa4d5y7/target/release/build/zstd-sys-946542f627153289/out` (signal: 9, SIGKILL: kill)
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `cargo rustc --manifest-path Cargo.toml --message-format json --release --lib --`
      🍹 Building a mixed python/rust project
      🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
      🐍 Not using a specific python interpreter
      warning: unnecessary lifetime parameter `'de`
         --> local_dependencies/polars-core/src/datatypes.rs:806:62
          |
      806 | #[cfg_attr(feature = "serde-lazy", serde(bound(deserialize = "'de: 'static")))]
          |                                                              ^^^^^^^^^^^^^^
          |
          = help: you can use the `'static` lifetime directly, in place of `'de`

      warning: unnecessary lifetime parameter `'de`
        --> local_dependencies/polars-core/src/frame/asof_join/mod.rs:14:57
         |
      14 | #[cfg_attr(feature = "serde", serde(bound(deserialize = "'de: 'static")))]
         |                                                         ^^^^^^^^^^^^^^
         |
         = help: you can use the `'static` lifetime directly, in place of `'de`

      warning: unnecessary lifetime parameter `'de`
        --> local_dependencies/polars-core/src/schema.rs:12:31
         |
      12 |     serde(bound(deserialize = "'de: 'static"))
         |                               ^^^^^^^^^^^^^^
         |
         = help: you can use the `'static` lifetime directly, in place of `'de`

      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/app/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for polars
Failed to build polars
ritchie46 commented 2 years ago

Sorry, I don't understand? Why does it compile from source?

mathiaHT commented 2 years ago

This is a good question, I did update the dockerfile above, it contains everything

ritchie46 commented 2 years ago

Could you try rust nightly?

ghuls commented 2 years ago

Probably it is an older pip version that doesn't try to install binaries by default. You can upgrade pip or use: pip3 install --only-binary :all: polars.

FROM public.ecr.aws/lambda/python:3.9

WORKDIR /app

RUN yum update -y && \
    yum install -y \
        cmake3 \
        curl \
        Cython \
        findutils \
        gcc \
        gcc-c++ \
        git \
        gzip \
        jq \
        libxml2 \
        libxslt \
        make \
        man \
        man-pages \
        openssl-devel \
        rsync \
        tar \
        wget \
        which \
        zip && \
    ln -s /usr/bin/cmake3 /usr/bin/cmake

RUN pip3 install --only-binary :all: polars
mathiaHT commented 2 years ago

Thanks, how would you do that with poetry? I don't find a way of specifying such option

ghuls commented 2 years ago

You probably need to upgrade pip in that case.

I understood that poetry uses pip which does not use the wheel version of numpy when pip <= 20.2.x.

https://stackoverflow.com/questions/62513005/how-does-poetry-work-regarding-binary-dependencies-esp-numpy