Cargo easyblock fails to generate `crates` for polars

[boegel]

$ git clone https://github.com/pola-rs/polars
$ cd polars
$ python -m easybuild.easyblocks.generic.cargo .
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data/gent/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_easyblocks-4.9.1.dev0-py3.6.egg/easybuild/easyblocks/generic/cargo.py", line 270, in <module>
    app_in_cratesio, crates, other = generate_crate_list(sys.argv[1])
  File "/data/gent/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_easyblocks-4.9.1.dev0-py3.6.egg/easybuild/easyblocks/generic/cargo.py", line 245, in generate_crate_list
    app_name = cargo_toml['package']['name']
KeyError: 'package'

Any ideas on this @Micket? Is generate_crate_list hitting an entry in Cargo.lock it doesn't expect?

[branfosj]

It is failing on the Cargo.toml file. Specifically failing on app_name = cargo_toml['package']['name'].

If I change to

    try:
        app_name = cargo_toml['package']['name']
    except:
        app_name = 'bananas'

Then I get

$ python -m easybuild.easyblocks.generic.cargo .
[('polars', '0.36.2'), ('polars-arrow', '0.36.2'), ('polars-compute', '0.36.2'), ('polars-core', '0.36.2'), ('polars-doc-examples', '0.36.2'), ('polars-error', '0.36.2'), ('polars-ffi', '0.36.2'), ('polars-
io', '0.36.2'), ('polars-json', '0.36.2'), ('polars-lazy', '0.36.2'), ('polars-ops', '0.36.2'), ('polars-parquet', '0.36.2'), ('polars-pipe', '0.36.2'), ('polars-plan', '0.36.2'), ('polars-row', '0.36.2'),
('polars-sql', '0.36.2'), ('polars-time', '0.36.2'), ('polars-utils', '0.36.2'), ('py-polars', '0.20.4')]
crates = [
    ('addr2line', '0.21.0'),
    ('adler', '1.0.2'),
    ('adler32', '1.2.0'),
    ('ahash', '0.8.7'),
    ('aho-corasick', '1.1.2'),
    ('alloc-no-stdlib', '2.0.4'),
    ('alloc-stdlib', '0.2.2'),
    ('allocator-api2', '0.2.16'),
    ('android-tzdata', '0.1.1'),
    ('android_system_properties', '0.1.5'),
    ('anes', '0.1.6'),
    ('anstyle', '1.0.4'),
    ('anyhow', '1.0.79'),
    ('apache-avro', '0.16.0'),
    ('argminmax', '0.6.1'),
    ('array-init-cursor', '0.2.0'),
    ('arrow-array', '49.0.0'),
    ('arrow-buffer', '49.0.0'),
    ('arrow-data', '49.0.0'),
    ('arrow-schema', '49.0.0'),
    ('arrow2', '0.17.4'),
    ('async-stream', '0.3.5'),
    ('async-stream-impl', '0.3.5'),
    ('async-trait', '0.1.77'),
    ('atoi', '2.0.0'),
    ('atoi_simd', '0.15.6'),
    ('autocfg', '1.1.0'),
    ('avro-schema', '0.3.0'),
    ('aws-config', '1.1.1'),
    ('aws-credential-types', '1.1.1'),
    ('aws-http', '0.60.1'),
    ('aws-runtime', '1.1.1'),
    ('aws-sdk-s3', '1.11.0'),
    ('aws-sdk-sso', '1.9.0'),
    ('aws-sdk-ssooidc', '1.9.0'),
    ('aws-sdk-sts', '1.9.0'),
    ('aws-sigv4', '1.1.1'),
    ('aws-smithy-async', '1.1.1'),
    ('aws-smithy-checksums', '0.60.1'),
    ('aws-smithy-eventstream', '0.60.1'),
    ('aws-smithy-http', '0.60.1'),
    ('aws-smithy-json', '0.60.1'),
    ('aws-smithy-query', '0.60.1'),
    ('aws-smithy-runtime', '1.1.1'),
    ('aws-smithy-runtime-api', '1.1.1'),
    ('aws-smithy-types', '1.1.1'),
    ('aws-smithy-xml', '0.60.1'),
    ('aws-types', '1.1.1'),
    ('backtrace', '0.3.69'),
    ('base16ct', '0.1.1'),
    ('base64', '0.21.5'),
    ('base64-simd', '0.8.0'),
    ('base64ct', '1.6.0'),
    ('bincode', '1.3.3'),
    ('bitflags', '1.3.2'),
    ('bitflags', '2.4.1'),
    ('block-buffer', '0.10.4'),
    ('brotli', '3.4.0'),
    ('brotli-decompressor', '2.5.1'),
    ('built', '0.7.1'),
    ('bumpalo', '3.14.0'),
    ('bytemuck', '1.14.0'),
    ('bytemuck_derive', '1.5.0'),
    ('bytes', '1.5.0'),
    ('bytes-utils', '0.1.4'),
    ('cargo-lock', '9.0.0'),
    ('casey', '0.4.0'),
    ('cast', '0.3.0'),
    ('cc', '1.0.83'),
    ('cfg-if', '1.0.0'),
    ('chrono', '0.4.31'),
    ('chrono-tz', '0.8.5'),
    ('chrono-tz-build', '0.2.1'),
    ('ciborium', '0.2.1'),
    ('ciborium-io', '0.2.1'),
    ('ciborium-ll', '0.2.1'),
    ('clap', '4.4.12'),
    ('clap_builder', '4.4.12'),
    ('clap_lex', '0.6.0'),
    ('cmake', '0.1.50'),
    ('comfy-table', '7.1.0'),
    ('const-oid', '0.9.6'),
    ('const-random', '0.1.17'),
    ('const-random-macro', '0.1.16'),
    ('core-foundation', '0.9.4'),
    ('core-foundation-sys', '0.8.6'),
    ('core2', '0.4.0'),
    ('cpufeatures', '0.2.11'),
    ('crc', '2.1.0'),
    ('crc-catalog', '1.1.1'),
    ('crc32c', '0.6.4'),
    ('crc32fast', '1.3.2'),
    ('criterion', '0.5.1'),
    ('criterion-plot', '0.5.0'),
    ('crossbeam-channel', '0.5.10'),
    ('crossbeam-deque', '0.8.4'),
    ('crossbeam-epoch', '0.9.17'),
    ('crossbeam-queue', '0.3.10'),
    ('crossbeam-utils', '0.8.18'),
    ('crossterm', '0.27.0'),
    ('crossterm_winapi', '0.9.1'),
    ('crunchy', '0.2.2'),
    ('crypto-bigint', '0.4.9'),
    ('crypto-bigint', '0.5.5'),
    ('crypto-common', '0.1.6'),
    ('dary_heap', '0.3.6'),
    ('der', '0.6.1'),
    ('deranged', '0.3.11'),
    ('digest', '0.10.7'),
    ('doc-comment', '0.3.3'),
    ('dyn-clone', '1.0.16'),
    ('ecdsa', '0.14.8'),
    ('either', '1.9.0'),
    ('elliptic-curve', '0.12.3'),
    ('encoding_rs', '0.8.33'),
    ('enum_dispatch', '0.3.12'),
    ('env_logger', '0.8.4'),
    ('equivalent', '1.0.1'),
    ('errno', '0.3.8'),
    ('ethnum', '1.5.0'),
    ('fallible-streaming-iterator', '0.1.9'),
    ('fast-float', '0.2.0'),
    ('fastrand', '2.0.1'),
    ('ff', '0.12.1'),
    ('flate2', '1.0.28'),
    ('float-cmp', '0.9.0'),
    ('fnv', '1.0.7'),
    ('foreign_vec', '0.1.0'),
    ('form_urlencoded', '1.2.1'),
    ('futures', '0.3.30'),
    ('futures-channel', '0.3.30'),
    ('futures-core', '0.3.30'),
    ('futures-executor', '0.3.30'),
    ('futures-io', '0.3.30'),
    ('futures-macro', '0.3.30'),
    ('futures-sink', '0.3.30'),
    ('futures-task', '0.3.30'),
    ('futures-util', '0.3.30'),
    ('generic-array', '0.14.7'),
    ('getrandom', '0.2.11'),
    ('gimli', '0.28.1'),
    ('git2', '0.18.1'),
    ('glob', '0.3.1'),
    ('group', '0.12.1'),
    ('h2', '0.3.22'),
    ('half', '1.8.2'),
    ('half', '2.3.1'),
    ('halfbrown', '0.2.4'),
    ('hash_hasher', '2.0.3'),
    ('hashbrown', '0.13.2'),
    ('hashbrown', '0.14.3'),
    ('heck', '0.4.1'),
    ('hermit-abi', '0.3.3'),
    ('hex', '0.4.3'),
    ('hmac', '0.12.1'),
    ('home', '0.5.9'),
    ('http', '0.2.11'),
    ('http-body', '0.4.6'),
    ('httparse', '1.8.0'),
    ('httpdate', '1.0.3'),
    ('humantime', '2.1.0'),
    ('hyper', '0.14.28'),
    ('hyper-rustls', '0.24.2'),
    ('iana-time-zone', '0.1.59'),
    ('iana-time-zone-haiku', '0.1.2'),
    ('idna', '0.5.0'),
    ('indexmap', '2.1.0'),
    ('indoc', '2.0.4'),
    ('inventory', '0.3.14'),
    ('ipnet', '2.9.0'),
    ('is-terminal', '0.4.10'),
    ('itertools', '0.10.5'),
    ('itertools', '0.11.0'),
    ('itoa', '1.0.10'),
    ('itoap', '1.0.1'),
    ('jemalloc-sys', '0.5.4+5.3.0-patched'),
    ('jemallocator', '0.5.4'),
    ('jobserver', '0.1.27'),
    ('js-sys', '0.3.66'),
    ('jsonpath_lib', '0.3.0', 'git+https://github.com/ritchie46/jsonpath?branch=improve_compiled'),
    ('lazy_static', '1.4.0'),
    ('lexical-core', '0.8.5'),
    ('lexical-parse-float', '0.8.5'),
    ('lexical-parse-integer', '0.8.6'),
    ('lexical-util', '0.8.5'),
    ('lexical-write-float', '0.8.5'),
    ('lexical-write-integer', '0.8.5'),
    ('libc', '0.2.151'),
    ('libflate', '1.4.0'),
    ('libflate', '2.0.0'),
    ('libflate_lz77', '1.2.0'),
    ('libflate_lz77', '2.0.0'),
    ('libgit2-sys', '0.16.1+1.7.1'),
    ('libloading', '0.8.1'),
    ('libm', '0.2.8'),
    ('libmimalloc-sys', '0.1.35'),
    ('libz-ng-sys', '1.1.12'),
    ('libz-sys', '1.1.12'),
    ('linux-raw-sys', '0.4.12'),
    ('lock_api', '0.4.11'),
    ('log', '0.4.20'),
    ('lz4', '1.24.0'),
    ('lz4-sys', '1.9.4'),
    ('matrixmultiply', '0.3.8'),
    ('md-5', '0.10.6'),
    ('memchr', '2.7.1'),
    ('memmap2', '0.7.1'),
    ('memoffset', '0.9.0'),
    ('mimalloc', '0.1.39'),
    ('mime', '0.3.17'),
    ('miniz_oxide', '0.7.1'),
    ('mio', '0.8.10'),
    ('multiversion', '0.7.3'),
    ('multiversion-macros', '0.7.3'),
    ('ndarray', '0.15.6'),
    ('now', '0.1.3'),
    ('ntapi', '0.4.1'),
    ('num', '0.4.1'),
    ('num-bigint', '0.4.4'),
    ('num-complex', '0.4.4'),
    ('num-integer', '0.1.45'),
    ('num-iter', '0.1.43'),
    ('num-rational', '0.4.1'),
    ('num-traits', '0.2.17'),
    ('num_cpus', '1.16.0'),
    ('numpy', '0.20.0'),
    ('object', '0.32.2'),
    ('object_store', '0.8.0'),
    ('once_cell', '1.19.0'),
    ('oorandom', '11.1.3'),
    ('openssl-probe', '0.1.5'),
    ('outref', '0.5.1'),
    ('p256', '0.11.1'),
    ('parking_lot', '0.12.1'),
    ('parking_lot_core', '0.9.9'),
    ('parquet-format-safe', '0.2.4'),
    ('parse-zoneinfo', '0.3.0'),
    ('percent-encoding', '2.3.1'),
    ('phf', '0.11.2'),
    ('phf_codegen', '0.11.2'),
    ('phf_generator', '0.11.2'),
    ('phf_shared', '0.11.2'),
    ('pin-project-lite', '0.2.13'),
    ('pin-utils', '0.1.0'),
    ('pkcs8', '0.9.0'),
    ('pkg-config', '0.3.28'),
    ('planus', '0.3.1'),
    ('plotters', '0.3.5'),
    ('plotters-backend', '0.3.5'),
    ('plotters-svg', '0.3.5'),
    ('polars-arrow-format', '0.1.0'),
    ('powerfmt', '0.2.0'),
    ('ppv-lite86', '0.2.17'),
    ('proc-macro2', '1.0.74'),
    ('proptest', '1.4.0'),
    ('prost', '0.11.9'),
    ('prost-derive', '0.11.9'),
    ('pyo3', '0.20.1'),
    ('pyo3-build-config', '0.20.1'),
    ('pyo3-built', '0.4.7'),
    ('pyo3-ffi', '0.20.1'),
    ('pyo3-macros', '0.20.1'),
    ('pyo3-macros-backend', '0.20.1'),
    ('quad-rand', '0.2.1'),
    ('quick-xml', '0.31.0'),
    ('quickcheck', '1.0.3'),
    ('quote', '1.0.35'),
    ('rand', '0.8.5'),
    ('rand_chacha', '0.3.1'),
    ('rand_core', '0.6.4'),
    ('rand_distr', '0.4.3'),
    ('rand_regex', '0.15.1'),
    ('rand_xorshift', '0.3.0'),
    ('rawpointer', '0.2.1'),
    ('rayon', '1.8.0'),
    ('rayon-core', '1.12.0'),
    ('redox_syscall', '0.4.1'),
    ('ref-cast', '1.0.22'),
    ('ref-cast-impl', '1.0.22'),
    ('regex', '1.10.2'),
    ('regex-automata', '0.4.3'),
    ('regex-lite', '0.1.5'),
    ('regex-syntax', '0.6.29'),
    ('regex-syntax', '0.8.2'),
    ('reqwest', '0.11.23'),
    ('rfc6979', '0.3.1'),
    ('ring', '0.17.7'),
    ('rle-decode-fast', '1.0.3'),
    ('rustc-demangle', '0.1.23'),
    ('rustc-hash', '1.1.0'),
    ('rustc_version', '0.4.0'),
    ('rustix', '0.38.28'),
    ('rustls', '0.21.10'),
    ('rustls-native-certs', '0.6.3'),
    ('rustls-pemfile', '1.0.4'),
    ('rustls-webpki', '0.101.7'),
    ('rustversion', '1.0.14'),
    ('ryu', '1.0.16'),
    ('same-file', '1.0.6'),
    ('sample-arrow2', '0.17.1'),
    ('sample-std', '0.1.1'),
    ('sample-test', '0.1.1'),
    ('sample-test-macros', '0.1.1'),
    ('schannel', '0.1.23'),
    ('scopeguard', '1.2.0'),
    ('sct', '0.7.1'),
    ('sec1', '0.3.0'),
    ('security-framework', '2.9.2'),
    ('security-framework-sys', '2.9.1'),
    ('semver', '1.0.20'),
    ('seq-macro', '0.3.5'),
    ('serde', '1.0.194'),
    ('serde_derive', '1.0.194'),
    ('serde_json', '1.0.110'),
    ('serde_spanned', '0.6.5'),
    ('serde_urlencoded', '0.7.1'),
    ('sha1', '0.10.6'),
    ('sha2', '0.10.8'),
    ('signal-hook-registry', '1.4.1'),
    ('signature', '1.6.4'),
    ('simd-json', '0.13.4'),
    ('simdutf8', '0.1.4'),
    ('siphasher', '0.3.11'),
    ('slab', '0.4.9'),
    ('smallvec', '1.11.2'),
    ('smartstring', '1.0.1'),
    ('snafu', '0.7.5'),
    ('snafu-derive', '0.7.5'),
    ('snap', '1.1.1'),
    ('socket2', '0.5.5'),
    ('spin', '0.9.8'),
    ('spki', '0.6.0'),
    ('sqlparser', '0.39.0'),
    ('static_assertions', '1.1.0'),
    ('streaming-decompression', '0.1.2'),
    ('streaming-iterator', '0.1.9'),
    ('strength_reduce', '0.2.4'),
    ('strum', '0.25.0'),
    ('strum_macros', '0.25.3'),
    ('subtle', '2.5.0'),
    ('syn', '1.0.109'),
    ('syn', '2.0.46'),
    ('sysinfo', '0.30.3'),
    ('system-configuration', '0.5.1'),
    ('system-configuration-sys', '0.5.0'),
    ('target-features', '0.1.5'),
    ('target-lexicon', '0.12.12'),
    ('tempfile', '3.9.0'),
    ('thiserror', '1.0.56'),
    ('thiserror-impl', '1.0.56'),
    ('time', '0.3.31'),
    ('time-core', '0.1.2'),
    ('time-macros', '0.2.16'),
    ('tiny-keccak', '2.0.2'),
    ('tinytemplate', '1.2.1'),
    ('tinyvec', '1.6.0'),
    ('tinyvec_macros', '0.1.1'),
    ('tokio', '1.35.1'),
    ('tokio-macros', '2.2.0'),
    ('tokio-rustls', '0.24.1'),
    ('tokio-util', '0.7.10'),
    ('toml', '0.7.8'),
    ('toml_datetime', '0.6.5'),
    ('toml_edit', '0.19.15'),
    ('tower-service', '0.3.2'),
    ('tracing', '0.1.40'),
    ('tracing-attributes', '0.1.27'),
    ('tracing-core', '0.1.32'),
    ('try-lock', '0.2.5'),
    ('typed-builder', '0.16.2'),
    ('typed-builder-macro', '0.16.2'),
    ('typenum', '1.17.0'),
    ('unarray', '0.1.4'),
    ('unicode-bidi', '0.3.14'),
    ('unicode-ident', '1.0.12'),
    ('unicode-normalization', '0.1.22'),
    ('unicode-reverse', '1.0.8'),
    ('unicode-segmentation', '1.10.1'),
    ('unicode-width', '0.1.11'),
    ('unindent', '0.2.3'),
    ('untrusted', '0.9.0'),
    ('url', '2.5.0'),
    ('urlencoding', '2.1.3'),
    ('uuid', '1.6.1'),
    ('value-trait', '0.8.0'),
    ('vcpkg', '0.2.15'),
    ('version_check', '0.9.4'),
    ('vsimd', '0.8.0'),
    ('walkdir', '2.4.0'),
    ('want', '0.3.1'),
    ('wasi', '0.11.0+wasi-snapshot-preview1'),
    ('wasm-bindgen', '0.2.89'),
    ('wasm-bindgen-backend', '0.2.89'),
    ('wasm-bindgen-futures', '0.4.39'),
    ('wasm-bindgen-macro', '0.2.89'),
    ('wasm-bindgen-macro-support', '0.2.89'),
    ('wasm-bindgen-shared', '0.2.89'),
    ('wasm-streams', '0.3.0'),
    ('web-sys', '0.3.66'),
    ('webpki-roots', '0.25.3'),
    ('winapi', '0.3.9'),
    ('winapi-i686-pc-windows-gnu', '0.4.0'),
    ('winapi-util', '0.1.6'),
    ('winapi-x86_64-pc-windows-gnu', '0.4.0'),
    ('windows', '0.52.0'),
    ('windows-core', '0.52.0'),
    ('windows-sys', '0.48.0'),
    ('windows-sys', '0.52.0'),
    ('windows-targets', '0.48.5'),
    ('windows-targets', '0.52.0'),
    ('windows_aarch64_gnullvm', '0.48.5'),
    ('windows_aarch64_gnullvm', '0.52.0'),
    ('windows_aarch64_msvc', '0.48.5'),
    ('windows_aarch64_msvc', '0.52.0'),
    ('windows_i686_gnu', '0.48.5'),
    ('windows_i686_gnu', '0.52.0'),
    ('windows_i686_msvc', '0.48.5'),
    ('windows_i686_msvc', '0.52.0'),
    ('windows_x86_64_gnu', '0.48.5'),
    ('windows_x86_64_gnu', '0.52.0'),
    ('windows_x86_64_gnullvm', '0.48.5'),
    ('windows_x86_64_gnullvm', '0.52.0'),
    ('windows_x86_64_msvc', '0.48.5'),
    ('windows_x86_64_msvc', '0.52.0'),
    ('winnow', '0.5.31'),
    ('winreg', '0.50.0'),
    ('xmlparser', '0.13.6'),
    ('xxhash-rust', '0.8.8'),
    ('zerocopy', '0.7.32'),
    ('zerocopy-derive', '0.7.32'),
    ('zeroize', '1.7.0'),
    ('zstd', '0.13.0'),
    ('zstd-safe', '7.0.0'),
    ('zstd-sys', '2.0.9+zstd.1.5.5'),
]

[Micket]

Hm, well i used the 'package' segment in the toml to detect which is the library itself. But this thing has a "workspace.package" instead. Not really sure what to make of that, or what we want to have in the final list here.

[Flamefire]

('jsonpath_lib', '0.3.0', 'git+https://github.com/ritchie46/jsonpath?branch=improve_compiled'), looks wrong. The Cargo EasyBlock only accepts tuples of 2 or 4 elements, not 3. Also where did git+https came from? Don't know if this is somehow valid/understood by something.

[pavelToman]

For me there is another error when I try to to generate crates list:

python -m easybuild.easyblocks.generic.cargo . 
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/user/gent/470/vsc47063/easybuild/easybuild-easyblocks/easybuild/easyblocks/generic/cargo.py", line 270, in <module>
    app_in_cratesio, crates, other = generate_crate_list(sys.argv[1])
  File "/user/gent/470/vsc47063/easybuild/easybuild-easyblocks/easybuild/easyblocks/generic/cargo.py", line 240, in generate_crate_list
    import toml
ModuleNotFoundError: No module named 'toml'

[Micket]

It requires the python module toml to use the easyblock to generate the list.

[pavelToman]

No module named 'toml' error solved by module load Python/3.10.8-GCCcore-12.2.0 but after I got KeyError: 'package' - the original error.

[Micket]

I believe you are not using the PR'ed easyblock

[branfosj]

fixed in #3084