nix-community / crate2nix

rebuild only changed crates in CI with crate2nix and nix
https://nix-community.github.io/crate2nix/
Apache License 2.0
363 stars 86 forks source link

error: invalid metadata version found #353

Closed bradzacher closed 3 months ago

bradzacher commented 3 months ago

I've been trying to get crate2nix working in my company's monorepo for a while now. We have a heinous nix setup with some super complicated overlays and multiple nix commits via pins that I don't endorse - but it is what it is.

I have it almost working now - I'm able to build almost all of the crates and dependencies. However the build is failing right now on this one error:

error: builder for '/nix/store/h3br1jcgzsxfhxynja29ifkdfbqw9g4z-rust_thiserror-1.0.56.drv' failed with exit code 1;
       last 10 log lines:
       >    --> src/lib.rs:248:9
       >     |
       > 248 | pub use thiserror_impl::*;
       >     |         ^^^^^^^^^^^^^^
       >     |
       >     = note: invalid metadata version found: /nix/store/8a80l7533ias1mk6rngdxhr99gzh5vgz-rust_thiserror-impl-1.0.56-lib/lib/libthiserror_impl-73e857b6f1.dylib
       >
       > error: aborting due to previous error
       >
       > For more information about this error, try `rustc --explain E0786`.
       For full logs, run 'nix log /nix/store/h3br1jcgzsxfhxynja29ifkdfbqw9g4z-rust_thiserror-1.0.56.drv'.
error: 1 dependencies of derivation '/nix/store/i3li4g923i0nj259qdhg4yhghlkxr8cp-rust_crate2nix-0.14.0.drv' failed to build

I've tried a few things, but I'm a bit out of my depth in terms of debugging this. I doubt you'll be able to directly fix things - but I was wondering if you had any ideas on what might be causing this to help me debug more.


Separately (maybe related?) I have found I needed the following patch to get things working so that I could pass the correct rust version into crate2nix[^1]: https://github.com/nix-community/crate2nix/compare/master...bradzacher:crate2nix:patch-1

[^1]: tl;dr here - our base commit is quite old, but it's hard to update at our scale. So we pull in newer nix commits to grab specific, targeted dependencies - like rust. But this does create a world where there's two potential versions of rust at once and so I needed to make sure I was passing the overridden rust version all the way through into crate2nix.

apoelstra commented 3 months ago

The issue looks like your version of thiserror is incompatible with your version of rustc. You'll need to change the pin on one of them.

bradzacher commented 3 months ago

That's weird because the rust version (1.75.0) is the same one I can use outside of nix and it builds fine.

Maybe I haven't correctly composed all the rust bits together and there's an old version from our base slipping in somewhere.

apoelstra commented 3 months ago

You may want to tweak buildRustCratesForPkgs to echo some stats about your rustc version in the derivation so you can see exactly what's happening. You probably also need to find the derivation for thiserror-impl and see what version of rustc is being used there.

I would appreciate such a change upstream to stick a cargo -V and rustc -V at the top of everything, but I dunno how the maintainers would feel about it.

bradzacher commented 3 months ago

Hmmm - showing the derivation it looks like it's depending on the correct rust version (1.75.0) I can see both of these in the inputDrvs:

And I can see these in the nativeBuildInputs:

nix derivation show ```json { "/nix/store/h3br1jcgzsxfhxynja29ifkdfbqw9g4z-rust_thiserror-1.0.56.drv": { "args": [ "-e", "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh" ], "builder": "/nix/store/0j90xxj4in0yk2dp7mpvsm8jqd9mrbs3-bash-5.2p26/bin/bash", "env": { "__darwinAllowLocalNetworking": "", "__impureHostDeps": "/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom /bin/sh", "__propagatedImpureHostDeps": "", "__propagatedSandboxProfile": "", "__sandboxProfile": "", "__structuredAttrs": "", "build": "", "buildDependencies": "", "buildInputs": "/nix/store/hx90cddimxf9j93mynvkynjzxdiwqs5z-libiconv-50", "buildPhase": "runHook preBuild\n\n# configure & source common build functions\nLIB_RUSTC_OPTS=\"-C opt-level=3 -C codegen-units=1 --remap-path-prefix=$NIX_BUILD_TOP=/ --extern thiserror_impl=/nix/store/8a80l7533ias1mk6rngdxhr99gzh5vgz-rust_thiserror-impl-1.0.56-lib/lib/libthiserror_impl-73e857b6f1.dylib --cfg feature=\\\"default\\\" --edition 2021 -C metadata=9af28be496 -C extra-filename=-9af28be496 --crate-type lib\"\nBIN_RUSTC_OPTS=\"-C linker=/nix/store/k687y4hiraslz65qz8pli1wb5wmcabvn-clang-wrapper-16.0.6/bin/cc -C opt-level=3 -C codegen-units=1 --remap-path-prefix=$NIX_BUILD_TOP=/ --extern thiserror_impl=/nix/store/8a80l7533ias1mk6rngdxhr99gzh5vgz-rust_thiserror-impl-1.0.56-lib/lib/libthiserror_impl-73e857b6f1.dylib --cfg feature=\\\"default\\\" --edition 2021\"\nLIB_EXT=\".dylib\"\nLIB_PATH=\"\"\nLIB_NAME=\"thiserror\"\n\nCRATE_NAME='thiserror'\n\nsetup_link_paths\n\nif [[ -e \"$LIB_PATH\" ]]; then\n build_lib \"$LIB_PATH\"\n \nelif [[ -e src/lib.rs ]]; then\n build_lib src/lib.rs\n \nfi\n\n\n\n\n\n\n\n# If crateBin is empty and hasCrateBin is not set then we must try to\n# detect some kind of bin target based on some files that might exist.\nif [[ -e src/main.rs ]]; then\n mkdir -p target/bin\n build_bin thiserror src/main.rs\nfi\nfor i in src/bin/*.rs; do #*/\n mkdir -p target/bin\n build_bin \"$(basename $i .rs)\" \"$i\"\ndone\n\n# Remove object files to avoid \"wrong ELF type\"\nfind target -type f -name \"*.o\" -print0 | xargs -0 rm -f\nrunHook postBuild\n", "buildTests": "", "builder": "/nix/store/0j90xxj4in0yk2dp7mpvsm8jqd9mrbs3-bash-5.2p26/bin/bash", "cmakeFlags": "", "codegenUnits": "1", "colors": "always", "completeBuildDeps": "", "completeDeps": "/nix/store/8a80l7533ias1mk6rngdxhr99gzh5vgz-rust_thiserror-impl-1.0.56-lib /nix/store/jqrzzlgp8n001d0m8w2h05iv06fk3l2q-rust_proc-macro2-1.0.76-lib /nix/store/7q5m8qg509k41wlk0m7a8bk5am3k3jvc-rust_quote-1.0.35-lib /nix/store/yw7ifapaabpb40isd5faala6pgnd0nbs-rust_syn-2.0.48-lib /nix/store/zziq8l3d9w957jhwsny5jpdrpka1wdpn-rust_unicode-ident-1.0.12-lib", "configureFlags": "", "configurePhase": "echo_colored() {\n local echo_args=\"\";\nwhile [[ \"x$1\" =~ ^x- ]]; do\n echo_args+=\" $1\"\n shift\ndone\n\nlocal start_escape=\"$(printf '\\033[0;1;32m')\"\nlocal reset=\"$(printf '\\033[0m')\"\necho $echo_args $start_escape\"$@\"$reset\n\n}\n\necho_error() {\n local echo_args=\"\";\nwhile [[ \"x$1\" =~ ^x- ]]; do\n echo_args+=\" $1\"\n shift\ndone\n\nlocal start_escape=\"$(printf '\\033[0;1;31m')\"\nlocal reset=\"$(printf '\\033[0m')\"\necho $echo_args $start_escape\"$@\"$reset\n\n}\n\n noisily() {\n\t echo_colored -n \"Running \"\n echo $@\n\t \n\t $@\n }\n\nsource /nix/store/vnl78pgq13niprbkdnn1367cdsr1l63i-lib.sh\n\nnoisily cd \".\"\n\n\n\nrunHook preConfigure\n\nsymlink_dependency() {\n # $1 is the nix-store path of a dependency\n # $2 is the target path\n i=$1\n ln -s -f $i/lib/*.rlib $2 #*/\n ln -s -f $i/lib/*.so $i/lib/*.dylib $2 #*/\n if [ -e $i/env ]; then\n source $i/env\n fi\n}\n\n# The following steps set up the dependencies of the crate. Two\n# kinds of dependencies are distinguished: build dependencies\n# (used by the build script) and crate dependencies. For each\n# dependency we have to:\n#\n# - Make its Rust library available to rustc. This is done by\n# symlinking all library dependencies into a directory that\n# can be provided to rustc.\n# - Accumulate linking flags. These flags are largely used for\n# linking native libraries.\n#\n# The crate link flags are added to the `link` and `link.final`\n# files. The `link` file is used for linkage in the current\n# crate. The `link.final` file will be copied to the output and can\n# be used by downstream crates to get the linker flags of this\n# crate.\n\nmkdir -p target/{deps,lib,build,buildDeps}\nchmod uga+w target -R\necho > target/link\necho > target/link.final\n\n# Prepare crate dependencies\nfor i in /nix/store/8a80l7533ias1mk6rngdxhr99gzh5vgz-rust_thiserror-impl-1.0.56-lib /nix/store/jqrzzlgp8n001d0m8w2h05iv06fk3l2q-rust_proc-macro2-1.0.76-lib /nix/store/7q5m8qg509k41wlk0m7a8bk5am3k3jvc-rust_quote-1.0.35-lib /nix/store/yw7ifapaabpb40isd5faala6pgnd0nbs-rust_syn-2.0.48-lib /nix/store/zziq8l3d9w957jhwsny5jpdrpka1wdpn-rust_unicode-ident-1.0.12-lib; do\n symlink_dependency $i target/deps\n if [ -e \"$i/lib/link\" ]; then\n cat $i/lib/link >> target/link\n cat $i/lib/link >> target/link.final\n fi\ndone\n\n# Prepare crate build dependencies that are used for the build script.\nfor i in ; do\n symlink_dependency $i target/buildDeps\n if [ -e \"$i/lib/link\" ]; then\n cat $i/lib/link >> target/link.build\n fi\ndone\n\n# Remove duplicate linker flags from the build dependencies.\nif [[ -e target/link.build ]]; then\n sort -uo target/link.build target/link.build\nfi\n\n# Remove duplicate linker flags from the dependencies.\nsort -uo target/link target/link\ntr '\\n' ' ' < target/link > target/link_\n\n# Remove duplicate linker flags from the that are written\n# to the derivation's output.\nsort -uo target/link.final target/link.final\n\nEXTRA_BUILD=\"\"\nBUILD_OUT_DIR=\"\"\n\n# Set up Cargo Environment variables: https://doc.rust-lang.org/cargo/reference/environment-variables.html\nexport CARGO_PKG_NAME=thiserror\nexport CARGO_PKG_VERSION=1.0.56\nexport CARGO_PKG_AUTHORS=\"David Tolnay \"\nexport CARGO_PKG_DESCRIPTION=\"\"\n\nexport CARGO_CFG_TARGET_ARCH=aarch64\nexport CARGO_CFG_TARGET_OS=macos\nexport CARGO_CFG_TARGET_FAMILY=\"unix\"\nexport CARGO_CFG_UNIX=1\nexport CARGO_CFG_TARGET_ENV=\"gnu\"\nexport CARGO_CFG_TARGET_ENDIAN=little\nexport CARGO_CFG_TARGET_POINTER_WIDTH=64\nexport CARGO_CFG_TARGET_VENDOR=apple\n\nexport CARGO_MANIFEST_DIR=$(pwd)\nexport CARGO_MANIFEST_LINKS=\nexport DEBUG=\"\"\nexport OPT_LEVEL=\"3\"\nexport TARGET=\"aarch64-apple-darwin\"\nexport HOST=\"aarch64-apple-darwin\"\nexport PROFILE=release\nexport OUT_DIR=$(pwd)/target/build/thiserror.out\nexport CARGO_PKG_VERSION_MAJOR=1\nexport CARGO_PKG_VERSION_MINOR=0\nexport CARGO_PKG_VERSION_PATCH=56\nexport CARGO_PKG_VERSION_PRE=\"\"\nexport CARGO_PKG_HOMEPAGE=\"\"\nexport CARGO_PKG_LICENSE=\"\"\nexport CARGO_PKG_LICENSE_FILE=\"\"\nexport CARGO_PKG_README=\"\"\nexport CARGO_PKG_REPOSITORY=\"\"\nexport CARGO_PKG_RUST_VERSION=\"\"\nexport NUM_JOBS=$NIX_BUILD_CORES\nexport RUSTC=\"rustc\"\nexport RUSTDOC=\"rustdoc\"\n\nBUILD=\"\"\nif [[ ! -z \"\" ]] ; then\n BUILD=\nelif [[ -e \"build.rs\" ]]; then\n BUILD=\"build.rs\"\nfi\n\n# Compile and run the build script, when available.\nif [[ ! -z \"$BUILD\" ]] ; then\n echo_build_heading \"$BUILD\" thiserror\n mkdir -p target/build/thiserror\n EXTRA_BUILD_FLAGS=\"\"\n if [ -e target/link.build ]; then\n EXTRA_BUILD_FLAGS=\"$EXTRA_BUILD_FLAGS $(tr '\\n' ' ' < target/link.build)\"\n fi\n noisily rustc --crate-name build_script_build $BUILD --crate-type bin -C opt-level=3 -C codegen-units=1 --edition 2021 \\\n --cfg feature=\\\"default\\\" --out-dir target/build/thiserror --emit=dep-info,link \\\n -L dependency=target/buildDeps --cap-lints allow $EXTRA_BUILD_FLAGS --color always\n\n mkdir -p target/build/thiserror.out\n export RUST_BACKTRACE=1\n BUILD_OUT_DIR=\"-L $OUT_DIR\"\n mkdir -p $OUT_DIR\n\n (\n # Features should be set as environment variable for build scripts:\n # https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts\n for feature in DEFAULT; do\n export CARGO_FEATURE_$feature=1\n done\n\n target/build/thiserror/build_script_build > target/build/thiserror.opt\n )\n\n set +e\n EXTRA_BUILD=$(sed -n \"s/^cargo:rustc-flags=\\(.*\\)/\\1/p\" target/build/thiserror.opt | tr '\\n' ' ' | sort -u)\n EXTRA_FEATURES=$(sed -n \"s/^cargo:rustc-cfg=\\(.*\\)/--cfg \\1/p\" target/build/thiserror.opt | tr '\\n' ' ')\n EXTRA_LINK_ARGS=$(sed -n \"s/^cargo:rustc-link-arg=\\(.*\\)/-C link-arg=\\1/p\" target/build/thiserror.opt | tr '\\n' ' ')\n EXTRA_LINK_ARGS_BINS=$(sed -n \"s/^cargo:rustc-link-arg-bins=\\(.*\\)/-C link-arg=\\1/p\" target/build/thiserror.opt | tr '\\n' ' ')\n EXTRA_LINK_ARGS_LIB=$(sed -n \"s/^cargo:rustc-link-arg-lib=\\(.*\\)/-C link-arg=\\1/p\" target/build/thiserror.opt | tr '\\n' ' ')\n EXTRA_LINK_LIBS=$(sed -n \"s/^cargo:rustc-link-lib=\\(.*\\)/\\1/p\" target/build/thiserror.opt | tr '\\n' ' ')\n EXTRA_LINK_SEARCH=$(sed -n \"s/^cargo:rustc-link-search=\\(.*\\)/\\1/p\" target/build/thiserror.opt | tr '\\n' ' ' | sort -u)\n\n # We want to read part of every line that has cargo:rustc-env= prefix and\n # export it as environment variables. This turns out tricky if the lines\n # have spaces: we can't wrap the command in double quotes as that captures\n # all the lines in single output. We can't use while read loop because\n # exporting from inside of it doesn't make it to the outside scope. We\n # can't use xargs as export is a built-in and does not work from it. As a\n # last resort then, we change the IFS so that the for loop does not split\n # on spaces and reset it after we are done. See ticket #199298.\n #\n _OLDIFS=\"$IFS\"\n IFS=$'\\n'\n for env in $(sed -n \"s/^cargo:rustc-env=\\(.*\\)/\\1/p\" target/build/thiserror.opt); do\n export \"$env\"\n done\n IFS=\"$_OLDIFS\"\n\n CRATENAME=$(echo thiserror | sed -e \"s/\\(.*\\)-sys$/\\U\\1/\" -e \"s/-/_/g\")\n grep -P \"^cargo:(?!(rustc-|warning=|rerun-if-changed=|rerun-if-env-changed))\" target/build/thiserror.opt \\\n | awk -F= \"/^cargo:/ { sub(/^cargo:/, \\\"\\\", \\$1); gsub(/-/, \\\"_\\\", \\$1); print \\\"export \\\" toupper(\\\"DEP_$(echo $CRATENAME)_\\\" \\$1) \\\"=\\\" \\$2 }\" > target/env\n set -e\nfi\nrunHook postConfigure\n", "crateAuthors": "David Tolnay ", "crateDescription": "", "crateFeatures": "default", "crateHomepage": "", "crateLicense": "", "crateLicenseFile": "", "crateLinks": "", "crateName": "thiserror", "crateReadme": "", "crateRepository": "", "crateRustVersion": "", "crateType": "lib", "crateVersion": "1.0.56", "dependencies": "/nix/store/8a80l7533ias1mk6rngdxhr99gzh5vgz-rust_thiserror-impl-1.0.56-lib", "depsBuildBuild": "/nix/store/k687y4hiraslz65qz8pli1wb5wmcabvn-clang-wrapper-16.0.6", "depsBuildBuildPropagated": "", "depsBuildTarget": "", "depsBuildTargetPropagated": "", "depsHostHost": "", "depsHostHostPropagated": "", "depsTargetTarget": "", "depsTargetTargetPropagated": "", "doCheck": "", "doInstallCheck": "", "dontStrip": "1", "edition": "2021", "extraLinkFlags": "", "extraRustcOpts": "", "extraRustcOptsForBuildRs": "--edition 2021", "installPhase": "runHook preInstall\n# always create $out even if we do not have binaries. We are detecting binary targets during compilation, if those are missing there is no way to only have $lib\nmkdir $out\nif [[ -s target/env ]]; then\n mkdir -p $lib\n cp target/env $lib/env\nfi\nif [[ -s target/link.final ]]; then\n mkdir -p $lib/lib\n cp target/link.final $lib/lib/link\nfi\nif [[ \"$(ls -A target/lib)\" ]]; then\n mkdir -p $lib/lib\n cp -r target/lib/* $lib/lib #*/\n for library in $lib/lib/*.so $lib/lib/*.dylib; do #*/\n ln -s $library $(echo $library | sed -e \"s/-9af28be496//\")\n done\nfi\nif [[ \"$(ls -A target/build)\" ]]; then # */\n mkdir -p $lib/lib\n cp -r target/build/* $lib/lib # */\nfi\nif [[ -d target/bin ]]; then\n if [[ \"$(ls -A target/bin)\" ]]; then\n mkdir -p $out/bin\n cp -rP target/bin/* $out/bin # */\n fi\nfi\nrunHook postInstall\n", "lib": "/nix/store/wff12a9kqpbq4himqr3pb451l4wb6lhw-rust_thiserror-1.0.56-lib", "libName": "thiserror", "libPath": "", "mesonFlags": "", "metadata": "9af28be496", "name": "rust_thiserror-1.0.56", "nativeBuildInputs": "/nix/store/4j1xc3lq15g9y3vwdspi3g8ygl4xqajf-rustc-wrapper-1.75.0 /nix/store/k687y4hiraslz65qz8pli1wb5wmcabvn-clang-wrapper-16.0.6 /nix/store/dwb29sssx8m0kccpf886h5yqd9v9fjrj-cargo-1.75.0 /nix/store/h5jkagp2zfimj0sl7j7jv5pm162xzhnn-jq-1.7.1-dev /nix/store/hx90cddimxf9j93mynvkynjzxdiwqs5z-libiconv-50", "out": "/nix/store/9r5xq385kgw21bl8acrj1wc4007dxyip-rust_thiserror-1.0.56", "outputDev": "lib", "outputs": "out lib", "patches": "", "postBuild": "", "postConfigure": "", "postInstall": "", "postPatch": "", "postUnpack": "", "preBuild": "", "preConfigure": "", "preInstall": "", "prePatch": "", "preUnpack": "", "propagatedBuildInputs": "", "propagatedNativeBuildInputs": "", "release": "1", "sha256": "1b9hnzngjan4d89zjs16i01bcpcnvdwklyh73lj16xk28p37hhym", "src": "/nix/store/iqxix1ma0xnshg3ffyl7wxjjmjc28bpm-thiserror-1.0.56.tar.gz", "stdenv": "/nix/store/zf8hff9bnlfix7l7gjsnv3jc22lxzzxx-stdenv-darwin", "strictDeps": "", "stripExclude": "*.rlib", "system": "aarch64-darwin", "version": "1.0.56", "workspace_member": "." }, "inputDrvs": { "/nix/store/0gbnvdyw236jz0bj8bjhparx5z08p7cd-bash-5.2p26.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] }, "/nix/store/2vy66wxpxg5fnfpv3aixc2r3n2vqwgi0-rust_proc-macro2-1.0.76.drv": { "dynamicOutputs": {}, "outputs": [ "lib" ] }, "/nix/store/4bm1hds2ykayp2k7dr96nkn43k01ags6-clang-wrapper-16.0.6.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] }, "/nix/store/b3mh923y92pkpy7rpvb5xh2i0pr0awsi-thiserror-1.0.56.tar.gz.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] }, "/nix/store/f6bqnnzvc565cnpczldz1md686d3ryg4-rust_syn-2.0.48.drv": { "dynamicOutputs": {}, "outputs": [ "lib" ] }, "/nix/store/kznn438k9hic5b0xsrj3sfjragfhj288-rustc-wrapper-1.75.0.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] }, "/nix/store/lq02537d61ifqr64d9pyb16x9rfwnrqk-rust_unicode-ident-1.0.12.drv": { "dynamicOutputs": {}, "outputs": [ "lib" ] }, "/nix/store/nmcc5am7a4d2x9jdws9qj6ima5ww80gq-libiconv-50.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] }, "/nix/store/p6q6gb0n7abqki0ka8mb40ja461w0934-rust_thiserror-impl-1.0.56.drv": { "dynamicOutputs": {}, "outputs": [ "lib" ] }, "/nix/store/pl498vrpfy20j3k4dhscqmn42rf1can3-jq-1.7.1.drv": { "dynamicOutputs": {}, "outputs": [ "dev" ] }, "/nix/store/w6kz7sb33vdzlzb49fcfs19sncx8y23z-stdenv-darwin.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] }, "/nix/store/x8wgahxc04bma3qbk8jikfddklzs5i7l-rust_quote-1.0.35.drv": { "dynamicOutputs": {}, "outputs": [ "lib" ] }, "/nix/store/yl3gvn5s94jq7b50x1z86synzgwm489p-cargo-1.75.0.drv": { "dynamicOutputs": {}, "outputs": [ "out" ] } }, "inputSrcs": [ "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh", "/nix/store/vnl78pgq13niprbkdnn1367cdsr1l63i-lib.sh" ], "name": "rust_thiserror-1.0.56", "outputs": { "lib": { "path": "/nix/store/wff12a9kqpbq4himqr3pb451l4wb6lhw-rust_thiserror-1.0.56-lib" }, "out": { "path": "/nix/store/9r5xq385kgw21bl8acrj1wc4007dxyip-rust_thiserror-1.0.56" } }, "system": "aarch64-darwin" } } ```
bradzacher commented 3 months ago

I added cargo -V and rustc -V to build-rust-crate/build-crate.nix locally and I can see both at 1.75.0:

cargo 1.75.0
rustc 1.75.0 (82e1608df 2023-12-21) (built from a source tarball)

This is consistent with each crate I can see being built too.

Googling does suggest that this error is likely something to do with the versioning though...

bradzacher commented 3 months ago

I'm going to close this as a problem with our infra.