[package.] Sections Getting Stripped From Cargo.toml

[benwis]

Hello! I think this is another change with the -Z build-std functionality, but it's stripping the package. sections from my Cargo.toml now, and I need those.

[package.metadata.cargo-all-features]
denylist = ["axum", "tower", "tower-http", "tokio", "sqlx", "leptos_axum", "femark", "chrono", "argon2","rand", "rand_core"]
skip_feature_sets = [["csr", "ssr"], ["csr", "hydrate"], ["ssr", "hydrate"]]

[package.metadata.leptos]
# The name used by wasm-bindgen/cargo-leptos for the JS/WASM bundle. Defaults to the crate name
output-name = "benwis_leptos"

Running nix build . --keep-failed shows every other tag in the file except these two. The first one isn't custom, and probably should stay, the second is.

[ipetkov]

Package metadata is intentionally stripped by default by buildDepsOnly to avoid invalidating the build cache if it changes (generally speaking package metadata does not affect how dependencies are built).

Are you using buildPackage with a custom build command by any chance? If so, my recommendation would be to explicitly set cargoArtifacts = buildDepsOnly ...; so that the custom command (which I assume needs to consume the metadata in Cargo.toml) only runs for building the final binary and not while caching dependency artifacts!

[benwis]

I am, I moved towards your example with this:

{
  description = "Build the Benwis Blog!";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

    crane = {
      url = "github:ipetkov/crane";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    flake-utils.url = "github:numtide/flake-utils";

    cargo-leptos = {
      #url= "github:leptos-rs/cargo-leptos/v1.7";
      url = "github:benwis/cargo-leptos";
      flake = false;
    };

    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      inputs = {
        nixpkgs.follows = "nixpkgs";
        flake-utils.follows = "flake-utils";
      };
    };
    advisory-db = {
      url = "github:rustsec/advisory-db";
      flake = false;
    };
  };

  outputs = { self, nixpkgs, crane, flake-utils, advisory-db, rust-overlay, ... } @inputs:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
            overlays = [ (import rust-overlay) ];
          };

          rustTarget = pkgs.rust-bin.selectLatestNightlyWith (toolchain: toolchain.default.override {
            extensions = [ "rust-src" "rust-analyzer" ];
            targets = [ "wasm32-unknown-unknown" ];
          });

          # NB: we don't need to overlay our custom toolchain for the *entire*
          # pkgs (which would require rebuidling anything else which uses rust).
          # Instead, we just want to update the scope that crane will use by appendings
          inherit (pkgs) lib;
          # our specific toolchain there.
          craneLib = (crane.mkLib pkgs).overrideToolchain rustTarget;
          #craneLib = crane.lib.${system};
          # Only keeps markdown files
          protoFilter = path: _type: builtins.match ".*proto$" path != null;
          sqlxFilter = path: _type: builtins.match ".*json$" path != null;
          sqlFilter = path: _type: builtins.match ".*sql$" path != null;
          cssFilter = path: _type: builtins.match ".*css$" path != null;
          ttfFilter = path: _type: builtins.match ".*ttf$" path != null;
          woff2Filter = path: _type: builtins.match ".*woff2$" path != null;
          webpFilter = path: _type: builtins.match ".*webp$" path != null;
          jpegFilter = path: _type: builtins.match ".*jpeg$" path != null;
          pngFilter = path: _type: builtins.match ".*png$" path != null;
          icoFilter = path: _type: builtins.match ".*ico$" path != null;
          protoOrCargo = path: type:
            (protoFilter path type) || (craneLib.filterCargoSources path type) || (sqlxFilter path type) || (sqlFilter path type) || (cssFilter path type) || (woff2Filter path type) || (ttfFilter path type) || (webpFilter path type) || (icoFilter path type) || (jpegFilter path type) || (pngFilter path type);
          # other attributes omitted

          # Include more types of files in our bundle
          src = lib.cleanSourceWith {
            src = ./.; # The original, unfiltered source
            filter = protoOrCargo;
          };
          #    src = craneLib.cleanCargoSource ./.;

          # Common arguments can be set here
          commonArgs = {
            inherit src;
          buildInputs = [
            # Add additional build inputs here
            cargo-leptos
            pkgs.pkg-config
            pkgs.openssl
            pkgs.protobuf
            pkgs.binaryen
            pkgs.cargo-generate
          ] ++ lib.optionals pkgs.stdenv.isDarwin [
            # Additional darwin specific inputs can be set here
            pkgs.libiconv
          ];
        };

          # Build *just* the cargo dependencies, so we can reuse
          # all of that work (e.g. via cachix) when running in CI
          cargoArtifacts = craneLib.buildDepsOnly (commonArgs // {

          });

          # Build the actual crate itself, reusing the dependency
          # artifacts from above.
          benwis_leptos = craneLib.buildPackage (commonArgs // {
            pname = "benwis_leptos";
            buildPhaseCargoCommand = "cargo leptos build --release";
            installPhaseCommand = ''
            mkdir -p $out/bin
            cp target/server/release/benwis_leptos $out/bin/
            cp -r target/site $out/bin/
            '';
            # Prevent cargo test and nextest from duplicating tests
            doCheck = false;
            inherit cargoArtifacts;
            # ALL CAPITAL derivations will get forwarded to mkDerivation and will set the env var during build
            SQLX_OFFLINE = "true";
            LEPTOS_BIN_TARGET_TRIPLE = "x86_64-unknown-linux-gnu"; # Adding this allows -Zbuild-std to work and shave 100kb off the WASM
            APP_ENVIRONMENT = "production";
          });

          cargo-leptos = pkgs.rustPlatform.buildRustPackage rec {
            pname = "cargo-leptos";
            #version = "0.1.7";
            version = "0.1.8.1";
            buildFeatures = ["no_downloads"]; # cargo-leptos will try to download Ruby and other things without this feature

            src = inputs.cargo-leptos; 

            cargoSha256 = "sha256-FBtbVli9qJQYsd6aLiizy9qup8E0VOVxkmYX6K09aO0=";

            nativeBuildInputs = [pkgs.pkg-config pkgs.openssl];

            buildInputs = with pkgs;
              [openssl pkg-config]
              ++ lib.optionals stdenv.isDarwin [
              Security
            ];

            doCheck = false; # integration tests depend on changing cargo config

            meta = with lib; {
            description = "A build tool for the Leptos web framework";
            homepage = "https://github.com/leptos-rs/cargo-leptos";
            changelog = "https://github.com/leptos-rs/cargo-leptos/blob/v${version}/CHANGELOG.md";
            license = with licenses; [mit];
            maintainers = with maintainers; [benwis];
          };
      };
          flyConfig = ./fly.toml;

          # Deploy the image to Fly with our own bash script
          flyDeploy = pkgs.writeShellScriptBin "flyDeploy" ''
            OUT_PATH=$(nix build --print-out-paths .#container)
            HASH=$(echo $OUT_PATH | grep -Po "(?<=store\/)(.*?)(?=-)")
            ${pkgs.skopeo}/bin/skopeo --insecure-policy --debug copy docker-archive:"$OUT_PATH" docker://registry.fly.io/$FLY_PROJECT_NAME:$HASH --dest-creds x:"$FLY_AUTH_TOKEN" --format v2s2
            ${pkgs.flyctl}/bin/flyctl deploy -i registry.fly.io/$FLY_PROJECT_NAME:$HASH -c ${flyConfig} --remote-only
          '';
        in
        {
          checks = {
            # Build the crate as part of `nix flake check` for convenience
            inherit benwis_leptos;

            # Run clippy (and deny all warnings) on the crate source,
            # again, resuing the dependency artifacts from above.
            #
            # Note that this is done as a separate derivation so that
            # we can block the CI if there are issues here, but not
            # prevent downstream consumers from building our crate by itself.
            benwis_leptos-clippy = craneLib.cargoClippy (commonArgs // {
              inherit cargoArtifacts;
              cargoClippyExtraArgs = "--all-targets -- --deny warnings";
            });

            benwis_leptos-doc = craneLib.cargoDoc (commonArgs //{
              inherit cargoArtifacts;
            });

            # Check formatting
            benwis_leptos-fmt = craneLib.cargoFmt {
              inherit src;
            };

            # Audit dependencies
            benwis_leptos-audit = craneLib.cargoAudit {
              inherit src advisory-db;
            };

            # Run tests with cargo-nextest
            # Consider setting `doCheck = false` on `benwis_leptos` if you do not want
            # the tests to run twice
            # benwis_leptos-nextest = craneLib.cargoNextest {
            #  inherit cargoArtifacts src buildInputs;
            #  partitions = 1;
            #  partitionType = "count";
            #};
          } // lib.optionalAttrs (system == "x86_64-linux") {
            # NB: cargo-tarpaulin only supports x86_64 systems
            # Check code coverage (note: this will not upload coverage anywhere)
            #benwis_leptos-coverage = craneLib.cargoTarpaulin {
            #  inherit cargoArtifacts src;
            #};

          };

          packages.default = benwis_leptos;

          apps.default = flake-utils.lib.mkApp {
            drv = benwis_leptos;
          };

          # Create an option to build a docker image from this package 
          packages.container = pkgs.dockerTools.buildImage {
            name = "benwis_leptos";
            #tag = "latest";
            created = "now";
            copyToRoot = pkgs.buildEnv {
              name = "image-root";
              paths = [ pkgs.cacert ./.  ];
              pathsToLink = [ "/bin" "/db" "/migrations" ];
            };
            config = {
              Env = [ "PATH=${benwis_leptos}/bin" "APP_ENVIRONMENT=production" "LEPTOS_OUTPUT_NAME=benwis_leptos" "LEPTOS_SITE_ADDR=0.0.0.0:3000" "LEPTOS_SITE_ROOT=${benwis_leptos}/bin/site" ];

              ExposedPorts = {
                "3000/tcp" = { };
              };

              Cmd = [ "${benwis_leptos}/bin/benwis_leptos" ];
            };

          };

          apps.flyDeploy = flake-utils.lib.mkApp {
            drv = flyDeploy;
          };
          devShells.default = pkgs.mkShell {
            inputsFrom = builtins.attrValues self.checks;

            # Extra inputs can be added here
            nativeBuildInputs = with pkgs; [
              rustTarget
              openssl
              mysql80
              dive
              sqlx-cli
              wasm-pack
              pkg-config
              binaryen
              nodejs
              nodePackages.tailwindcss
              cargo-leptos
              protobuf
              skopeo
              flyctl
            ];
            RUST_SRC_PATH = "${rustTarget}/lib/rustlib/src/rust/library";
          };
        });
}

I thought buildDepsOnly only applied for workspaces, but I'm wondering if I need to add that back in for this single package. I'm not sure I understand how it's supposed to fix things

[ipetkov]

buildDepsOnly is meant to build and cache all dependency crates (and their dependencies, and so on), regardless if the workspace has only one crate or not. The idea is that usually the workspace source changes more often than updating dependencies (i.e. updating Cargo.toml or Cargo.lock), so we can reuse the result of running cargo {check,build,test} on the dependency tree (minus the source of the workspace itself).

buildPackage is a somewhat opinionated shortcut for common project formats. If you call buildPackage args and args does not define cargoArtifacts, then buildPackage will behave as if you had called buildPackage (args // { cargoArtifacts = buildDepsOnly args);

The key here is you are using cargo leptos build --release to build the actual crate which (I assume) needs the package.metadata field to be present, but what's happening is buildPhaseCargoCommand is also being inherited by buildDepsOnly which tries to run with the stripped version of the source which lacks the metadata.

Instead, it is possible to separate the two build steps (by being a bit more explicit) and control which build command is used at which stage of the build. I see your example above is doing an equivalent of this, but just for the sake of posterity I'd like to give an annotated example:

let
  commonArgs = {
    inherit src;
    # etc. other parameters but don't set 
  };

  # This will effectively make a workspace which contains all dependencies, but not the source of the workspace.
  # Then it will run `cargo check; cargo build; cargo test` and cache the results
  cargoArtifacts = craneLib.buildDepsOnly commonArgs;
in
craneLib.buildPackage (commonArgs // {
   # Reuse the artifacts from before but this time build with the real workspace source
  inherit cargoArtifacts;

  # This time the build will be done by `cargo leptos` which will see the entire source
  # including the extra package.metadata
  buildPhaseCargoCommand = "cargo leptos build --release";
}

I thought buildDepsOnly only applied for workspaces, but I'm wondering if I need to add that back in for this single package. I'm not sure I understand how it's supposed to fix things

In short, you always want to have buildDepsOnly at the base of the derivation chain, otherwise you will rebuild all crates from scratch as if you ran cargo clean; cargo build. If you do not specify cargoArtifacts then buildPackage will assume an invocation for you!

I hope this clarifies things!

[benwis]

It does, thank you very much. Adding that and setting up Vendor for both builds, as well as disabling doCheck, fixed it for me. Thanks for all you work on Crane!