pandaman64
commented
2 years ago Another reproduction: using aws-config causes stack overflow: https://github.com/pandaman64/crate2nix-aws-infinite-recursion/blob/main/Cargo.toml
Closed sphw closed 10 months ago
pandaman64
commented
2 years ago Another reproduction: using aws-config causes stack overflow: https://github.com/pandaman64/crate2nix-aws-infinite-recursion/blob/main/Cargo.toml
andir
commented
2 years ago Whats the nix Version you are using? It might be worth checking against the latest master branch of Nix as that had some changes that enabled the compiler to do tail-call-optimization in a couple of places.
pandaman64
commented
2 years ago I'm using nix (Nix) 2.4pre20211006_53e4794. Let me try newer version later.
pandaman64
commented
2 years ago I updated nix command to nix (Nix) 2.5pre20211007_844dd90, but it still fails.
I have not updated the system nix yet (user-space command only), so this might explain the failure.
pandaman64
commented
2 years ago @andir I was unable to successfully run Nix against my repository for the following versions:
nix (Nix) 2.5pre20211007_844dd90 from nixos-unstable's nixUnstablenix (Nix) 2.4 from nixos-unstable's nixSo this issue is not fixed, I guess.
Fuuzetsu
commented
10 months ago I have ran into this in the wild today. I made a reproducer.
In the reproducer you I bundle a patched nix which allows us to further figure out what's happening. We can see that it gets stuck running in circles in the expandFeatures function in Cargo.nix. I attach a (compressed) trace.
You can debug what's going on like so:
diff --git a/nix/Cargo.nix b/nix/Cargo.nix
index 0edef57..ff00f34 100644
--- a/nix/Cargo.nix
+++ b/nix/Cargo.nix
@@ -5044,6 +5044,7 @@ rec {
mergePackageFeatures =
{ crateConfigs ? crates
, packageId
+ , depth ? 0
, rootPackageId ? packageId
, features ? rootFeatures
, dependencyPath ? [ crates.${packageId}.crateName ]
@@ -5063,8 +5064,12 @@ rec {
assert (builtins.isBool runTests);
let
crateConfig = crateConfigs."${packageId}" or (builtins.throw "Package not found: ${packageId}");
- expandedFeatures = expandFeatures (crateConfig.features or { }) features;
- enabledFeatures = enableFeatures (crateConfig.dependencies or [ ]) expandedFeatures;
+ dbgStr = d: s: if d == 0 then s else " ${dbgStr (d - 1) s}";
+ dbg = s: lib.debug.traceSeq (dbgStr depth s);
+ expandedFeatures_ = expandFeatures depth (crateConfig.features or { }) (dbg "Expanding for ${packageId} (${builtins.toString features})" features);
+ expandedFeatures = dbg "Expanded for ${packageId} (${builtins.toString (builtins.length expandedFeatures_)})" expandedFeatures_;
+ enabledFeatures_ = enableFeatures (crateConfig.dependencies or [ ]) (dbg "Enabling for ${packageId} (${builtins.toString (builtins.length expandedFeatures)})" expandedFeatures);
+ enabledFeatures = dbg "Enabled for ${packageId} (${builtins.toString (builtins.length enabledFeatures_)})" enabledFeatures_;
depWithResolvedFeatures = dependency:
let
packageId = dependency.packageId;
@@ -5093,6 +5098,7 @@ rec {
then cache
else
mergePackageFeatures {
+ depth = depth + 1;
features = combinedFeatures;
featuresByPackageId = cache;
inherit crateConfigs packageId target runTests rootPackageId;
@@ -5157,16 +5163,21 @@ rec {
featureMap is an attribute set which maps feature names to lists of further
feature names to enable in case this feature is selected.
*/
- expandFeatures = featureMap: inputFeatures:
+ expandFeatures = depth: featureMap: inputFeatures:
assert (builtins.isAttrs featureMap);
assert (builtins.isList inputFeatures);
let
+ dbgStr = d: s: if d == 0 then s else " ${dbgStr (d - 1) s}";
+ dbg = s: lib.debug.traceSeq (dbgStr depth s);
expandFeature = feature:
assert (builtins.isString feature);
- [ feature ] ++ (expandFeatures featureMap (featureMap."${feature}" or [ ]));
+ let
+ enables = featureMap."${feature}" or [ ];
+ in
+ (dbg "${feature} --> ${builtins.toString enables}") [ feature ] ++ (expandFeatures (depth + 1) featureMap enables);
outFeatures = lib.concatMap expandFeature inputFeatures;
in
- sortedUnique outFeatures;
+ /*dbg "Expanding ${builtins.toString inputFeatures} with ${builtins.toString (builtins.attrNames featureMap)}"*/ (sortedUnique outFeatures);
/* This function adds optional dependencies as features if they are enabled
indirectly by dependency features. This function mimics Cargo's behaviorIf you do this (being careful about nix laziness), you can spot a cycle like this:
temporal feature expands to the features to right of -->
temporal --> dtype-datetime dtype-date dtype-time dtype-duration polars-plan/temporalWe process the features in order, dtype-datetime expands to the stuff after
-->. Notably see that it expands to another temporal!
dtype-datetime --> polars-plan/dtype-datetime polars-time/dtype-datetime temporalpolars-plan/dtype-datetime expands to nothing
polars-plan/dtype-datetime --> polars-time/dtype-datetime expands to nothing
polars-time/dtype-datetime --> We make finally make it to temporal at the back of the dtype-datetime expansion and the cycle starts again!
temporal --> dtype-datetime dtype-date dtype-time dtype-duration polars-plan/temporalI'm considering what the right patch is. I tried patching it by making sure it doesn't go into a cycle but I failed and it didn't work properly. I'll try again but I thought I'd at least put out the information out there.
Fuuzetsu
commented
10 months ago The below patch should work I think. I'm able to apply this to our rather large tree and it fetches from the binary cache which implies that usual behaviour is completely unchanged. I'll try to cook up a patch.
diff --git a/nix/Cargo.nix b/nix/Cargo.nix
index c89bae956..44f8c5a19 100644
--- a/nix/Cargo.nix
+++ b/nix/Cargo.nix
@@ -36514,12 +36514,19 @@ rec {
assert (builtins.isAttrs featureMap);
assert (builtins.isList inputFeatures);
let
- expandFeature = feature:
- assert (builtins.isString feature);
- [ feature ] ++ (expandFeatures featureMap (featureMap."${feature}" or [ ]));
- outFeatures = lib.concatMap expandFeature inputFeatures;
- in
- sortedUnique outFeatures;
+ expandFeaturesNoCycle = oldSeen: inputFeatures:
+ if inputFeatures != []
+ then
+ let
+ feature = builtins.head inputFeatures;
+ tail = builtins.tail inputFeatures;
+ seen = oldSeen // { ${feature} = 1; };
+ enables = builtins.filter (f: !(seen ? "${f}")) (featureMap."${feature}" or []);
+ in [ feature ] ++ (expandFeaturesNoCycle seen (tail ++ enables))
+ # We're done.
+ else [];
+ outFeatures = expandFeaturesNoCycle { } inputFeatures;
+ in sortedUnique outFeatures;
/* This function adds optional dependencies as features if they are enabled
indirectly by dependency features. This function mimics Cargo's behavior
I have a similar issue to the one described in https://github.com/kolloch/crate2nix/issues/42, but I suspect the root cause is somewhat different. I am attempting to build a Rust project that makes extensive use of Intel SGX and the Rust SGX SDK. To use the Rust SGX SDK you have to use patched versions of common crates. These patched versions replace
stdwith the special SGX std version. When using some of these crates withcrate2nixI get stack-overflows.I can reproduce the issue in a few different ways, but I have put up the easiest minimal reproduction here: https://github.com/sphw/crate2nix-sgx-test/blob/main/Cargo.nix
When I run
nix-instantiate Cargo.nix -A rootCrate.build -vvvvvv --show-traceI get the following output:I'm not sure the root-cause of this issue. I have run the build with
ulimit -s unlimited, which causes the process to use all the remaining system memory.