[Bug]: `devbox add <pkg>` fails because `nix profile list` needs `--extra-experimental-features`

[philipp-baumann]

Current Behavior (bug) devbox add python3: getting error .devbox/nix/profile/default/manifest.json' has unsupported version 2

Expected Behavior (fix) Install should not fail and devbox add should fall back to --extra-experimental-features when calling nix profile list

Additional context

$ devbox version -v
2023/08/18 12:24:55 findProjectDir: path is 
2023/08/18 12:24:55 finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics
2023/08/18 12:24:55 findProjectDir: path is 
2023/08/18 12:24:55 finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics
[DEBUG] 2023/08/18 12:24:55 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:51: Debug mode enabled.
Version:     0.5.11
Platform:    linux_amd64
Commit:      1052c4e37c16ce0df5d60afb8c82076cd290eee8
Commit Time: 2023-08-10T23:32:09Z
Go Version:  go1.20.7
Launcher:    0.2.0
[DEBUG] 2023/08/18 12:24:55 go.jetpack.io/devbox/internal/impl/dir.go:22: findProjectDir: path is 
[DEBUG] 2023/08/18 12:24:55 go.jetpack.io/devbox/internal/impl/dir.go:66: finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics

cat devbox.json
{
  "packages": [],
  "shell": {
    "init_hook": [
      "echo 'Welcome to devbox!' > /dev/null"
    ],
    "scripts": {
      "test": [
        "echo \"Error: no test specified\" && exit 1"
      ]
    }
  }
}

[philipp-baumann]

more details:

$ devbox add python3
2023/08/18 12:19:01 findProjectDir: path is 
2023/08/18 12:19:01 finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics
2023/08/18 12:19:01 findProjectDir: path is 
2023/08/18 12:19:01 finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics
[DEBUG] 2023/08/18 12:19:01 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:51: Debug mode enabled.
[DEBUG] 2023/08/18 12:19:01 go.jetpack.io/devbox/internal/impl/dir.go:22: findProjectDir: path is 
[DEBUG] 2023/08/18 12:19:01 go.jetpack.io/devbox/internal/impl/dir.go:66: finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics
[DEBUG] 2023/08/18 12:19:02 go.jetpack.io/devbox/internal/nix/search.go:95: running command: /usr/bin/nix search --json github:NixOS/nixpkgs/3007746b3f5bfcb49e102b517bca891822a41b31#python3 --extra-experimental-features ca-derivations --option experimental-features nix-command flakes

Error: There was an error installing nix packages
source: error running "nix profile list": exit status 1

[DEBUG] 2023/08/18 12:19:02 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:72: Command stderr: error: profile manifest '/home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics/.devbox/nix/profile/default/manifest.json' has unsupported version 2

[DEBUG] 2023/08/18 12:19:02 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:74: 
ExecutionID:4b7b23f7941e42ee89650ab6a7be8301
error running "nix profile list": exit status 1
go.jetpack.io/devbox/internal/nix.ProfileList
        go.jetpack.io/devbox/internal/nix/profiles.go:26
go.jetpack.io/devbox/internal/nix/nixprofile.ProfileListItems
        go.jetpack.io/devbox/internal/nix/nixprofile/profile.go:65
go.jetpack.io/devbox/internal/impl.(*Devbox).pendingPackagesForInstallation
        go.jetpack.io/devbox/internal/impl/packages.go:380
go.jetpack.io/devbox/internal/impl.(*Devbox).addPackagesToProfile
        go.jetpack.io/devbox/internal/impl/packages.go:267
go.jetpack.io/devbox/internal/impl.(*Devbox).syncPackagesToProfile
        go.jetpack.io/devbox/internal/impl/packages.go:241
go.jetpack.io/devbox/internal/impl.(*Devbox).ensurePackagesAreInstalled
        go.jetpack.io/devbox/internal/impl/packages.go:199
go.jetpack.io/devbox/internal/impl.(*Devbox).Add
        go.jetpack.io/devbox/internal/impl/packages.go:101
go.jetpack.io/devbox/internal/boxcli.addCmdFunc
        go.jetpack.io/devbox/internal/boxcli/add.go:68
go.jetpack.io/devbox/internal/boxcli.addCmd.func1
        go.jetpack.io/devbox/internal/boxcli/add.go:42
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.6.1/command.go:968
go.jetpack.io/devbox/internal/boxcli/midcobra.(*midcobraExecutable).Execute
        go.jetpack.io/devbox/internal/boxcli/midcobra/midcobra.go:61
go.jetpack.io/devbox/internal/boxcli.Execute
        go.jetpack.io/devbox/internal/boxcli/root.go:103
go.jetpack.io/devbox/internal/boxcli.Main
        go.jetpack.io/devbox/internal/boxcli/root.go:124
main.main
        ./main.go:11
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598
[DEBUG] 2023/08/18 12:19:02 go.jetpack.io/devbox/internal/impl/dir.go:22: findProjectDir: path is 
[DEBUG] 2023/08/18 12:19:02 go.jetpack.io/devbox/internal/impl/dir.go:66: finding devbox.json in dir: /home/philipp/git/spectral-cockpit/renku/applied-infrared-spectroscopic-modeling-workshop-for-soil-and-plant-diagnostics

[savil]

@philipp-baumann thanks for filing the issue. This seems to work for me locally. I'm wondering if there's a bug with how Devbox handles some local state/state-transition in your scenario.

A couple of questions:

1. What is your nix version? Did you recently update your local nix?
2. To unblock yourself, could you run rm -rf .devbox and try again? It will claim to "re-install" your existing packages too, but it should be decently fast because those are already present in your local /nix/store.

[savil]

I'm curious why you said "should fall back to --extra-experimental-features"?

[philipp-baumann]

@philipp-baumann thanks for filing the issue. This seems to work for me locally. I'm wondering if there's a bug with how Devbox handles some local state/state-transition in your scenario.

A couple of questions:

1. What is your nix version? Did you recently update your local nix?
2. To unblock yourself, could you run rm -rf .devbox and try again? It will claim to "re-install" your existing packages too, but it should be decently fast because those are already present in your local /nix/store.

thanks @savil for the quick reply! :-) actually

1. nix (Nix) 2.17.0
2. that really helped. I have no clue what corrupted my setup. Happy it all works now.

[philipp-baumann]

I'm curious why you said "should fall back to --extra-experimental-features"?

that is just what I got from the devbox debug report. That was a bit too assertive from my side.

[philipp-baumann]

So the solution was to remove .devbox folder.

[savil]

Got it. Glad you are unblocked. Deleting .devbox folder is a sledgehammer workaround; we'd really like users to never have to do that.

Had you recently upgraded nix? I'm wondering if you'd previously added packages using an older version of nix which created the nix profile manifest that the newer version of nix is failing to read.

[philipp-baumann]

Got it. Glad you are unblocked. Deleting .devbox folder is a sledgehammer workaround; we'd really like users to never have to do that.

Had you recently upgraded nix? I'm wondering if you'd previously added packages using an older version of nix which created the nix profile manifest that the newer version of nix is failing to read.

Fair point about smoothness, like that philosophy. Haven't upgraded nix, I used Determinate Systems installer once. I think I have switched between two R nix revisions added trough devbox (found on nixhub.io), guess that should not be a problem? Is there anything else I can help that would help diagnosing what has happened or to reproduce, if it is not that random?

[savil]

I think I have switched between two R nix revisions added trough devbox (found on nixhub.io), guess that should not be a problem?

Nope, that should be totally okay...

Don't have any bright leads as of now, but will drop a line in case something strikes me!

[philipp-baumann]

Great, thank you!