Package from inventory of type archive somehow evaluates elpa.nix on CI

[terlar]

This is a really strange one. I ran into the issue of this working locally, but not on CI.

It somehow finds the package in elpa (I guess, since it enters elpa.nix) on CI, but not locally. So the locked file is not correct. Locally the package is not found in the top inventories so it reaches the archive and hence when running nix run .#lock and nix run .#update it generates the archive.lock instead of adding it to the flake.lock.

If I put the archive above GNU ELPA it works on CI as well.

I have the current defined inventory:

[
  {
    name = "custom";
    type = "melpa";
    path = ../recipes;
  }
  {
    name = "gnu";
    type = "elpa";
    path = self.inputs.gnu-elpa.outPath + "/elpa-packages";
    core-src = emacsSrc;
    auto-sync-only = true;
  }
  {
    name = "melpa";
    type = "melpa";
    path = self.inputs.melpa.outPath + "/recipes";
  }
  {
    name = "nongnu";
    type = "elpa";
    path = self.inputs.nongnu-elpa.outPath + "/elpa-packages";
  }
  {
    name = "gnu-archive";
    type = "archive";
    url = "https://elpa.gnu.org/packages/";
  }
]

and archive.lock:

{
...
  "jinx": {
    "archive": {
      "narHash": "sha256-NA2ulVZuhetCYxwvUgOv8nDAdC9LYFDpj7O42pD/wyI=",
      "type": "tarball",
      "url": "https://elpa.gnu.org/packages/jinx-0.6.tar"
    },
    "inventory": {
      "type": "archive",
      "url": "https://elpa.gnu.org/packages/"
    },
    "packageRequires": {
      "compat": "29.1.4.0",
      "emacs": "27.1"
    },
    "version": "0.6"
  },
...
}

Here is the CI build: https://github.com/terlar/emacs-config/actions/runs/4751840604/jobs/8441454659 and the source commit: https://github.com/terlar/emacs-config/tree/0f1615ddb22b81d9f61a22166cbe95e8a8819fcf

Did you run into something like this?

[terlar]

I think the issue stems from elpa actually having this package, but not in the elpa I have in my lock-file.

I am currently not able to update elpa as eglot package build is broken in the later versions, not sure if you've run into that and found a solution:

$ nix build
warning: Git tree '/home/terje.larsen/src/github.com/terlar/emacs-config' is dirty
error: builder for '/nix/store/gsfjrx3kin5if637lw0720qjr39ia31p-emacs-pgtk-20230317-0-eglot-1.13.drv' failed with exit code 1;
       last 10 log lines:
       > unpacking source archive /nix/store/xinkqs5cd540w4p238jisshd70z9mfh2-source
       > source root is /build/build
       > building
       > Package autoload is deprecated
       >   SCRAPE   .
       >   INFO     Scraping files for eglot-autoloads.el...
       >   INFO     Scraping files for eglot-autoloads.el...done
       > gzip.texi:81: @include: could not find gpl.texinfo
       > gzip.texi:471: warning: @refill is obsolete
       > /nix/store/pw17yc3mwmsci4jygwalj8ppg0drz31v-stdenv-linux/setup: line 136: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/gsfjrx3kin5if637lw0720qjr39ia31p-emacs-pgtk-20230317-0-eglot-1.13.drv'.

[akirak]

@terlar Do you have any reason for installing eglot from ELPA? Since you are using the Git version of Emacs, eglot is now built-in.

The package is now even a core package (as you can see in elpa-packages), so they are basically built from the same source.

If you switch to the built-in version of eglot and update elpa input in your flake.nix, your config should successfully build.

If you still have to install eglot from ELPA, I suggest you install eglot from the archive. Here are a few option how to achieve it:

(a). Ignore the eglot entry in the ELPA repository

    {
      name = "gnu";
      type = "elpa";
      path = self.inputs.gnu-elpa.outPath + "/elpa-packages";
      core-src = emacsSrc;
      auto-sync-only = true;
      exclude = [
        "eglot"
      ];
    }

(b). Explicitly specify the registry of eglot package using :pin keyword. This is supported by twist for use-package:

  (use-package eglot
    :ensure t
    :pin gnu-archive)

[terlar]

Indeed, I thought it would automatically exclude, but that makes sense, thanks for the tip. This was a side-note from this issue though. But this issue is not an issue for me when updating the packages. However, something seems wrong with that behavior doesn't it? The case where it tries to pull jinx from elpa on my CI, while it locally uses the archive?

Edit, I realized what you meant, so I removed the :ensure t config that made it pull the package from the inventories instead. I should probably go through my list of packages to exclude any other packages that might be bundled with Emacs nowadays.

[akirak]

@terlar I see, you had already resolved the issue by yourself.

Edit, I realized what you meant, so I removed the :ensure t config that made it pull the package from the inventories instead. I should probably go through my list of packages to exclude any other packages that might be bundled with Emacs nowadays.

Yes, twist allows you to override built-in packages by explicitly installing them. This is because you have to rebuild all packages every time you update Emacs, but you may sometimes want to install the latest version of a particular package for a particular feature/bugfix or for development.

[akirak]

Should probably go through all my packages to check which ones are builtin now.

Maybe I should provide a better interface for checking the statuses of packages in a config.

[akirak]

I have created a discussion thread (#113) to address the UX issue.

[akirak]

It somehow finds the package in elpa (I guess, since it enters elpa.nix) on CI, but not locally. So the locked file is not correct. Locally the package is not found in the top inventories so it reaches the archive and hence when running nix run .#lock and nix run .#update it generates the archive.lock instead of adding it to the flake.lock.

One possibility is that the package source is already in the Nix store on your local machine but not on CI. The Nix store is not completely synced, which can cause inconsistencies in your build. I am not sure what is happening though. Please report if you encounter a similar situation again.

[terlar]

I think you might be right. That package was probably added recently and locally I built before that happened.

Okay, I will close in the meantime. I tried to re-build, but not sure which derivation is the one contained the stale data/inconsistencies, so not sure what to rebuild.

Thank you for the support.