search

tweag / opam-nix

Turn opam-based OCaml projects into Nix derivations
MIT License
111 stars 33 forks source link

Fails to find repositories specified in `pin-depends` #1

Closed brendanzab closed 2 years ago

brendanzab commented 2 years ago

Describe the bug Opam-nix seems not to look for pin-depends.

I've been trying to experiment with yocaml. It seems not to have been published to opam yet, and people seem to be using pin-depends as a way to use it in their projects?

I searched on the opam-nix repo and it seems like pin-depends is not mentioned, so maybe it is not yet supported? I understand that it might be tricky to do this while maintaining reproducibility however? So I'm not sure what that would look like.

To Reproduce

flake.nix

{
  description = "A very basic flake";

  inputs = {
    # Utilities for writing flakes
    flake-utils.url = github:numtide/flake-utils;
    # Precisely filter files copied to the nix store
    nix-filter.url = "github:numtide/nix-filter";
    # Generate derivations from Dune and OCaml files
    opam-nix.url = github:tweag/opam-nix;
  };

  outputs = { self, nixpkgs, flake-utils, nix-filter, opam-nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # Legacy packages that have not been converted to flakes
        legacyPackages = nixpkgs.legacyPackages.${system};
        # OCaml packages available on nixpkgs
        ocamlPackages = legacyPackages.ocamlPackages;

        # OCaml source files
        ocaml-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            "dune-project"
            "garden.opam.template"
            (nix-filter.lib.inDirectory "bin")
          ];
        };

        # Nix source files
        nix-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            (nix-filter.lib.matchExt "nix")
          ];
        };
      in
      {
        # Executed by `nix build .#<name>`
        packages.garden =
          (opam-nix.lib.${system}.buildDuneProject { } "garden" ocaml-src { }).garden;

        # Executed by `nix build`
        defaultPackage = self.packages.${system}.garden;

        # Executed by `nix run .#<name>`
        apps.garden = {
          type = "app";
          program = "${self.packages.${system}.garden}/bin/garden";
        };

        # Executed by `nix run`
        defaultApp = self.apps.${system}.garden;

        # Used by `nix develop`
        devShell = legacyPackages.mkShell {
          nativeBuildInputs = [
            ocamlPackages.dune_2
            ocamlPackages.ocaml
            legacyPackages.opam

            # Editor support
            # pkgs.ocamlformat # FIXME: fails to build `uunf` on my M1 mac :(
            ocamlPackages.merlin
            ocamlPackages.ocaml-lsp
          ];
          inputsFrom = [
            self.defaultPackage.${system}
          ];
        };
      });
}

dune-project

(lang dune 2.9)

(generate_opam_files true)

(package
 (name garden)
 (synopsis "Static page generator for my digital garden")
 (depends
  (preface (>= "0.1.0"))
  (yocaml :pinned)
  (yocaml_yaml :pinned)
  (yocaml_markdown :pinned)
  (yocaml_unix :pinned)))

bin/dune

(executable
 (name garden)
 (public_name garden)
 (promote (until-clean))
 (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))

bin/garden.ml

let () =
  ()

garden.opam.template

pin-depends: [
  ["yocaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_unix.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_yaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_markdown.dev" "git://github.com/xhtmlboi/yocaml.git"]
]

After running nix build I get:

nix log ...-garden-local.drv

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/l2y5prikdgks7i4a5awc7vkxdxzxi2h9-source-copy
source root is source-copy
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
patching script interpreter paths in .
@nix { "action": "setPhase", "phase": "buildPhase" }
building
File "bin/dune", line 5, characters 55-66:
5 |  (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))
                                                           ^^^^^^^^^^^
Error: Library "yocaml_unix" not found.
Hint: try:
  dune external-lib-deps --missing --no-config --root . --ignore-promoted-rules --default-target @install --always-show-command-line --promote-install-files false --promote-install-files --release --only-packages garden -p garden --profile release -j 10 @install

The generated opam file is:

/nix/store/...-source-copy/garden.opam

# This file is generated by dune, edit dune-project instead
opam-version: "2.0"
depends: [
  "dune" {>= "2.9"}
  "preface" {>= "0.1.0"}
  "yocaml" {pinned}
  "yocaml_yaml" {pinned}
  "yocaml_markdown" {pinned}
  "yocaml_unix" {pinned}
  "odoc" {with-doc}
]
build: [
  ["dune" "subst"] {dev}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "--promote-install-files=false"
    "@install"
    "@runtest" {with-test}
    "@doc" {with-doc}
  ]
  ["dune" "install" "-p" name "--create-install-files" name]
]
pin-depends: [
  ["yocaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_unix.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_yaml.dev" "git://github.com/xhtmlboi/yocaml.git"]
  ["yocaml_markdown.dev" "git://github.com/xhtmlboi/yocaml.git"]
]

Expected behavior

The build should find the dependencies

Environment

Additional context Add any other context about the problem here.

balsoft commented 2 years ago

It's indeed not possible to support pin-depends reproducibly without an opam lockfile (since it doesn't have any notion of actually pinning the dependency version in the package description). It can probably be supported with --impure in some way.

For now, you can try passing yocaml as a repo. First, add an input to your flake:

  yocaml = {
    url = "github:xhtmlboi/yocaml";
    flake = false;
  };

Then, pass it (together with the default opam-repository) to the buildDuneProject:

      packages.garden = (with opam-nix.lib.${system}; buildDuneProject {
        repos = [
          (makeOpamRepo yocaml)
          opam-nix.inputs.opam-repository
        ];
      } "garden" ocaml-src { }).garden;
balsoft commented 2 years ago

opam-nix now supports pin-depends, but in your particular case it won't work because yocaml has a non-master default branch. See https://github.com/NixOS/nix/issues/5139 and https://github.com/NixOS/nix/issues/4456

brendanzab commented 2 years ago

Oh nice, thanks!

Hmm for some reason the suggestion above doesn't seem to be working for me?

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/l2y5prikdgks7i4a5awc7vkxdxzxi2h9-source-copy
source root is source-copy
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
patching script interpreter paths in .
@nix { "action": "setPhase", "phase": "buildPhase" }
building
File "bin/dune", line 5, characters 55-66:
5 |  (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))
                                                           ^^^^^^^^^^^
Error: Library "yocaml_unix" not found.
Hint: try:
  dune external-lib-deps --missing --no-config --root . --ignore-promoted-rules --default-target @install --always-show-command-line --promote-install-files false --promote-install-files --release --only-packages garden -p garden --profile release -j 10 @install
balsoft commented 2 years ago

Ah, yeah, sorry. I forgot to makeOpamRepo and also opam doesn't resolve pinned packages with opam admin list, so we have to ask for those ourselves:

        packages.garden = (with opam-nix.lib.${system};
          buildDuneProject {
            pinDepends = false;
            repos = [ (makeOpamRepo yocaml) opam-nix.inputs.opam-repository ];
          } "garden" ocaml-src {
            yocaml = null;
            yocaml_unix = null;
            yocaml_yaml = null;
            yocaml_markdown = null;
          }).garden;

(Note that you'll also need the latest version of opam-nix since there was an overlay issue)

brendanzab commented 2 years ago

Wooo, I think that worked! Thanks a bunch. Here's the full flake I made if it's of any use to anyone:

{
  description = "Static page generator for my digital garden";

  inputs = {
    # Utilities for writing flakes
    flake-utils.url = github:numtide/flake-utils;
    # Precisely filter files copied to the nix store
    nix-filter.url = "github:numtide/nix-filter";
    # Generate derivations from Dune and OCaml files
    opam-nix.url = github:tweag/opam-nix;
    # Static site generator for OCaml (not yet published to opam)
    yocaml.url = "github:xhtmlboi/yocaml";
    yocaml.flake = false;
  };

  outputs = { self, nixpkgs, flake-utils, nix-filter, opam-nix, yocaml }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # Legacy packages that have not been converted to flakes
        legacyPackages = nixpkgs.legacyPackages.${system};
        # OCaml packages available on nixpkgs
        ocamlPackages = legacyPackages.ocamlPackages;

        # OCaml source files
        ocaml-src = nix-filter.lib.filter {
          root = ./.;
          include = [
            "dune-project"
            "garden.opam.template"
            (nix-filter.lib.inDirectory "bin")
          ];
        };
      in
      {
        # Executed by `nix build .#<name>`
        packages.garden =
          (opam-nix.lib.${system}.buildDuneProject
            {
              pinDepends = false;
              repos = [
                (opam-nix.lib.${system}.makeOpamRepo yocaml)
                opam-nix.inputs.opam-repository
              ];
            }
            "garden"
            ocaml-src
            {
              yocaml = null;
              yocaml_unix = null;
              yocaml_yaml = null;
              yocaml_markdown = null;
            }).garden;

        # Executed by `nix build`
        defaultPackage = self.packages.${system}.garden;

        # Executed by `nix run .#<name>`
        apps.garden = {
          type = "app";
          program = "${self.packages.${system}.garden}/bin/garden";
        };

        # Executed by `nix run`
        defaultApp = self.apps.${system}.garden;

        # Used by `nix develop`
        devShell = legacyPackages.mkShell {
          nativeBuildInputs = [
            # Editor support
            legacyPackages.fswatch # for `dune build --watch --terminal-persistence=clear-on-rebuild`
            # legacyPackages.ocamlformat # FIXME: fails to build `uunf` on aarch64-darwin :(
            ocamlPackages.merlin
            ocamlPackages.ocaml-lsp
          ];
          inputsFrom = [
            self.defaultPackage.${system}
          ];
        };
      });
}
brendanzab commented 2 years ago

Ahh wierd, for some reason when I do dune build --watch --terminal-persistence=clear-on-rebuild using the above nix file, builds a bin/garden.exe, rather than putting it in the _build directory?

brendanzab commented 2 years ago

Ahh wait, I think that must be because of the recommended dune file:

(executable
 (name garden)
 (public_name garden)
 (promote (until-clean)) ; <-- this is the culprit!
 (libraries preface yocaml yocaml_yaml yocaml_markdown yocaml_unix))
balsoft commented 2 years ago

Thanks to work of @rizo now non-master main branches are also supported! E.g. this works: https://github.com/tweag/opam-nix/blob/main/examples/readme/my-package/my-package.opam#L16

brendanzab commented 2 years ago

Should I be looking at https://github.com/tweag/opam-nix#examples-8 for this?

balsoft commented 2 years ago

No, it should just work with buildOpamProject and buildDuneProject now.

brendanzab commented 2 years ago

So I switched on pinDepends = true:

flake.nix ```nix { description = "Static page generator for my digital garden"; inputs = { # Utilities for writing flakes flake-utils.url = github:numtide/flake-utils; # Precisely filter files copied to the nix store nix-filter.url = "github:numtide/nix-filter"; # Generate derivations from Dune and OCaml files opam-nix.url = github:tweag/opam-nix; # Static site generator for OCaml (not yet published to opam) yocaml.url = "github:xhtmlboi/yocaml"; yocaml.flake = false; }; outputs = { self, nixpkgs, flake-utils, nix-filter, opam-nix, yocaml }: flake-utils.lib.eachDefaultSystem (system: let # Legacy packages that have not been converted to flakes legacyPackages = nixpkgs.legacyPackages.${system}; # OCaml packages available on nixpkgs ocamlPackages = legacyPackages.ocamlPackages; # OCaml source files ocaml-src = nix-filter.lib.filter { root = ./.; include = [ "dune-project" "garden.opam.template" (nix-filter.lib.inDirectory "bin") ]; }; # Nix source files nix-src = nix-filter.lib.filter { root = ./.; include = [ (nix-filter.lib.matchExt "nix") ]; }; in { # Used for `nixpkgs` packages, also accessible via `nix build .#` legacyPackages = (opam-nix.lib.${system}.buildDuneProject { pinDepends = true; repos = [ opam-nix.inputs.opam-repository (opam-nix.lib.${system}.makeOpamRepo yocaml) ]; } "garden" ocaml-src { # Include `yocaml` packages yocaml = null; yocaml_markdown = null; yocaml_mustache = null; yocaml_unix = null; yocaml_yaml = null; # Ask `opam-nix `to include editor tools in the same scope as our # package, ensuring that they will be built using the same # compiler version etc. merlin = null; ocaml-lsp-server = null; }); # Executed by `nix build` defaultPackage = self.legacyPackages.${system}.garden; # Executed by `nix run .#` apps.garden = { type = "app"; program = "${self.legacyPackages.${system}.garden}/bin/garden"; }; # Executed by `nix run` defaultApp = self.apps.${system}.garden; # Used by `nix develop` devShell = legacyPackages.mkShell { nativeBuildInputs = [ # Editor support legacyPackages.fswatch # for `dune build --watch --terminal-persistence=clear-on-rebuild` # legacyPackages.ocamlformat # FIXME: fails to build `uunf` on aarch64-darwin :( self.legacyPackages.${system}.merlin self.legacyPackages.${system}.ocaml-lsp-server ]; inputsFrom = [ self.defaultPackage.${system} ]; }; }); } ```

but now I am getting warnings like: 

trace: warning: pin-depends without an explicit sha1 is not supported in pure evaluation mode; try with --impure

I also get a build error:

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/4d0ix5djms3n2jnjdc58l916cwack1rp-empty-directory
source root is empty-directory
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
patching script interpreter paths in .
cp: cannot create regular file '/tmp/opam-subst': Permission denied
brendanzab commented 2 years ago

Oh, so I need to set a sha in .opam.template? (sorry, I was expecting it to be ok, given I had supplied the flake input)

brendanzab commented 2 years ago

I'm wondering if it's possible to use an opam lockfile to simplify this kind of thing? This is what naersk does, but I'm not too clear on the specifics of how opam works to know if it's possible here too… eg. I'm not sure how to generate the lockfile in the first place. I'm guessing it might require opam to be installed globally?

balsoft commented 2 years ago

but now I am getting warnings like:

Well, the warning tells you exactly what to do: run your command again with --impure, or set the git commit sha1 (a.k.a rev) in the opam file (like done in the example: https://github.com/tweag/opam-nix/blob/main/examples/readme/my-package/my-package.opam#L14). To be clear, this is also supported by opam, and is a good idea for reproducibility purposes regardless of opam-nix.

I also get a build error:

That error looks weird, I'll look into it

brendanzab commented 2 years ago

Well, the warning tells you exactly what to do

Yeah, I didn't want to run it in --impure, as I thought this was the point of flakes: offering isolation. I also couldn't tell from the warning where to set the sha - I was looking for somewhere in the nix file, not the .opam.template.

To be clear, this is also supported by opam, and is a good idea for reproducibility purposes regardless of opam-nix.

Ahh yeah, this might be me confused by opam's model… (which I find confoundingly frustrating at the best of times). Coming from Cargo git dependencies are automatically pinned in a lockfile - ie. the 'correct way' is enabled by default. I'll go ahead and add the sha to the pin-depends URLs, but it is kind of frustrating as it loses information on what branch it is supposed to be tracking, and can't be updated with tooling. :(

Sorry again for all my confusion!

brendanzab commented 2 years ago

Could it be that there is some weirdness in how opam-nix is understanding my pinned deps?

pin-depends: [
  # The sha1 hashes currently point to the main branch of the yocaml repository
  ["yocaml.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_markdown.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_mustache.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_unix.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
  ["yocaml_yaml.dev" "git+https://github.com/xhtmlboi/yocaml.git#167cddf111bec1095fc35a10f5c10d2d25e9adbd"]
]
balsoft commented 2 years ago

Certainly possible :)

What is the exact problem you're experiencing?

I've just checked and that snippet does what I would expect. Are you on latest opam-nix?