opam pin --recursive fails to pin dune

josecastillolema commented 1 year ago

opam pin --recursive fails to pin dune package. Have not observed this behavior with other packets.

❯ opam pin --recursive .
Fatal error:
In packages/dune-3.10.0/test/blackbox-tests/test-cases/coq/compose-plugin.t/cplugin.opam:
unsupported or missing file format version; should be 2.0 or older

❯ opam --version
2.2.0~alpha2

❯ tree -L 2 .
.
├── _opam
│   ├── bin
│   ├── doc
│   ├── lib
│   ├── man
│   ├── sbin
│   └── share
├── packages
│   ├── cppo-1.6.9
│   ├── csexp-1.5.2
│   ├── dot-merlin-reader-4.9
│   ├── dune-3.10.0
│   ├── merlin-4.10-500
│   ├── merlin-lib-4.10-500
│   ├── ocaml-5.0.0
│   ├── ocamlfind-1.9.6
│   ├── seq-base
│   └── yojson-2.1.0
└── repo

19 directories, 1 file

rjbou commented 1 year ago

Dune contains some opam files in its tests, and they are detected by the recursive pin. But 2 of them are empty files for test purpose.

> opam pin scan git+https://github.com/ocaml/dune --rec
[ERROR] In test/blackbox-tests/test-cases/coq/ml-lib.t/ml_lib.opam:
        unsupported or missing file format version; should be 2.0 or older [skipped]

[ERROR] In test/blackbox-tests/test-cases/coq/compose-plugin.t/cplugin.opam:
        unsupported or missing file format version; should be 2.0 or older [skipped]

# Name              # Version  # Url                              # Subpath
mypackage           -          git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/public_name-exe.t
baz                 0.1.0      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lockfile-generation.t/mock-opam-repository/packages/baz/baz.0.1.0
baz                 0.0.1      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lockfile-generation.t/mock-opam-repository/packages/baz/baz.0.0.1
bar                 0.5.0      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lockfile-generation.t/mock-opam-repository/packages/bar/bar.0.5.0
bar                 0.4.0      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lockfile-generation.t/mock-opam-repository/packages/bar/bar.0.4.0
bar                 0.0.1      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lockfile-generation.t/mock-opam-repository/packages/bar/bar.0.0.1
foo                 0.0.1      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lock-per-context.t/mock-opam-repository/packages/foo/foo.0.0.1
baz                 0.1.0      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lock-per-context.t/mock-opam-repository/packages/baz/baz.0.1.0
baz                 0.0.1      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lock-per-context.t/mock-opam-repository/packages/baz/baz.0.0.1
bar                 0.5.0      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lock-per-context.t/mock-opam-repository/packages/bar/bar.0.5.0
bar                 0.4.0      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lock-per-context.t/mock-opam-repository/packages/bar/bar.0.4.0
bar                 0.0.1      git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/pkg/lock-per-context.t/mock-opam-repository/packages/bar/bar.0.0.1
lib2                -          git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/github568.t
lib1                -          git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/github568.t
dune_inline_tests   -          git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/github1549.t/backend
ml_lib              -          git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/coq/ml-lib.t
cplugin             -          git+https://github.com/ocaml/dune  test/blackbox-tests/test-cases/coq/compose-plugin.t
plugin1             -          git+https://github.com/ocaml/dune  otherlibs/dune-site/test/plugin_require_thread.t/plugin
myproject           1.0        git+https://github.com/ocaml/dune  example/with-configure-step.t
hello_world         1.0        git+https://github.com/ocaml/dune  example/hello_world.t
xdg                 -          git+https://github.com/ocaml/dune  -
stdune              -          git+https://github.com/ocaml/dune  -
ordering            -          git+https://github.com/ocaml/dune  -
ocamlc-loc          -          git+https://github.com/ocaml/dune  -
dyn                 -          git+https://github.com/ocaml/dune  -
dune                -          git+https://github.com/ocaml/dune  -
dune-site           -          git+https://github.com/ocaml/dune  -
dune-rpc            -          git+https://github.com/ocaml/dune  -
dune-rpc-lwt        -          git+https://github.com/ocaml/dune  -
dune-private-libs   -          git+https://github.com/ocaml/dune  -
dune-glob           -          git+https://github.com/ocaml/dune  -
dune-configurator   -          git+https://github.com/ocaml/dune  -
dune-build-info     -          git+https://github.com/ocaml/dune  -
dune-action-plugin  -          git+https://github.com/ocaml/dune  -
chrome-trace        -          git+https://github.com/ocaml/dune  -

You can use pin scan to select/unselect some paths opam pin scan url --normalise | grep -v <path> | xargs opam pin

kit-ty-kate commented 1 year ago

What is your goal by doing that? It looks like you've created an opam repository structure (repo file, packages directory) but not using it as a repository, rather a storage place for vendored dependencies (which in the case of the packages you've added seems to be rather complicated to setup so I'm not sure you want that)

If you want an opam repository instead of vendored dependencies you can replace the vendored subdirectories by opam files (taken from https://github.com/ocaml/opam-repository/ for example) and call opam repository add your-local-repo .

josecastillolema commented 1 year ago

Thanks! Will look into that. And as there is a proposed workaround will close this issue.

josecastillolema commented 1 year ago

ps. As for the goal question, I am trying to package ocaml (with some dependencies) into a FlatPak application. For that, in the first step I need to download all the sources, and in the second step to build them (the second step does not have Internet access). Any suggestions will be welcomed.

rjbou commented 1 year ago

You can have an opam repository with only selected packages. Then run opam admin cache at root directory, it will download all repository archives. In your repo file, add the field archive-mirrors: cache. From there, you have all archives, you can work offline (if there is no extra-sources defined in selected opam files). You initialise opam with that repository opam init <path/to/repository> (or add it via opam repository add), create a switch and install the packages. By default, opam first retrieves repository cached sources.

josecastillolema commented 1 year ago

So @rjbou unfortunately to the best of my knowledge that approach wont work for flatpak packaging. There are two phases when building flatpak packages:

The download phase where you can download any file from the Internet but you cannot run any commands (i.e. you wont be able to run opam admin cache)
The build phase where you are able to run any command but you cannot access the Internet

The way I am trying to approach the problem is:

Developed a tool called flatpak-opam-generator that generates a flatpak builder source files with all the .tars of the dependencies of a given package
Then during the build phase I pin those folders

This approach is not ideal, if you have any better alternatives please let me know.

Thanks!

ocaml / opam

opam pin --recursive fails to pin dune #5658