opam (ocaml package manager) front end

pmetzger commented 5 years ago

opam is the package manager for the OCaml ecosystem. Almost all OCaml libraries and programs have associated opam files.

Generally, it's better for people working in the ocaml ecosystem to use opam for their day to day operations, but if you're building a package that is written in ocaml in macports, you probably don't care and don't want to care about opam and the opam ecosystem. We thus have to maintain a bunch of packages that are also packaged by opam, and it's painful. Automation would help a lot.

quick overview of the opam format

opam file description

repository of opam files used by the opam system

typical opam file for the zarith package

my portfile for zarith, not really made from the opam file

Steap commented 5 years ago

I tried writing a frontend for OPAM, but my limited experience with parsers made this difficult :-/ It would also have required the frontend to keep the opam git repo up to date.

Could we parse this page instead? It is much simpler, even though I usually don't like relying on Web pages. Does opam.ocaml.org give enough data to be helpful?

pmetzger commented 5 years ago

I don't think going off the web page is the right thing. It doesn't even contain the information on how to build the package! The right way to do this is for me to build a small tool (in ocaml) that uses the opam file parser library itself to extract the data you want from the opam file.

@reneeotten and you and I can figure out the correct way to do all this.

Steap commented 5 years ago

I don't think going off the web page is the right thing. It doesn't even contain the information on how to build the package! The right way to do this is for me to build a small tool (in ocaml) that uses the opam file parser library itself to extract the data you want from the opam file.

So to be honest, upt does not currently provide a way to store package-specific build instructions. Is there no (de facto) standard in OCaml? This is something that could probably be added - but we're gonna need some patience.

If the tool is written in OCaml, it is going to be a bit painful to package it along the new frontend. Do you know the people behind opam? One thing that could be nice is to have a JSON/YAML/... file online that would show the interesting metadata. It could be generated by an OCaml tool. People would then be able to fetch it at https://opam.ocaml.org/packages/coccinelle.json. What do you think?

pmetzger commented 5 years ago

So to be honest, upt does not currently provide a way to store package-specific build instructions. Is there no (de facto) standard in OCaml? This is something that could probably be added - but we're gonna need some patience.

MacPorts has package specific build instructions of course.

If the tool is written in OCaml, it is going to be a bit painful to package it along the new frontend. Do you know the people behind opam?

Yes. But of course, since they are ocamlers, they wrote the parsing tools for the format in ocaml. They're unlikely to provide an alternate format for opam file data, but it would be easy for us to privately generate other formats as an intermediate step.

mojca commented 5 years ago

@pmetzger: what would really help would be:

Creating a simple working example of an ocaml port with some dependencies (or rather a few of them).
If you create a tool: could you convince that tool to output package description in something like json or yaml? It needs to contain name, version, URL from which the sources can be fetched, dependencies, ... or any other metainformation that would be relevant for building a package. Ignore upt for a moment, just try to extract the relevant information in a nice way.
Ideally the next step would be to convince upstream to setup a service offering such information, but if you come up with a prototype, they might be willing to continue from there.

You should be able to come up with a simple easily parsable standard text format with sufficient information to build a package. If this is not feasible, creating macports packages will also be anything but trivial.

pmetzger commented 5 years ago

The opam file is already an easily parsable standard text format with sufficient information to build a package. It contains most of the same information that MacPorts' Portfiles contain. I linked above to the MacPorts zarith port and to the corresponding opam file, FWIW. Note that I'm willing to write code to do much of the work here. I could probably get to it later in the week.

Korusuke commented 5 years ago

So from what I understand we will have to keep a local copy of git repo or get the latest opam files using github api which can be easily done eliminating the need for keeping the local repo updated.

Next is parsing, as @pmetzger said there is a parser available in ocaml, so we can either directly use that executing a sys command like we did for port info --version in macports backend and then convert that info to upt standard json or we can write a parser in python itself from scratch (if the file is easy to parse which at first look seems easy).

As @pmetzger is there for providing feedback and helping out, we can get this ready for testing in coming week as I am a bit free as both recursive and update are now under review.

reneeotten commented 5 years ago

As @pmetzger is there for providing feedback and helping out, we can get this ready for testing in coming week as I am a bit free as both recursive and update are now under review.

I appreciate your enthusiasm, but would encourage you to finish the other parts of your proposal first before starting on new a frontend. I'll follow-up with an e-mail shortly.

pmetzger commented 5 years ago

I appreciate your enthusiasm, but would encourage you to finish the other parts of your proposal first before starting on new a frontend. I'll follow-up with an e-mail shortly.

I encourage that too. And @Korusuke, wait until I get my part of the work done to be enthusiastic about how fast we can finish! :)

macports / upt-macports

opam (ocaml package manager) front end #46