Installing public C headers

[createyourpersonalaccount]

The project

See https://github.com/createyourpersonalaccount/dune_install_c_headers for the problematic code.

Expected Behavior

I am on Linux.

I hoped that the headers would be installed under /usr/local/include and the shared object would be named lib*.so.

Actual Behavior

The headers are installed under /usr/local/lib:

_build/install/default/lib/foo/CAPI/cstub.h => /usr/local/lib/foo/CAPI/cstub.h

The .so file is installed with the naem dllCAPI_stubs.so:

_build/install/default/lib/stublibs/dllCAPI_stubs.so => /usr/local/lib/stublibs/dllCAPI_stubs.so

Reproduction

1. Clone https://github.com/createyourpersonalaccount/dune_install_c_headers
2. dune build -p foo-ocaml,foo
3. opam-installer foo-ocaml (from root)
4. opam-installer foo (from root)

Specifications

• Version of dune (output of dune --version): 3.11.1
• Version of ocaml (output of ocamlc --version): 4.13.1
• Operating system (distribution and version): Debian 12

[emillon]

Hi, The reason you're hitting these issues is that dune does not really have a first class notion of installed C libraries:

• dune installs files based on opam's sections and there's no section for include files. So we're using lib for that purpose. This sort of works when installing headers for C libraries, but will probably will look weird from other systems
• foreign stubs are meant to be installed in stublibs and have this naming scheme. Packaging a .so can be done, sort of, but it is going to be fairly manual. See #854 for an old example
• don't run opam-installer as root. this is going to install to all various places where it's going to be difficult to undo.

[createyourpersonalaccount]

How to install for the system without opam-installer as root?

I think the solution right now is to use two build systems, dune for producing the files (and perhaps installing them in _install) and another one for installing them by grabbing them from _install.

Do you agree with this idea? I could write an example or two with meson/CMake and dune if you're interested and submit for PR.

[emillon]

What are you trying to do?

I don't think that dune is a good replacement for meson or cmake. It's a build system primarily fit for OCaml projects. Installing things is not a primary concern of dune. Same with opam, really - it's designed to be a package manager for developers. Opam is not meant to install on non-development machines.

[emillon]

What are you trying to do?

To clarify, I mean that literally. It looks like you're trying to use dune for something that's out of its scope. That can help me or someone else guide you.

[createyourpersonalaccount]

I was just trying to write a tutorial on how to create a formally verified C library via the Coq -> OCaml -> C path. Not for any particular reason other than showing how it's done; I haven't yet gotten to C extraction in my studies. Maybe I should just use ocamlopt, gcc and makefiles.

[emillon]

If you're not trying to make an actual opam package, I would recommend manually copying things out of the _build directory.

[createyourpersonalaccount]

@emillon that's what I meant when I suggested with dune+another build system. Are you interested in a PR that shows this example? I think some more examples would be nice, honestly I wouldn't mind writing some other examples I have in mind.

[rgrinberg]

I don't know if you're aware, but opam-installer does have options that allow you to customize the paths of certain files. I'm not sure if it's flexible enough for your use case, but it's worth trying. $ opam-installer --help.

[createyourpersonalaccount]

@rgrinberg Seems just as limited as Dune.

[Alizter]

@createyourpersonalaccount I would like to keep this issue open as a documentation one, so that we can improve the manual. I agree that it is sometimes not clear how much C Dune can build exactly. Some places mention that support is only for stubs, whilst others advocate the building of "foreign sources". We should attempt to clear this up and give guidance to users wishing to have an include install location, since I doubt you will be the last user to request this.

At the end of the day, we are supporting what opam supports as mentioned above, but this limitation need not be a strict one if a good proposal to add flexibility is put forward.

[createyourpersonalaccount]

@Alizter I would like to help with the manual. My first thought is to PR an example project (with comments) that takes it as far as it Dune can take it with regards to this situation.

I keep asking if anyone of the maintainers is interested because I don't want to bother PRing something that is not interesting. I think you've made clear it is, so expect it soon...

[Alizter]

@createyourpersonalaccount I think we definitely welcome documentation improvements!

[createyourpersonalaccount]

Can you explain a bit how to work the hello_world example? I've tried following the directions in https://github.com/ocaml/dune/blob/f26279c826c31fe9dc5e782afaf120dd52c90a46/example/hello_world.t/README.md and https://github.com/ocaml/dune/blob/f26279c826c31fe9dc5e782afaf120dd52c90a46/example/hello_world.t/run.t but neither work for me. The former outputs nothing, and the latter complains that ./bin/main.exe is missing.

I thought I'd follow the format of the other examples, but I don't quite understand it.

[Alizter]

hello_world.t is a cram test which you can directly run. In Dune all cram tests generate an alias with their name so you can do

./dune.exe build @hello_world

after bootstrapping the dune repo (make bootstrap).

Some further tips would be to use -w for watch mode and --auto-promote which will update the cram test with any changes. That way ./dune.exe build @hello_world -w --auto-promote let's you update the cram test interactively.

It's probably best to make another cram test for this particular example, simply add your_example.t as a file in examples/. Same as before, run ./dune.exe build @your_example -w --auto-promote and you can interactively setup a project. Since this is a regular cram file rather than a directory test like hello_world you have to create the files in the test, which I perhaps easier and more instructive to read.

You can create files like:

In cram tests, this is a comment because it doesn't start with 2 spaces.
  $ cat > dune-project <<EOF
  > (lang dune 3.11)
  > EOF
  $ cat > dune << EOF
  > (executable
  >  (name test))
  > EOF
  $ cat > hello.ml <<EOF
  > let () = print_endline "hi"
  > EOF
  $ dune exec ./hello
  hi 

Have a look in test/blackbox-tests/test-cases/ for more examples of cram tests.

[createyourpersonalaccount]

@Alizter I have trouble after the build step. I do not seem to be able to run the hello world example.

Yes, my intention is to add separate examples, but first I wanted to understand how to run the hello world example.

[Alizter]

@createyourpersonalaccount Could you give some more information about what you are trying, what you expect to happen and what is actually happening?

[createyourpersonalaccount]

@Alizter I'm trying to get "Hello world" to display on screen from the hello_world example. Which command does that? The files under the hello_world example seem to indicate that this is what the example does.

[Alizter]

@createyourpersonalaccount The hello_world example is a cram test meaning it is an isolated environment. You cannot build the executable it is building directly, but you have to interact with it within the test (run.t). This is what I have described above.

There is more documentation here https://dune.readthedocs.io/en/stable/tests.html#cram-tests

When you "run" a cram test you do dune build @hello_world and it will diff the output of the cram test against the expected on in the file. If there is a difference then it will fail.

[createyourpersonalaccount]

@Alizter Thanks for the explanation. I'm still lost:

I did not understand what a cram test is. It's not common knowledge and the Dune documentation under https://dune.readthedocs.io/en/stable/tests.html#cram-tests should link to https://bitheap.org/cram/ (I did find the link at https://github.com/ocaml/dune/tree/main/test though) or at least elaborate more upon what a cram test is. (it's comparing test binary output to expected output?)

Is the purpose of cram tests on Dune to guarantee stability (scriptability) of Dune's output?

With that being said, here are the issues I have:

1) When I use dune build @hello_world -w --auto-promote and I make changes to hello_world, it always prints Success. For example, if I change Hello_world.message to be something else, I do not get an error (but I'd expect the test to fail? then again, it builds successfully, but I'd expect the tests to fail? How do you run the tests?) 2) If cram tests are comparing cmdline command output, it does not necessarily apply for a dune project that generates .ml and .so files, right? Is it possible to PR under the example directory a dune project without cram tests?

[Alizter]

I did not understand what a cram test is.

@createyourpersonalaccount Thank you for sharing, this is really valuable feedback. The reality is that our documentation on cram tests is poor at best, so this is perhaps a good opportunity to do better. I've spent some time to write a more detailed answer to answer your questions so that you can get a better feel for them. Hopefully this can be turned into documentation at some point.

What is a cram test?

A cram test is a file with a simple syntax that consists of command, comments and output. Stepping back from our previous discussion with regard to Dune, a cram test is able to run commands in a shell environment and compare their outputs.

A cram test may look like this. This very sentence you are reading is a comment,
because it does not begin with two spaces. Our next component is a shell
command. This can look like this:

  $ echo 'Hello World'

This cram test could be saved in a file my_test.t and we can run it with Dune and see what happens. When we write

dune build @my_test

Dune will "run" the cram test. But what does "running" a cram test entail? Well, Dune will execute each line beginning with $ in a sandboxed environment and compare the output. This as written will fail because the command echo outputs something and we have not expected any output.

We therefore expect Dune to say something like this:

File "my_test.t", line 1, characters 0-0:                                                                                                                  
------ my_test.t                                                                                                                                           
++++++ my_test.t.corrected                                                                                                                                 
File "my_test.t", line 5, characters 0-1:                                                                                                                 
 |  $ echo 'Hello World'
+|  Hello World

When diffs like this are reported by Dune, they get registered for something called promotion. All that means is that Dune found that running the file produced a different version, and promoting it will replace your original file. In our case, this just consists of adding Hello World with two spaces before under our echo command, but not wanting to write this manually, the promotion mechanism provides a shortcut.

Our updated test will now look like this (I've removed the comments):

  $ echo 'Hello World'
  Hello World

As you can see above, anything that follows with two spaces after a command is considered "output" and each `$` line of a cram will have its output diffed with that.

What are cram tests good for?

Cram tests allow you to test the functionality of a binary in a shell like environment. We only played around with echo above, but we could very well include a binary we built in our project. In the case of Dune itself, we use cram tests to test Dune's own functionality. This is what the examples are trying to demonstrate.

Addressing your questions:

1. When I use dune build @hello_world -w --auto-promote and I make changes to hello_world, it always prints Success. For example, if I change Hello_world.message to be something else, I do not get an error (but I'd expect the test to fail? then again, it builds successfully, but I'd expect the tests to fail? How do you run the tests?)

I maybe was a little too fast to introduce --auto-promote there, so let's take a step back and think about what is happening. If you only write dune build @hello_world -w you will see that the cram test will be run each time you update it. Using our previous example, if our cram test was edited to look like:

  $ echo 'Hello'
  Hello World

Then Dune would complain and say that it expected 'Hello' rather than 'Hello World' as the output. As I mentioned before, when diffs are reported by Dune, they get registered for promotion, which is a shortcut way of updating a file. In this case, I can stop dune build -w and do dune promote. I will now see that my cram test got updated to:

  $ echo 'Hello'
  Hello

I can now run dune build @hello_world -w again and see that it is successful, because the cram test now has the correct expected output. --auto-promote simplifies this process, well, by automatically promoting each time Dune reports a diff. This means in your editor whilst editing your cram test you can write:

  $ echo 'Greetings'

and when you save, Dune will rebuild since it is in watch mode. It will then temporarily raise an error complaining that Greetings was expected in the output. But because --auto-promote was set, the diff will be automatically promoted causing the file to change and Dune to yet again rebuild. This time when it rebuilds however it will be successful and your cram test will have the correct output.

2. If cram tests are comparing cmdline command output, it does not necessarily apply for a dune project that generates .ml and .so files, right? Is it possible to PR under the example directory a dune project without cram tests?

I mentioned above that cram tests run in a sandboxed environment. This means that any files I create in the cram test will be isolated from the rest of the project. I'll give an example of how this might be used to demonstrate a simple hello world project in C:

Let's imagine we are writing a beginners tutorial for people wanting to write a hello world program in C. It's easy to share the exact .c file you would need, but there are obviously some other steps that are required such as using the compiler, running the program etc. A cram test should be able to do all of this, and here is how.

First we create a file called my_first_c_project.t and we run it the way I mentioned above with dune build @my_first_c_project -w --auto-promote. In our editor, we can edit and save the file and it should update the way I mentioned before. We could start with something like this:

In order to write "Hello World" in C, you first have to make a .c file:

  $ cat > hello_world.c <<EOF
  > #include <stdio.h>
  > int main() {
  >   printf("Hello, World!");
  >   return 0;
  > }
  > EOF

This is using cat and sh's multi-line command functionality to write the output to a file. The > beginning after two spaces is a way to give a "multiline $".

This will not output anything, so when saving, Dune will be happy with the cram test and not update it. What is really happening is that we are creating a file hello_world.c in the sandbox. We can even make sure by using ls:

In order to write "Hello World" in C, you first have to make a .c file:

  $ cat > hello_world.c <<EOF
  > #include <stdio.h>
  > int main() {
  >   printf("Hello, World!");
  >   return 0;
  > }
  > EOF

Checking that we have actually created the file:
  $ ls
  hello_world.c

Now I only wrote $ ls but after saving, Dune rebuilt and promoted the output so I didn't write the hello_world.c part. I won't explain the promotion every time now, so just assume I am writing commands and Dune is updating the output.

We can continue the rest of the test as so:

In order to write "Hello World" in C, you first have to make a .c file:

  $ cat > hello_world.c <<EOF
  > #include <stdio.h>
  > int main() {
  >   printf("Hello, World!");
  >   return 0;
  > }
  > EOF

Checking that we have actually created the file:
  $ ls 
  hello_world.c
  helpers.sh

To compile our program, we use the gcc command:
  $ gcc hello_world.c -o hello_world  

Now we run our program
  $ ./hello_world
  Hello, World!

Hopefully that explains how we might use cram tests in order to test and demonstrate executables that rely on other files.

Is the purpose of cram tests on Dune to guarantee stability (scriptability) of Dune's output?

As demonstrated above, cram tests lets you run executables in an isolated environment so its not just Dune's output we are testing but also it's functionality. This can be thought of a blackbox approach to testing. Testing what the user will see and use.

Final comments

Regarding the examples directory, this might be a bit of a misnomer and a point of improvement. As you can see, cram tests can be very useful, but they can be a bit rich in information especially for newcomers looking for examples. As they stand today, you cannot run the cram tests yourself, but you can "drop into" it's environment in the test file when running Dune with watch mode and auto promotion to get an interactive environment to test out commands.

Going forward there are probably some things we can take from this:

1. Improve the documentation of cram tests. We have mentioned in other issues about the importance of reference vs tutorial documentation. And I think cram tests would be a good example of this. There is a lot to explain, but sometimes the best explanations come as good examples.
2. The examples directory is not beginner-friendly enough. Cram tests might be too advanced if your goal is to understand some basics of Dune.

[createyourpersonalaccount]

@Alizter Thanks for the elaborate response.

I understand the Cram test workflow now, the programmer writes a test project and then opens the cram test file (preferably in an editor that auto-loads on disk save, e.g. on Emacs with M-x auto-revert-mode on the buffer) and writes a literate programming styled notebook-like (bash?) isolated shell session.

Some observations and questions about cram tests:

1. (question) Is this a pure Dune feature inspired from the Python/Perl cram test thing?
2. (observation) Making data part of the isolated environment with cat seems not ideal because it embeds everything in the .t file. I assume for binary data one would do echo "base64 output here" | base64 -d > data.bin.
3. (question) Shouldn't there be a stanza/mechanism to make files part of the isolated environment?
4. (question) How isolated is this environment? Is it e.g. using namespaces on Linux? What about other OSes? What does the environment provide? Is gcc available? (or cc?)
5. (question) If the environment is native, then cram tests suffer from incompatibility between environments. Does it use something like busybox?
6. (question) Can I access a built library and produced header files from the isolated environment? E.g. with gcc -I ... -L ...
7. (suggestion) Can example be renamed to cram-examples inside a directory example? Now it seems to be the other way around -- sample-projects (which stands nonexistent/empty!) inside example.

That being said, I have enough information now to contribute, so I'm working on the example PRs. Perhaps I'll try to edit the cram documentation when I'm done.

UPDATE

After attempting to write an example under sample-projects, I realized that it is impossible to even play around with these examples because they're under a data_only_dirs stanza (and dune looks at the parent directory)! That means you can't actually build the examples! This setup is broken in my opinion (very unintuitive) and due for a change. I propose that the suggestion in 7. is followed through. There ought to be some way for beginners to build examples and see them in action.

[Alizter]

1. (question) Is this a pure Dune feature inspired from the Python/Perl cram test thing?

Yes, I believe so.

2. (observation) Making data part of the isolated environment with cat seems not ideal because it embeds everything in the .t file. I assume for binary data one would do echo "base64 output here" | base64 -d > data.bin.

For this reason, we have the directory version of cram tests which the tests in examples/ are. These consist of test.t/run.t where you would write your test in run.t but the test is called test. Any files you include in test.t will be available in the test.

Sometimes it is better to use the file version, especially if you are explaining the contents of the file in the test. But I agree for something like binary data, you would want that in the directory version instead.

3. (question) Shouldn't there be a stanza/mechanism to make files part of the isolated environment?

If you are referring to the previous environment, then no, you can just include them in the directory. If you are referring to making executables and other dependencies available to the test then you have the cram stanza. For some examples of its usage, have a look in test/blackbox-tests/test-cases/dune.

4. (question) How isolated is this environment? Is it e.g. using namespaces on Linux? What about other OSes? What does the environment provide? Is gcc available? (or cc?)

It's not that isolated. It's nowhere near as sophisticated as a BSD jail or bubblewrap sandbox. It simply uses a separate directory based on the hash of the dependencies of the test. The location looks something like

_build/.sandbox/70c0c2e7b0991d808e788b058d70ec81/

but this is an internal implementation detail that isn't really relevant to users. Sandboxing is a more general Dune feature that allows us to run actions in an isolated environment. When we ave an action that turns a file A into a file B using an executable C say, sandboxing this action will setup a directory with only A in it, C will be run in that directory and then B would be expected to be generated and copied back to the regular build directory.

In practice this allows commands that are sensitive to the existence of other files to work reliably. It also gives guarantees on reproduciblity and correctness of the rules, as they wouldn't work correctly if everything wasn't specified fully. Such guarantees are important for the stability of the Dune cache.

Anyway, coming back to cram tests, when you run a cram test they produce an action which is sandboxed.

Currently PATH is inherited from the running Dune, but using the env and cram stanzas you can include additional things.

5. (question) If the environment is native, then cram tests suffer from incompatibility between environments. Does it use something like busybox?

We don't use anything like busybox, especially since Dune aims to have little-to-no system dependencies. Cram tests are run using the system sh which can of course differ between linux and macos. Generally keeping to posix sh can keep things working between platforms. There is a section in the documentation about scrubbing the output, since you might need to pipe output through sed in order to make sure it is reproducible or consistently silent. Cram tests also work on Windows, but as far as I know they are using a bash executable, which is required if you managed to install OCaml on Windows. dkml/cygwin etc.

Here are some related issues about specifying the sh used for cram tests:

• https://github.com/ocaml/dune/issues/7829
• https://github.com/ocaml/dune/issues/8134

6. (question) Can I access a built library and produced header files from the isolated environment? E.g. with gcc -I ... -L ...

Cram tests do not produce any artefacts that you can later use in your builds. This is because, as explained earlier, they were sandboxed and their action did not declare any targets which would be copied back to the build directory.

In order to do that you would need to setup the rules to build those directly, using rule stanzas with the system action perhaps, and maybe even sandbox those if you want them to be more isolated. You can then specify the header files to be the targets of this rule so that you can produce them.

If you want to try them out in your cram test, you can include the targets of that rule in your cram stanza, but I'm not sure why you would want to do that.

7. (suggestion) Can example be renamed to cram-examples inside a directory example? Now it seems to be the other way around -- sample-projects (which stands nonexistent/empty!) inside example.

AFAIK, we only use examples/ for the dune.build website, which is updated very infrequently. I would be in favour of getting rid of it or replacing it with something more useful. I think we should create another issue about this however as this issue has become a little long.