[WIP] OCaml 5.0 support

[kit-ty-kate]

~For now this branch will only work with ocaml-src pointing to https://github.com/kit-ty-kate/ocaml/tree/solo5-500 (all changes are waiting to be backported to the 5.0 branch)~ EDIT: ocaml-src from opam-repository works fine now.

[copying my first comment for a more visible disclaimer] For the ocaml-solo5 devs: Feel free to ask questions, rebase the branch, add changes to this branch (you should have the rights to, though don't forget to use git push --force-with-lease if you force-push in case i also add some changes at the same time). I might have not time to finish the work so please do continue it.

joint work with @TheLortex at the MirageOS retreat 2022 with some help of all the people present at the event.

[kit-ty-kate]

For the ocaml-solo5 devs: Feel free to ask questions, rebase the branch, add changes to this branch (you should have the rights to, though don't forget to use git push --force-with-lease if you force-push in case i also add some changes at the same time)

[dinosaure]

On the other side, for your commit on OCaml: extend required artifact for coldstart rule, I wonder if it's possible to set the Makefile.config before to call ./configure && make as we usual do into our ./Makefile.

The rest of commits on the OCaml side seems fair.

[dinosaure]

In anyway, that's a great work and a good foundation to really figure out the OCaml 5 support. Thanks for that 👍 .

[kit-ty-kate]

Some progress (I can only test spt as hvt doesn't seem to work on my M1 laptop, see https://github.com/Solo5/solo5/issues/529):

using ocaml-solo5 from opam-alpha-repository, which points to this PR. The only bug in the mirage toolchain i had to hotfix to get opam-monorepo/mirage working is to entirely remove the --ocaml-version argument from the generated Makefile (see related PR https://github.com/mirage/mirage/pull/1363)

[kit-ty-kate]

            |      ___|                   
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version afa7fdf
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x227fff)
Solo5:     rodata @ (0x228000 - 0x23ffff)
Solo5:       data @ (0x240000 - 0x2fbfff)
Solo5:       heap >= 0x2ff000 < stack < 0x20000000
zsh: segmentation fault (core dumped)  solo5-spt dist/noop.spt

Running it now results in a segfault with the latest commits. Does anyone know how to debug this? gdb and valgrind seem to be useless in that situation.

[haesbaert]

            |      ___|                   
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version afa7fdf
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x227fff)
Solo5:     rodata @ (0x228000 - 0x23ffff)
Solo5:       data @ (0x240000 - 0x2fbfff)
Solo5:       heap >= 0x2ff000 < stack < 0x20000000
zsh: segmentation fault (core dumped)  solo5-spt dist/noop.spt

Running it now results in a segfault with the latest commits. Does anyone know how to debug this? gdb and valgrind seem to be useless in that situation.

https://github.com/Solo5/solo5/blob/master/docs/debugging.md

Did you try that ^ ?

[kit-ty-kate]

I didn't but even with it i don't really get further:

$ gdb -q --args solo5-spt dist/noop.spt
Reading symbols from solo5-spt...
(gdb) b spt_launch
Breakpoint 1 at 0x2708: file spt/spt_launch_aarch64.S, line 27.
(gdb) r
Starting program: /home/kit_ty_kate/.opam/next/bin/solo5-spt dist/noop.spt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Breakpoint 1, spt_launch () at spt/spt_launch_aarch64.S:27
27          mov  sp, x0
(gdb) add-symbol-file dist/noop.spt 0x100000
add symbol table from file "dist/noop.spt" at
    .text_addr = 0x100000
(y or n) y
Reading symbols from dist/noop.spt...
(gdb) b solo5_app_main
Breakpoint 2 at 0x1844c4
(gdb) c
Continuing.
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version afa7fdf
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x227fff)
Solo5:     rodata @ (0x228000 - 0x23ffff)
Solo5:       data @ (0x240000 - 0x2fbfff)
Solo5:       heap >= 0x2ff000 < stack < 0x20000000
thread_get_info_callback: cannot get thread info: generic error

[palainp]

Hi, if that's easy to setup, can you try also adding breakpoints on mem_init, block_init, net_init, mem_lock_heap? mem_init prints the memory banner, so I expect it works correctly but block_init and net_init currently use the arg structure pointer.

[kit-ty-kate]

The following inputs do result in the same output:

b spt_launch
r
add-symbol-file dist/noop.spt 0x100000
b solo5_app_main
b mem_init
b block_init
b net_init
b mem_lock_heap
c

$ cat gdbrun | gdb -q --args solo5-spt dist/noop.spt
Reading symbols from solo5-spt...
(gdb) Breakpoint 1 at 0x2708: file spt/spt_launch_aarch64.S, line 27.
(gdb) Starting program: /home/kit_ty_kate/.opam/next/bin/solo5-spt dist/noop.spt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Breakpoint 1, spt_launch () at spt/spt_launch_aarch64.S:27
27          mov  sp, x0
(gdb) add symbol table from file "dist/noop.spt" at
    .text_addr = 0x100000
(y or n) [answered Y; input not from terminal]
Reading symbols from dist/noop.spt...
(gdb) Breakpoint 2 at 0x1844c4
(gdb) Breakpoint 3 at 0x101790: file mem.c, line 47.
(gdb) Breakpoint 4 at 0x102000: file spt/block.c, line 27.
(gdb) Breakpoint 5 at 0x102230: file spt/net.c, line 30.
(gdb) Breakpoint 6 at 0x101720: file mem.c, line 35.
(gdb) Continuing.
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version afa7fdf
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x227fff)
Solo5:     rodata @ (0x228000 - 0x23ffff)
Solo5:       data @ (0x240000 - 0x2fbfff)
Solo5:       heap >= 0x2ff000 < stack < 0x20000000
thread_get_info_callback: cannot get thread info: generic error
(gdb) 
quit
A debugging session is active.

    Inferior 1 [process 22778] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

[palainp]

I'm really surprised that the mem_init breakpoint do not require a continue directive. I was expecting to have the "solo5" banner, then a break at the begining of mem_init, then continue, then the memory layout printing, etc. I guess that there's some sort of relocation somewhere here and gdb can't see the correct addresses of the functions. As a hard way to investigate, would it be possible to run gdb in interractive mode and type a couple of 'n' after the first breakpoint in spt_launch?

[kit-ty-kate]

$ gdb -q --args solo5-spt dist/noop.spt
Reading symbols from solo5-spt...
(gdb) b spt_launch
Breakpoint 1 at 0x2708: file spt/spt_launch_aarch64.S, line 27.
(gdb) r
Starting program: /home/kit_ty_kate/.opam/next/bin/solo5-spt dist/noop.spt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Breakpoint 1, spt_launch () at spt/spt_launch_aarch64.S:27
27          mov  sp, x0
(gdb) add-symbol-file dist/noop.spt 0x100000
add symbol table from file "dist/noop.spt" at
    .text_addr = 0x100000
(y or n) y
Reading symbols from dist/noop.spt...
(gdb) n
spt_launch () at spt/spt_launch_aarch64.S:28
28          mov  x0, x2
(gdb) n
29          mov  x30, xzr
(gdb) n
31          blr x1
(gdb) n
_start (arg=0xaaaaaaab84f0) at spt/start.c:27
27      crt_init_ssp();
(gdb) n
28      crt_init_tls();
(gdb) n
32      platform_init(arg);
(gdb) n
33      si.cmdline = cmdline_parse(platform_cmdline());
(gdb) n
thread_get_info_callback: cannot get thread info: generic error

[haesbaert]

Join the church of printf ! You're very likely reaching solo5_app_main() and then caml_startup(), I'd make sure that is the case, and then think from there, it might even be easier to just bisect your commits and see which one broke and then make an informed guess. Assuming that holds, it's very likely caml_startup() doing some call into the nolibc which explodes.

[palainp]

I am very confused here as going step by step doesn't properly shows the "solo5" banner. So I think that the gdb error is occuring at some point and may not be related to the solo5 segfault (quick search on the internet indicates this is related to a mismatch of libc thread structure, gdb can't decode it, and of course we don't have glibc here in spt).

tldr: I guess @haesbaert's wise advice is the way to go, join the printf debugging club :)

[palainp]

Update: Not sure if that can be a step forward or not, I'm trying a second time with gdb:

Breakpoint 1, spt_launch () at spt/spt_launch_x86_64.S:27
27  spt/spt_launch_x86_64.S: No such file or directory.
(gdb) c
Continuing.
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000

Program received signal SIGSEGV, Segmentation fault.
0x00000000001e8f89 in fresh_domain_unique_id () at runtime/domain.c:517
517     uintnat next = next_domain_unique_id++;
(gdb) info threads
  Id   Target Id                  Frame 
* 1    process 149844 "solo5-spt" 0x00000000001e8f89 in fresh_domain_unique_id ()
    at runtime/domain.c:517
(gdb) bt
#0  0x00000000001e8f89 in fresh_domain_unique_id () at runtime/domain.c:517
#1  domain_create (initial_minor_heap_wsize=262144) at runtime/domain.c:583
#2  0x00000000001e938f in caml_init_domains (minor_heap_wsz=262144)
    at runtime/domain.c:873
#3  0x0000000000205189 in init_segments () at runtime/startup_nat.c:57
#4  caml_startup_common (pooling=<optimized out>, argv=0x2c6a10 <unused_argv>)
    at runtime/startup_nat.c:114
#5  caml_startup_common (argv=0x2c6a10 <unused_argv>, pooling=<optimized out>)
    at runtime/startup_nat.c:85
#6  0x000000000020529b in ?? ()    # <- this is in 0000000000205290 <caml_startup>:
#7  0x00000000002d1000 in ?? ()    # <- this is strange there is no code at this address, with the memory layout we can see that this is at the bottom of the heap.
#8  0x0000000000173f8e in ?? ()     # <- this is in 0000000000173f40 <solo5_app_main>:
#9  0x0000000000000000 in ?? ()
(gdb) p $sp
$4 = (void *) 0x1fffff40
(gdb) p next_domain_unique_id
$5 = 0
(gdb) p &next_domain_unique_id
$6 = (uintnat *) 0x2c7148 <next_domain_unique_id>
(gdb) 

The stack seems to be fine for allocating next and next_domain_unique_id also seems to be reachable. I currently don't know why the SIGSEGV is triggered.

EDIT: The faulty asm instruction is 1e8f89: 64 48 89 1c 25 e8 ff mov %rbx,%fs:0xffffffffffffffe8 fs=0 at the segfault time and 0xfff...e8 seems to be a negative offset :/

[haesbaert]

I think the debugging is imprecise, this is pretty much just after the first deference of s: s->unique_id = fresh_domain_unique_id(); s is probably screwed up, which means &d->interruptor is bad. Don't be tricked by the CAMLassert before, I think that is not compiled in.

[haesbaert]

I think the debugging is imprecise, this is pretty much just after the first deference of s: s->unique_id = fresh_domain_unique_id(); s is probably screwed up, which means &d->interruptor is bad. Don't be tricked by the CAMLassert before, I think that is not compiled in.

I can't really see how &d->interruptor would be bad, but I'd try to dereference s before your call just to make sure the problem is that.

[palainp]

Here are some printing (sorry, this will be a bit long :/), we probably segfault before accessing s and gdb seems to misalign the code and the assembly:

$ gdb -q --args solo5-spt dist/hello.spt
Reading symbols from solo5-spt...
(gdb) add-symbol-file dist/hello.spt 0x100000
add symbol table from file "dist/hello.spt" at
    .text_addr = 0x100000
(y or n) y
Reading symbols from dist/hello.spt...
(gdb) r
Starting program: /home/user/.opam/5.0.0/bin/solo5-spt dist/hello.spt
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000

Program received signal SIGSEGV, Segmentation fault.
0x00000000001e8f89 in fresh_domain_unique_id () at runtime/domain.c:517
warning: Source file is more recent than executable.
517     uintnat next = next_domain_unique_id++;
Missing separate debuginfos, use: dnf debuginfo-install libseccomp-2.5.3-1.fc34.x86_64
(gdb) up
#1  domain_create (initial_minor_heap_wsize=262144) at runtime/domain.c:583
583   s->unique_id = fresh_domain_unique_id();
(gdb) p *d
$1 = {id = 0, state = 0x0, interruptor = {interrupt_word = 0x0, lock = 0, 
    cond = {cond = 0, mutex = 0x2c7598 <all_domains+24>}, running = 0, 
    terminating = 0, unique_id = 0, interrupt_pending = 0}, 
  backup_thread_running = 0, backup_thread = 0, backup_thread_msg = 3, 
  domain_lock = 0, domain_cond = {cond = 0, 
    mutex = 0x2c75e0 <all_domains+96>}, minor_heap_area_start = 3223552, 
  minor_heap_area_end = 5320704}
(gdb) p *s
$2 = {interrupt_word = 0x0, lock = 0, cond = {cond = 0, 
    mutex = 0x2c7598 <all_domains+24>}, running = 0, terminating = 0, 
  unique_id = 0, interrupt_pending = 0}
(gdb) p s
$3 = (struct interruptor *) 0x2c7590 <all_domains+16>
(gdb) p &d->interruptor
$4 = (struct interruptor *) 0x2c7590 <all_domains+16>
(gdb) p domain_state
$5 = (caml_domain_state *) 0x80
(gdb) p $fs
$6 = 0

EDIT: So far I'm confused why we have d->state==NULL and why domain_state==0x80 as the following block code should update the d->state field and provides a legit address for domain_state with the calloc call:

  if (d->state == NULL) {
    /* FIXME: Never freed. Not clear when to. */
    domain_state = (caml_domain_state*)
      caml_stat_calloc_noexc(1, sizeof(caml_domain_state));
    if (domain_state == NULL)
      goto domain_init_complete;
    d->state = domain_state;
  } else {
    domain_state = d->state;
  }

Here is how I read the assembly for domain_create:

00000000001e8f20 <domain_create>:
  1e8f20:       41 56                   push   %r14
  1e8f22:       41 55                   push   %r13
  1e8f24:       41 54                   push   %r12
  1e8f26:       49 89 fc                mov    %rdi,%r12
  1e8f29:       55                      push   %rbp
  1e8f2a:       53                      push   %rbx

;   uintnat stack_wsize = caml_get_init_stack_wsize();
  1e8f2b:       e8 00 44 00 00          callq  1ed330 <caml_get_init_stack_wsize>
  1e8f30:       bf 88 b9 2c 00          mov    $0x2cb988,%edi
  1e8f35:       49 89 c5                mov    %rax,%r13

;   
  1e8f38:       e8 73 ff f8 ff          callq  178eb0 <pthread_mutex_lock>
  1e8f3d:       85 c0                   test   %eax,%eax
  1e8f3f:       74 19                   je     1e8f5a <domain_create+0x3a>
  1e8f41:       e9 7c 03 00 00          jmpq   1e92c2 <domain_create+0x3a2>

;  /* Wait until any in-progress STW sections end. */
;  while (atomic_load_acq(&stw_leader)) {
;    /* [caml_plat_wait] releases [all_domains_lock] until the current
;       STW section ends, and then takes the lock again. */
;    caml_plat_wait(&all_domains_cond);
;  }
  1e8f46:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  1e8f4d:       00 00 00 
  1e8f50:       bf 90 6a 2c 00          mov    $0x2c6a90,%edi
  1e8f55:       e8 d6 3f 01 00          callq  1fcf30 <caml_plat_wait>
  1e8f5a:       48 8b 05 1f 2a 0e 00    mov    0xe2a1f(%rip),%rax        # 2cb980 <stw_leader>
  1e8f61:       48 85 c0                test   %rax,%rax
  1e8f64:       75 ea                   jne    1e8f50 <domain_create+0x30>

;   d = next_free_domain(); (inlined? Max_domains==128?)
;   if (d == NULL)
;     goto domain_init_complete;
  1e8f66:       48 63 05 f3 e1 0d 00    movslq 0xde1f3(%rip),%rax        # 2c7160 <stw_domains>
  1e8f6d:       3d 80 00 00 00          cmp    $0x80,%eax
  1e8f72:       0f 84 e9 02 00 00       je     1e9261 <domain_create+0x341>  ;  this is a goto end
  1e8f78:       48 8b 1c c5 68 71 2c    mov    0x2c7168(,%rax,8),%rbx
  1e8f7f:       00 
  1e8f80:       48 85 db                test   %rbx,%rbx
  1e8f83:       0f 84 d8 02 00 00       je     1e9261 <domain_create+0x341>  ;  this is a goto end

;  s = &d->interruptor;
;  CAMLassert(!s->running);
;  CAMLassert(!s->interrupt_pending);
;
;  domain_self = d;

; here is the segfault line, so far %fs==0
  1e8f89:       64 48 89 1c 25 e8 ff    mov    %rbx,%fs:0xffffffffffffffe8
  1e8f90:       ff ff 
  1e8f92:       48 8b 6b 08             mov    0x8(%rbx),%rbp
  1e8f96:       48 85 ed                test   %rbp,%rbp
  1e8f99:       0f 84 e9 02 00 00       je     1e9288 <domain_create+0x368>  ;  this is the test   if (d->state == NULL) {

; this is the return from the calloc_noexec call, so this is probably   d->state = domain_state; ?
  1e8f9f:       64 48 89 2c 25 f0 ff    mov    %rbp,%fs:0xfffffffffffffff0
  1e8fa6:       ff ff 

; here are the computation around inligned static uintnat fresh_domain_unique_id(void) ?
  1e8fa8:       48 8b 05 99 e1 0d 00    mov    0xde199(%rip),%rax        # 2c7148 <next_domain_unique_id>
  1e8faf:       b9 01 00 00 00          mov    $0x1,%ecx
  1e8fb4:       48 89 6b 10             mov    %rbp,0x10(%rbx)
  1e8fb8:       48 89 c2                mov    %rax,%rdx
  1e8fbb:       48 89 43 38             mov    %rax,0x38(%rbx)
  1e8fbf:       48 83 c2 01             add    $0x1,%rdx
  1e8fc3:       c7 43 30 01 00 00 00    movl   $0x1,0x30(%rbx)
  1e8fca:       48 0f 44 d1             cmove  %rcx,%rdx
  1e8fce:       48 89 15 73 e1 0d 00    mov    %rdx,0xde173(%rip)        # 2c7148 <next_domain_unique_id>
  1e8fd5:       48 8b 05 64 e1 0d 00    mov    0xde164(%rip),%rax        # 2c7140 <caml_num_domains_running>
  1e8fdc:       48 83 c0 01             add    $0x1,%rax
  1e8fe0:       48 87 05 59 e1 0d 00    xchg   %rax,0xde159(%rip)        # 2c7140 <caml_num_domains_running>
  1e8fe7:       48 8d 7b 60             lea    0x60(%rbx),%rdi
  1e8feb:       e8 c0 fe f8 ff          callq  178eb0 <pthread_mutex_lock>
  1e8ff0:       85 c0                   test   %eax,%eax
  1e8ff2:       0f 85 ca 02 00 00       jne    1e92c2 <domain_create+0x3a2>
...
; this is the end of the function
; domain_init_complete:
;   caml_gc_log("domain init complete");
;   caml_plat_unlock(&all_domains_lock);
  1e9261:       bf 72 57 21 00          mov    $0x215772,%edi
  1e9266:       31 c0                   xor    %eax,%eax
  1e9268:       e8 63 2f 01 00          callq  1fc1d0 <caml_gc_log>
  1e926d:       bf 88 b9 2c 00          mov    $0x2cb988,%edi
  1e9272:       e8 59 fc f8 ff          callq  178ed0 <pthread_mutex_unlock>
  1e9277:       85 c0                   test   %eax,%eax
  1e9279:       75 53                   jne    1e92ce <domain_create+0x3ae>
  1e927b:       5b                      pop    %rbx
  1e927c:       5d                      pop    %rbp
  1e927d:       41 5c                   pop    %r12
  1e927f:       41 5d                   pop    %r13
  1e9281:       41 5e                   pop    %r14
  1e9283:       c3                      retq   

;    /* FIXME: Never freed. Not clear when to. */
;    domain_state = (caml_domain_state*)
;      caml_stat_calloc_noexc(1, sizeof(caml_domain_state)); // caml_stat_calloc_noexc is probably inligned as not called later
;    if (domain_state == NULL)
;      goto domain_init_complete;
  1e9284:       0f 1f 40 00             nopl   0x0(%rax)
  1e9288:       be d0 03 00 00          mov    $0x3d0,%esi
  1e928d:       bf 01 00 00 00          mov    $0x1,%edi
  1e9292:       e8 29 16 01 00          callq  1fa8c0 <caml_stat_calloc_noexc>
  1e9297:       48 89 c5                mov    %rax,%rbp
  1e929a:       48 85 c0                test   %rax,%rax
  1e929d:       74 c2                   je     1e9261 <domain_create+0x341>    ; this is the goto end
  1e929f:       48 89 43 08             mov    %rax,0x8(%rbx)
  1e92a3:       e9 f7 fc ff ff          jmpq   1e8f9f <domain_create+0x7f>

; this is a CAMLAssert failures for the function
  1e92a8:       bf 01 00 00 00          mov    $0x1,%edi
  1e92ad:       e8 7e a7 ff ff          callq  1e3a30 <caml_record_backtraces>
  1e92b2:       e9 0f ff ff ff          jmpq   1e91c6 <domain_create+0x2a6>
  1e92b7:       4c 89 c1                mov    %r8,%rcx
  1e92ba:       4c 89 ca                mov    %r9,%rdx
  1e92bd:       e9 49 ff ff ff          jmpq   1e920b <domain_create+0x2eb>
  1e92c2:       89 c6                   mov    %eax,%esi
  1e92c4:       bf a0 6d 21 00          mov    $0x216da0,%edi
  1e92c9:       e8 f2 3a 01 00          callq  1fcdc0 <caml_plat_fatal_error>
...

[palainp]

I edited the asm output, to my understanding, we fail before the test d->state==NULL at line 570.

As s seems to be properly affected to &d->interruptor in the debugger console, maybe we fail on the next line domain_self = d; which makes sense as fs seems to be used for thread stuff and domain_self is thread related. I'm not sure how fs is supposed to be initialised, I'll continue on that track.

[palainp]

I'm now sure that the failing line is domain_self = d (with domain_self defined as a thread static variable) and this comment may be helpful: https://github.com/Solo5/solo5/blob/85ce3239ce87a6e49e550a7a0756b072a8c9fa6c/bindings/crt_init.h#L47 . fs register set to zero_page and the need of threads in Ocaml 5 can explain our troubles right?

[haesbaert]

I'm now sure that the failing line is domain_self = d (with domain_self defined as a thread static variable) and this comment may be helpful: https://github.com/Solo5/solo5/blob/85ce3239ce87a6e49e550a7a0756b072a8c9fa6c/bindings/crt_init.h#L47 . fs register set to zero_page and the need of threads in Ocaml 5 can explain our troubles right?

Nice find ! For sure that is the problem. Interesting that the TLS is kept in fs, I was thinking it would be gs, but now I read it's more common to use fs in userland.

For now you can cheat, just change the runtime for domain_self to be a common pointer, as long as there is no actual pthread_create taking place, a global pointer will work

[palainp]

Ok, I tried and had to fight a bit with the caml_state teammate too, contrary to domain_self it's not static and used everywhere, so I decided to remove domains.c:581 caml_state = domain_state;. It continues well and now fails in 00000000001fe090 <caml_init_shared_heap>: on a line with talking about fs (1fe11c: 64 48 8b 00 mov %fs:(%rax),%rax). I guess the only way out from here is to follow the hard path of adding threads in solo5.

[haesbaert]

Ok, I tried and had to fight a bit with the caml_state teammate too, contrary to domain_self it's not static and used everywhere, so I decided to remove domains.c:581 caml_state = domain_state;. It continues well and now fails in 00000000001fe090 <caml_init_shared_heap>: on a line with talking about fs (1fe11c: 64 48 8b 00 mov %fs:(%rax),%rax). I guess the only way out from here is to follow the hard path of adding threads in solo5.

I guess there are moar cases:

balin:ocaml: find . -name '*.c' -exec grep __thread {} \; | wc -l
9

Writing a userland pthread scheduler is not an insane amount of work, especially because we need a limited subset of it. When I was planning to do it I checked how some other implementations were doing it, the trick part is allocating and switching between two stacks, I can do it for x86 but would have to study for anything different. Ofc it would have to be non-preemptive as we don't have an interrupting source on on solo5, but it's enough to get the ball rolling for single domain.

At any rate, if you're up to it we can do it at some point, I'm on holidays now contemplating the end of the universe and playing dwarf fortress, so can't really commit to anything.

[palainp]

Update: I tried the dumb approach by allocating some memory for TLS and running the thing:

$ git diff
diff --git a/nolibc/mmap.c b/nolibc/mmap.c
index 8a10fcc..330b0e0 100644
--- a/nolibc/mmap.c
+++ b/nolibc/mmap.c
@@ -5,7 +5,8 @@

 void *mmap(void *addr, size_t len, int prot, int flags, int fildes, off_t off) {
   if (addr != NULL) {
-    printf("mmap: non-null addr is unsupported.\n");
+    printf("mmap: non-null addr (=%p) is unsupported.\n", addr);
+    printf("Return addr (in caller function): %p\n", __builtin_return_address(0));
     abort();
   }
   if (fildes != -1) {
diff --git a/nolibc/sysdeps_solo5.c b/nolibc/sysdeps_solo5.c
index dacd021..b6dbc7e 100644
--- a/nolibc/sysdeps_solo5.c
+++ b/nolibc/sysdeps_solo5.c
@@ -96,6 +96,10 @@ void _nolibc_init(uintptr_t heap_start, size_t heap_size)

     sbrk_start = sbrk_cur = heap_start;
     sbrk_end = heap_start + heap_size;
+
+    size_t tdata = 4096; // FIXME: hard coded
+    uintptr_t tls = (uintptr_t)malloc(tdata);
+    solo5_set_tls_base(tls);
 }

And now I have the following:

            |      ___|                  
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000
mmap: non-null addr (=0x314000) is unsupported.
Return addr (in caller function): 0x1fd0f9
Aborted
Solo5: solo5_abort() called

The caller for mmap is caml_mem_commit (defined here https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/platform.c#L191 and seems to be platform dependant. The unix version uses mprotect, not sure why I have a mmap call here).

I'll be afk for the next week, but I think we're on good track for resolving that PR. Enjoy the end of the world and be safe ;)

[palainp]

Update: The mmap's manpage leaves the door open to disregard the addr argument, so with spt or qubes as target (with 1 domain limitation and 512MB of ram) and the PR on @kit-ty-kate repo https://github.com/kit-ty-kate/ocaml-solo5/pull/1:

[2022-12-22 16:44:18] Solo5: Xen console: port 0x2, ring @0x00000000FEFFF000
[2022-12-22 16:44:18]             |      ___|
[2022-12-22 16:44:18]   __|  _ \  |  _ \ __ \
[2022-12-22 16:44:18] \__ \ (   | | (   |  ) |
[2022-12-22 16:44:18] ____/\___/ _|\___/____/
[2022-12-22 16:44:18] Solo5: Bindings version v0.7.5
[2022-12-22 16:44:18] Solo5: Memory map: 512 MB addressable:
[2022-12-22 16:44:18] Solo5:   reserved @ (0x0 - 0xfffff)
[2022-12-22 16:44:18] Solo5:       text @ (0x100000 - 0x22afff)
[2022-12-22 16:44:18] Solo5:     rodata @ (0x22b000 - 0x248fff)
[2022-12-22 16:44:18] Solo5:       data @ (0x249000 - 0x31dfff)
[2022-12-22 16:44:18] Solo5:       heap >= 0x31e000 < stack < 0x20000000
[2022-12-22 16:44:18] 2022-12-22 15:44:18 -00:00: INF [application] hello
[2022-12-22 16:44:19] 2022-12-22 15:44:19 -00:00: INF [application] hello
[2022-12-22 16:44:20] 2022-12-22 15:44:20 -00:00: INF [application] hello
[2022-12-22 16:44:21] 2022-12-22 15:44:21 -00:00: INF [application] hello
[2022-12-22 16:44:22] Solo5: solo5_exit(0) called

The next step is to have a proper threads implementation and eventually checks why I can have with not enough memory (128MB is not sufficient :/):

[2022-12-22 16:43:35] Fatal error: Not enough heap memory to reserve minor heaps

[haesbaert]

Update: I tried the dumb approach by allocating some memory for TLS and running the thing:

$ git diff
diff --git a/nolibc/mmap.c b/nolibc/mmap.c
index 8a10fcc..330b0e0 100644
--- a/nolibc/mmap.c
+++ b/nolibc/mmap.c
@@ -5,7 +5,8 @@

 void *mmap(void *addr, size_t len, int prot, int flags, int fildes, off_t off) {
   if (addr != NULL) {
-    printf("mmap: non-null addr is unsupported.\n");
+    printf("mmap: non-null addr (=%p) is unsupported.\n", addr);
+    printf("Return addr (in caller function): %p\n", __builtin_return_address(0));
     abort();
   }
   if (fildes != -1) {
diff --git a/nolibc/sysdeps_solo5.c b/nolibc/sysdeps_solo5.c
index dacd021..b6dbc7e 100644
--- a/nolibc/sysdeps_solo5.c
+++ b/nolibc/sysdeps_solo5.c
@@ -96,6 +96,10 @@ void _nolibc_init(uintptr_t heap_start, size_t heap_size)

     sbrk_start = sbrk_cur = heap_start;
     sbrk_end = heap_start + heap_size;
+
+    size_t tdata = 4096; // FIXME: hard coded
+    uintptr_t tls = (uintptr_t)malloc(tdata);
+    solo5_set_tls_base(tls);
 }

And now I have the following:

            |      ___|                  
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x206fff)
Solo5:     rodata @ (0x207000 - 0x223fff)
Solo5:       data @ (0x224000 - 0x2d0fff)
Solo5:       heap >= 0x2d1000 < stack < 0x20000000
mmap: non-null addr (=0x314000) is unsupported.
Return addr (in caller function): 0x1fd0f9
Aborted
Solo5: solo5_abort() called

The caller for mmap is caml_mem_commit (defined here https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/platform.c#L191 and seems to be platform dependant. The unix version uses mprotect, not sure why I have a mmap call here).

I'll be afk for the next week, but I think we're on good track for resolving that PR. Enjoy the end of the world and be safe ;)

Can't you just point to a static buffer though ? no need to malloc, but anyway, good find !

[palainp]

Can't you just point to a static buffer though ?

Yes, done in an update branch, I'll PR it at some point.

And in the meantime, I discovered why the unikernel now needs a lot more memory. This is due to https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/domain.c#L218 and https://github.com/ocaml/ocaml/blob/ac52c3ab4a545d77eea034993a0947ba5e79762b/runtime/domain.c#L724. The heap reservation is done once for the upper limit of possible domains (here Max_domains==128 and the heap per domain seems to be 2MB). As Max_domains is defined as a macro (https://github.com/ocaml/ocaml/blob/021619f25d84dc0161537366f01c92f6a3980851/runtime/caml/domain.h#L32), I'm not sure if the code can be soon (regarding the comment This hard limit may go away in the future.) adapted to set it dynamically and this will probably cause issues for mirage with a 256MiB entry ticket for unikernels (I'm thinking in particular of qubes-mirage-firewall which aims at low memory consumption).

I got the traces:

$ solo5-spt dist/hello.spt
            |      ___|                  
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.7.5
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x207fff)
Solo5:     rodata @ (0x208000 - 0x224fff)
Solo5:       data @ (0x225000 - 0x2d2fff)
Solo5:       heap >= 0x2d3000 < stack < 0x20000000
malloc: 176 caller 0x1fa92d     ; this is in caml_stat_alloc
malloc: 176 caller 0x1fa92d
malloc: 176 caller 0x1fa92d
malloc: 176 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
... ; lots of similar lines
malloc: 16 caller 0x1fa92d
malloc: 16 caller 0x1fa92d
malloc: 32 caller 0x1fa92d
malloc: 262144 caller 0x1fa92d
malloc: 268439552 caller 0x1fd266  ; this is in caml_mem_map, this is our 256MiB allocation
malloc: 976 caller 0x1fab50   ; this is in caml_stat_calloc_noexc
malloc: 168 caller 0x1fb75c  ; caml_alloc_minor_tables
malloc: 1120 caller 0x1fe36c ; caml_init_shared_heap
malloc: 24 caller 0x1f925d  ; caml_init_major_gc
malloc: 65536 caller 0x1f927e ; caml_init_major_gc
malloc: 112 caller 0x1ee92c ; caml_alloc_final_info
malloc: 40 caller 0x20370c
malloc: 32 caller 0x1f9333
malloc: 2097152 caller 0x1fd2f9 ; caml_mem_commit
malloc: 40 caller 0x1ed5de
malloc: 32864 caller 0x1ed510
malloc: 557056 caller 0x1fd266
malloc: 24 caller 0x1fa92d
malloc: 40 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 40 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 48 caller 0x1fa92d
malloc: 64 caller 0x1fa92d
malloc: 7 caller 0x1fac38
free: 80 caller 0x202b8d
malloc: 24 caller 0x1fa92d
malloc: 232 caller 0x1fa92d
malloc: 52 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 264192 caller 0x1fb062
malloc: 65616 caller 0x1fa92d
malloc: 792576 caller 0x1fb062
malloc: 65616 caller 0x1fa92d
malloc: 65616 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
free: 48 caller 0x1fff94
free: 48 caller 0x1fff94
malloc: 45 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 40 caller 0x1fa92d
malloc: 58 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
malloc: 14 caller 0x1fac38
free: 32 caller 0x202041
malloc: 13 caller 0x1fac38
free: 32 caller 0x202041
malloc: 7 caller 0x1fac38
free: 32 caller 0x202041
malloc: 5 caller 0x1fac38
free: 32 caller 0x202041
malloc: 12 caller 0x1fac38
free: 32 caller 0x202041
malloc: 32 caller 0x1fa92d
malloc: 8192 caller 0x204162
2022-12-31 08:34:46 -00:00: INF [application] hello
free: 48 caller 0x1fff0e
malloc: 24 caller 0x1fa92d
2022-12-31 08:34:47 -00:00: INF [application] hello
2022-12-31 08:34:48 -00:00: INF [application] hello
2022-12-31 08:34:49 -00:00: INF [application] hello
malloc: 16 caller 0x1f5f4b
malloc: 16 caller 0x1f5f4b
free: 32 caller 0x1f5fea
malloc: 24 caller 0x1fa92d
malloc: 24 caller 0x1fa92d
free: 48 caller 0x1fff94
free: 64 caller 0x1fff94
malloc: 557056 caller 0x1fd266
free: 32 caller 0x1f5fea
Solo5: solo5_exit(0) called

[haesbaert]

At this stage it's probably easier to make a compiler patch like '--with-max-domains=1'. I haven't looked on how memory management is done in solo5, I'm assuming none, which means all mapped pages are wired in ? It should be fine to map 256MB (or even like a brazillion GB) if the pages are never wired. At any rate, in the future we need a better malloc+mmap, one that is multicore aware and that is mmap based, not just sbrking left and right.

[dinosaure]

At this stage it's probably easier to make a compiler patch like '--with-max-domains=1'.

It's probably a task that @shindere should be aware :+1:. Any requests like this one deserves special attention in order to improve the toolchain compiling OCaml for our specific use. However, I'm not sure how to organize that in the middle/long term - and we still are on the experiment stage.

[shindere]

Just commenting on the Max_domain aspect here but let me know whether it was a wider comment which was expected.

I quickly went through the code and read a comment in runtime/caml/comain.h saying that the hard limit on the maximum number of domains may go away in the future. This makes me unsure that upstream will be willing to introduce an interface to specify it, because it means that one will have to handle its deprecation if one day the hard limit indeed goes away. But that's just my uninformed opinion, so if being able to specify a max doamin is important to you it may be worth discussiing with developers more involved than me in this particular area of domains.

[palainp]

This makes me unsure that upstream will be willing to introduce an interface to specify it, because it means that one will have to handle its deprecation if one day the hard limit indeed goes away.

Hi @shindere, thanks for your feedback! I think you're right here :)

But that's just my uninformed opinion, so if being able to specify a max doamin is important to you it may be worth discussiing with developers more involved than me in this particular area of domains.

As @haesbaert said in https://github.com/mirage/ocaml-solo5/pull/122#issuecomment-1368194972 it's probably best at some point to change the allocation system in solo5 and to allow some memory to be requested and only reserve it when it's really used (the hint should be MAP_NONE in the flags argument to mmap), will have to read how to do that :). For the time being I think we can stay with this patch in the Makefile as solo5 is single core.

[hannesm]

I understand #124 superseeds this PR to some degree, but in this PR there are still some review comments left as far as I can tell.

Someone wants to drive the comments from here over to the other PR, so we can close this one and re-review the other one?

[dinosaure]

I think, most of what we said on this PR was integrated into #124. The MacOS compilation is missing I think - but Solo5 already has an issue on such platform. I think we can safely close this PR.

[hannesm]

What about e.g. https://github.com/mirage/ocaml-solo5/pull/122#discussion_r994574501 ?

[hannesm]

Given that #134 moved forward, I'll close this. If this is errorneous, please let me know. As I understand @shym, the review comments from this PR have been addressed in #134.