Moving to OCaml 4.10.x - Githubissues

lindig commented 3 years ago

If you are interested in supporting the toolstack implementation of XCP-NG/Citrix Hypervisor, here is something we from Citrix would be interested in: moving to OCaml 4.10.x.

Moving to OCaml 4.10.x involves two steps: updating the ocaml.spec file to have the new compiler available in the RPM build system. The Fedora ocaml.spec could be a starting point. The spec file includes some patches some of which can be dropped because the 4.10.x includes fixes.
Before embarking on the point above it would be a good idea to use OCaml 4.10.x to compile all toolstack components in https://github.com/xapi-project/xs-opam/ using the new compiler and to update libraries as needed. The CI for xs-opam is already configured to use OCaml 4.08 and 4.09 and something similar could be done for OCaml 4.10 to identify what libraries and code needs to be updated. I expect that several libraries will require updates and that it would be a good idea to do this on a separate branch as I don't expect these changes to be compatible with OCaml 4.08.1 that is currently in use.

psafont commented 3 years ago

Note that we're behind regarding upstream libraries available on the official repositories:

$ env OPAMNO=true opam upgrade | grep -E "↗|∗"
  ↗ upgrade   conf-pkg-config         1.1 to 1.2
  ↗ upgrade   astring                 0.8.3 to 0.8.4
  ↗ upgrade   dune                    2.5.0 to 2.6.1
  ↗ upgrade   conf-libssl             1 to 2
  ↗ upgrade   sexplib0                v0.12.0 to v0.14.0
  ↗ upgrade   result                  1.4 to 1.5
  ↗ upgrade   qcheck-core             0.9 to 0.13
  ↗ upgrade   ppx_tools               6.0+4.08.0 to 6.2
  ↗ upgrade   ounit2                  2.2.2 to 2.2.3
  ↗ upgrade   menhirSdk               20200211 to 20200624
  ↗ upgrade   menhirLib               20200211 to 20200624
  ↗ upgrade   macaddr                 3.1.0 to 5.0.0
  ↗ upgrade   jane-street-headers     v0.12.0 to v0.14.0
  ↗ upgrade   integers                0.3.0 to 0.4.0
  ↗ upgrade   dune-private-libs       2.5.0 to 2.6.1
  ↗ upgrade   dune-build-info         2.5.0 to 2.6.1
  ↗ upgrade   cstruct                 5.1.1 to 5.2.0
  ↗ upgrade   alcotest                1.1.0 to 1.2.1
  ↗ upgrade   ocaml-migrate-parsetree 1.7.1 to 1.7.3
  ↗ upgrade   ounit                   2.2.2 to 2.2.3
  ↗ upgrade   menhir                  20200211 to 20200624
  ↗ upgrade   ctypes                  0.16.0 to 0.17.1
  ↗ upgrade   dune-configurator       2.5.0 to 2.6.1
  ↗ upgrade   eqaf                    0.6 to 0.7
  ↗ upgrade   cstruct-unix            5.1.1 to 5.2.0
  ↗ upgrade   ipaddr                  3.1.0 to 5.0.0
  ↗ upgrade   angstrom                0.12.1 to 0.14.1
  ↗ upgrade   ppx_tools_versioned     5.3.0 to 5.4.0
  ↗ upgrade   qcheck-ounit            0.9 to 0.13
  ↗ upgrade   lwt                     4.5.0 to 5.3.0
  ↗ upgrade   base                    v0.12.2 to v0.14.0
  ↗ upgrade   mirage-crypto           0.6.2 to 0.8.1
  ↗ upgrade   bisect_ppx              1.4.1 to 2.4.1
  ↗ upgrade   qcheck                  0.9 to 0.13
  ↗ upgrade   cstruct-lwt             5.1.1 to 5.2.0
  ↗ upgrade   variantslib             v0.12.0 to v0.14.0
  ↗ upgrade   typerep                 v0.12.0 to v0.14.0
  ↗ upgrade   stdio                   v0.12.0 to v0.14.0
  ↗ upgrade   parsexp                 v0.12.0 to v0.14.0
  ↗ upgrade   fieldslib               v0.12.0 to v0.14.0
  ↗ upgrade   rpclib                  6.1.0 to 7.1.0
  ↗ upgrade   mirage-crypto-rng       0.6.2 to 0.8.1
  ↗ upgrade   alcotest-lwt            1.1.0 to 1.2.1
  ↗ upgrade   ppxlib                  0.8.1 to 0.13.0
  ↗ upgrade   sexplib                 v0.12.0 to v0.14.0
  ↗ upgrade   rpclib-lwt              6.1.0 to 7.1.0
  ∗ install   ppx_fixed_literal       v0.14.0              [required by ppx_jane]
  ∗ install   ppx_cold                v0.14.0              [required by ppx_assert, ppx_base]
  ↗ upgrade   ppx_variants_conv       v0.12.0 to v0.14.0
  ↗ upgrade   ppx_typerep_conv        v0.12.0 to v0.14.0
  ↗ upgrade   ppx_stable              v0.12.0 to v0.14.0
  ↗ upgrade   ppx_sexp_conv           v0.12.0 to v0.14.0
  ↗ upgrade   ppx_pipebang            v0.12.0 to v0.14.0
  ↗ upgrade   ppx_optional            v0.12.0 to v0.14.0
  ↗ upgrade   ppx_optcomp             v0.12.0 to v0.14.0
  ↗ upgrade   ppx_let                 v0.12.0 to v0.14.0
  ↗ upgrade   ppx_js_style            v0.12.0 to v0.14.0
  ↗ upgrade   ppx_here                v0.12.0 to v0.14.0
  ↗ upgrade   ppx_fields_conv         v0.12.0 to v0.14.0
  ↗ upgrade   ppx_enumerate           v0.12.0 to v0.14.0
  ↗ upgrade   ppx_deriving_rpc        6.1.0 to 7.1.0
  ↗ upgrade   ppx_compare             v0.12.0 to v0.14.0
  ↗ upgrade   ppx_cstruct             5.1.1 to 5.2.0
  ∗ install   ipaddr-sexp             5.0.0                [required by conduit, conduit-lwt-unix]
  ↗ upgrade   ppx_custom_printf       v0.12.1 to v0.14.0
  ↗ upgrade   mirage-crypto-pk        0.6.2 to 0.8.1
  ↗ upgrade   ppx_sexp_value          v0.12.0 to v0.14.0
  ↗ upgrade   ppx_sexp_message        v0.12.0 to v0.14.0
  ↗ upgrade   ppx_fail                v0.12.0 to v0.14.0
  ↗ upgrade   ppx_hash                v0.12.0 to v0.14.0
  ↗ upgrade   ppx_assert              v0.12.0 to v0.14.0
  ↗ upgrade   conduit                 1.4.0 to 2.1.0
  ↗ upgrade   bin_prot                v0.12.0 to v0.14.0
  ↗ upgrade   x509                    0.11.0 to 0.11.2
  ↗ upgrade   cohttp                  2.3.0 to 2.5.4
  ↗ upgrade   ppx_base                v0.12.0 to v0.14.0
  ↗ upgrade   jst-config              v0.12.0 to v0.14.0
  ↗ upgrade   conduit-lwt             1.4.0 to 2.1.0
  ↗ upgrade   ppx_bin_prot            v0.12.1 to v0.14.0
  ↗ upgrade   cohttp-lwt              2.3.0 to 2.5.4
  ∗ install   ppx_string              v0.14.0              [required by ppx_jane]
  ↗ upgrade   time_now                v0.12.0 to v0.14.0
  ↗ upgrade   conduit-lwt-unix        1.4.0 to 2.2.2
  ↗ upgrade   ppx_module_timer        v0.12.0 to v0.14.0
  ↗ upgrade   ppx_inline_test         v0.12.0 to v0.14.0
  ↗ upgrade   cohttp-lwt-unix         2.3.0 to 2.5.4
  ↗ upgrade   ppx_expect              v0.12.0 to v0.14.0
  ↗ upgrade   ppx_bench               v0.12.0 to v0.14.0
  ↗ upgrade   splittable_random       v0.12.0 to v0.14.0
  ↗ upgrade   base_quickcheck         v0.12.1 to v0.14.0
  ↗ upgrade   ppx_jane                v0.12.0 to v0.14.0
  ↗ upgrade   base_bigstring          v0.12.0 to v0.14.0
  ↗ upgrade   core_kernel             v0.12.3 to v0.14.0
  ∗ install   timezone                v0.14.0              [required by core]
  ↗ upgrade   protocol_version_header v0.12.0 to v0.14.0
  ↗ upgrade   async_kernel            v0.12.0 to v0.14.0
  ↗ upgrade   core                    v0.12.4 to v0.14.0
  ↗ upgrade   async_rpc_kernel        v0.12.0 to v0.14.0
  ↗ upgrade   async_extra             v0.12.0 to v0.14.0
  ↗ upgrade   textutils               v0.12.0 to v0.14.0
  ↗ upgrade   async_unix              v0.12.0 to v0.14.0
  ↗ upgrade   async                   v0.12.0 to v0.14.0
  ↗ upgrade   rpclib-async            6.1.0 to 7.1.0
  ↗ upgrade   conduit-async           1.4.0 to 2.1.0
  ↗ upgrade   async_find              v0.12.0 to v0.14.0
  ↗ upgrade   cohttp-async            2.3.0 to 2.5.4
  ↗ upgrade   async_inotify           v0.12.0 to v0.14.0
===== ∗ 5   ↻ 133   ↗ 102 =====

Looking at their metadata it looks like they should be compatible with 4.10 as well, but it's not sure and some might need upgrading to be compatible with the new version.

olivierlambert commented 3 years ago

First, thanks a lot for asking this, we are really excited here to provide/package "cutting edge" stuff into XCP-ng. Thanks to our testing repositories, we might be able to provide a "full real case" scenario of usage with latest XAPI code to impatient/curious users.

It also might be interesting for some users/customers having issues with memory leaks. Testing packages from the repo of the same name is trivial, and thanks to yum, in general, it's painless to rollback (we'll have to check that if it's true for XAPI).

Let's move forward :+1:

stormi commented 3 years ago

I have imported Fedora's ocaml into https://github.com/xcp-ng-rpms/ocaml/tree/8.2-ocaml-4.10 See the diff from previous: https://github.com/xcp-ng-rpms/ocaml/commit/496f3e90495c3aca5af46f82ea4610fef6474e01

There are several unknowns to me: CA patches to drop or keep? Handling of the differences between XS/XCP-ng's previous spec file and that new one. I've done a few adaptations that seemed obvious to me, but also put questions as comments and I don't know if the files being removed from the package are to be expected.

Built RPMs as well as build logs are available at https://koji.xcp-ng.org/taskinfo?taskID=16764

lindig commented 3 years ago

The CA patches can be dropped because these issues are fixed in the compiler.
I believe there are some RedHat specific hardening steps and macros (redhat-rpm-config) that I would not use.
For now I think its more important to have a compiler but work on the libraries. The diff looks otherwise sensible to me.

stormi commented 3 years ago

I've opened a draft PR for the libraries work. I reached the point where the minimum amount of libraries have been updated (dozens, already) for compatibility with 4.10 while retaining compatibility with 4.08.

Now there are build failures to handle.

psafont commented 3 years ago

4.10.1 is being integrated into the ci of the ocaml libraries repository: https://github.com/xapi-project/xs-opam/pull/514

This makes it even, slower so I've changed how the Ci is run to reduce runtime: https://github.com/xapi-project/xs-opam/pull/515

After that I'll integrate 4.10.1 into the RPM build system and test it before we drop 4.08.1.

psafont commented 3 years ago

Because of the large amount of changes needed to adopt 4.10 in existing ocaml libraries, Sam and I don't think it's feasable to get a build working on top of xcp-ng 8.2.

That would be better served for a build based on the latest commits in the toolstack. The integration into the citrix hypervisor is looking good and unless there are unexpected issues it should be available in future releases.

I'd like to thank @stormi for his work on the opam repository and in the spec file for ocaml, it's was necessary to get this effort rolling.

xcp-ng / xcp

Moving to OCaml 4.10.x #422