11.0 release preparation

[gasche]

@whitequark this is a thread to discuss the 11.x release

What I have to do before the release is to reorder the Changelog to keep a trace of 10.x, yet clarify to users that all the features since 9.x are, practically speaking, new in 11.x.

I wonder what we should do (and have the time/energy to do) on the testing front. The CI is already helpful (tahnks!), and obviously if we break OPAM we will know pretty soon, but there is no easy way for me to test a development version before the release.

[dbuenzli]

but there is no easy way for me to test a development version before the release.

Is it still not possible to reliably pin ocamlbuild ? I can certainly try to live on a pin for the next few days and check my opam reinstall $(opam list -s --installed --depends-on ocamlbuild) install base behaves correctly. Just give me a reliable tag or commit to pin to.

[gasche]

Yes, opam pin add ocamlbuild . should work correctly. (I don't remember when in the post-split past this would not have been possible.)

[whitequark]

Is it still not possible to reliably pin ocamlbuild ?

It is perfectly possible and in fact our CI relies on that.

[whitequark]

@gasche I have found one instance of an affected downstream package. Ironically, my own, ppx_deriving. I am not calling this a "regression" because it is merely exposing a bug that already was there; I would be inclined to call the previous behavior a bug.

Look here: https://github.com/whitequark/ppx_deriving/commit/c2fb119f#diff-7c9f76fe069008cd250e1fb989cb43ff. Before 0.11.0, this .mllib file got ignored (?! why). Now it is not, and ppx_deriving_make.cma correctly packs only ppx_deriving_create.cma...

[whitequark]

This is possibly related to https://github.com/ocaml/ocamlbuild/pull/140, https://github.com/ocaml/ocamlbuild/issues/166 and https://github.com/ocaml/ocamlbuild/issues/42...

[whitequark]

@gasche BTW, since the CI is green now: I went ahead and, for master, disabled force pushes, and added a requirement that the build pass before a PR can be merged.

[whitequark]

@gasche I've now installed, with --build-test, a fairly large amount of packages:

# Installed packages for 4.04.0+ocamlbuildtest:
alcotest                          0.7.2  Alcotest is a lightweight and colourful test framework.
astring                           0.8.3  Alternative String module for OCaml
atd                               1.2.0  Parser for the ATD data format description language
atdgen                           1.10.0  Generates efficient JSON serializers, deserializers and validators
base-bigarray                      base  Bigarray library distributed with the OCaml compiler
base-bytes                         base  Bytes library distributed with the OCaml compiler
base-threads                       base  Threads library distributed with the OCaml compiler
base-unix                          base  Unix library distributed with the OCaml compiler
base64                            2.0.0  Base64 encoding and decoding library
batteries                         2.5.3  a community-maintained standard library extension
bin_prot                 113.33.00+4.03  A binary protocol generator
biniou                           1.0.12  Binary data format designed for speed, safety, ease of use and backward compatibility as protocols evolve
bos                               0.1.4  Basic OS interaction for OCaml
camlp4                           4.04+1  Camlp4 is a system for writing extensible parsers for programming languages
camomile                          0.8.5  A comprehensive Unicode library
cmdliner                          0.9.8  Declarative definition of command line interfaces for OCaml
cohttp                           0.21.1  HTTP(S) library for Lwt, Async and Mirage
conduit                          0.15.0  Network connection library for TCP and SSL
conf-gmp                              1  Virtual package relying on a GMP lib system installation.
conf-gmp-powm-sec                     1  Virtual package relying on a GMP lib with constant-time modular exponentiation.
conf-libpcre                          1  Virtual package relying on a libpcre system installation.
conf-m4                               1  Virtual package relying on m4
conf-openssl                          1  Virtual package relying on an OpenSSL system installation.
conf-perl                             1  Virtual package relying on perl
conf-pkg-config                     1.0  Virtual package relying on pkg-config installation.
conf-which                            1  Virtual package relying on which
conf-zlib                             1  Virtual package relying on zlib
containers                          1.0  A modular, clean and powerful extension of the OCaml standard library.
core                     113.33.02+4.03  Industrial strength alternative to OCaml's standard library
core_kernel              113.33.02+4.03  Industrial strength alternative to OCaml's standard library
cppo                              1.4.1  Equivalent of the C preprocessor for OCaml programs
cryptokit                          1.11  Cryptographic primitives library. 
cstruct                           2.3.1  access C structures via a camlp4 extension
cudf                                0.7  CUDF library (part of the Mancoosi tools)
dbm                                 1.0  Binding to the NDBM/GDBM Unix "databases"
depext                            1.0.2  Query and install external dependencies of OPAM packages
deriving                          0.7.1  Extension to OCaml for deriving functions from type declarations
dose                         3.2.2+opam  Dose library (part of Mancoosi tools)
easy-format                       1.2.0  High-level and functional interface to the Format module of the OCaml standard library
eliom                             6.2.0  Client/server Web framework
expect                            0.0.4  Simple implementation of "expect" to help building unitary testing of interactive program
extlib                            1.7.1  A complete yet small extension for OCaml standard library (reduced, recommended)
fieldslib                     113.24.00  Syntax extension to define first class values representing record fields, to get and set record fields, iterate and 
fileutils                         0.5.1  Library to provide pure OCaml functions to manipulate real file (POSIX like) and filename.
fmt                               0.8.0  OCaml Format pretty-printer combinators
fpath                             0.7.1  File system paths for OCaml
functoria                         2.0.1  A bundle of useful runtime functions for applications built with Functoria.
functoria-runtime                 2.0.0  
github                            1.1.0  GitHub APIv3 client bindings
inotify                             2.3  Inotify bindings for ocaml.
ipaddr                            2.7.2  IP (and MAC) address manipulation
js_of_ocaml                       2.8.4  Compiler from OCaml bytecode to Javascript
jsonm                             0.9.1  Non-blocking streaming JSON codec for OCaml
lambda-term                      1.10.1  Terminal manipulation library for OCaml
logs                              0.6.2  Logging infrastructure for OCaml
lwt                               2.7.0  Monadic promises and concurrent I/O
magic-mime                        1.0.0  Convert file extensions to MIME types
menhir                         20170101  LR(1) parser generator
mirage                            3.0.0  The MirageOS library operating system
mirage-runtime                    3.0.0  A bundle of useful runtime functions for applications built with Mirage.
mtime                             0.8.4  Monotonic wall-clock time for OCaml
oasis                             0.4.8  Tooling for building OCaml libraries and applications
ocamlbuild                          dev (pinned)  OCamlbuild is a build system with builtin rules to easily build most OCaml projects.
ocamlfind                         1.7.1  A library manager for OCaml
ocamlgraph                        1.8.5  A generic graph library for OCaml
ocamlify                          0.0.1  Include files in OCaml code
ocamlmod                          0.0.8  Generate OCaml modules from source files
ocamlnet                          4.1.2  Internet protocols (HTTP, CGI, e-mail etc.) and helper data structures
ocplib-endian                       1.0  Optimised functions to read and write int16/32/64 from strings and bigarrays, based on new primitives added in versi
ocsigenserver                       2.8  A full-featured and extensible Web server
omake                            0.10.2  Build system designed for scalability and portability
opam-lib                          1.2.2  The OPAM library
opam-publish           0.3.0+transition  opam-publish transition package
opam-query                          1.3 (pinned)  A tool to query opam files from shell scripts
optcomp                             1.6  Optional compilation with cpp-like directives
ounit                             2.0.0  Unit testing framework loosely based on HUnit. It is similar to JUnit, and other XUnit testing frameworks
pcre                              7.2.3  pcre-ocaml - bindings to the Perl Compatibility Regular Expressions library
ppx_assert                    113.33.00  Assert-like extension nodes that raise useful errors on failure
ppx_bench                113.33.00+4.03  Syntax extension for writing in-line benchmarks in ocaml code
ppx_bin_prot             113.33.00+4.03  Generation of bin_prot readers and writers from types
ppx_compare              113.33.00+4.03  Generation of comparison functions from types
ppx_const                           1.1  Compile-time "if" statement for conditional inclusion of code.
ppx_core                 113.33.01+4.03  Standard library for ppx rewriters
ppx_custom_printf        113.33.00+4.03  Printf-style format-strings for user-defined string conversion
ppx_deriving                        4.1  Type-driven code generation for OCaml >=4.02
ppx_deriving_protobuf               2.5  A Protocol Buffers codec generator for OCaml >=4.02
ppx_deriving_yojson                 3.0  JSON codec generator for OCaml >=4.02
ppx_driver               113.33.02+4.03  Feature-full driver for OCaml AST transformers
ppx_enumerate            113.33.00+4.03  Generate a list containing all values of a finite type
ppx_expect               113.33.01+4.03  Cram like framework for OCaml
ppx_fail                 113.33.00+4.03  Add location to calls to failwiths
ppx_fields_conv          113.33.00+4.03  Generation of accessor and iteration functions for ocaml records
ppx_here                      113.33.00  Expands [%here] into its location
ppx_import                          1.1  A syntax extension for importing declarations from interface files
ppx_include                         1.1  Include OCaml source files in each other
ppx_inline_test          113.33.00+4.03  Syntax extension for writing in-line tests in ocaml code
ppx_jane                      113.33.00  Standard Jane Street ppx rewriters
ppx_let                  113.33.00+4.03  Monadic let-bindings
ppx_optcomp              113.33.01+4.03  Optional compilation for OCaml
ppx_pipebang             113.33.00+4.03  A ppx rewriter that inlines reverse application operators `|>` and `|!`
ppx_sexp_conv            113.33.01+4.03  Generation of S-expression conversion functions from type definitions
ppx_sexp_message         113.33.00+4.03  A ppx rewriter for easy construction of s-expressions
ppx_sexp_value           113.33.00+4.03  A ppx rewriter that simplifies building s-expressions from ocaml values
ppx_tools                           5.0  Tools for authors of ppx rewriters and other syntactic tools
ppx_type_conv            113.33.02+4.03  Support Library for type-driven code generators
ppx_typerep_conv         113.33.00+4.03  Generation of runtime types from type declarations
ppx_variants_conv        113.33.00+4.03  Generation of accessor and iteration functions for ocaml variant types
publish                           0.3.2  A tool to ease contributions to opam repositories.
qcheck                          0.5.3.1  QuickCheck inspired property-based testing for OCaml.
qtest                               2.5  iTeML / qtest : Inline (Unit) Tests for OCaml.
re                                1.7.1  RE is a regular expression library for OCaml
react                             1.2.0  Declarative events and signals for OCaml
reactiveData                      0.2.1  Functional reactive programming with incremental changes in data structures
result                              1.2  Compatibility Result module
rresult                           0.5.0  Result value combinators for OCaml
sexplib                  113.33.00+4.03  Library for serializing OCaml values to and from S-expressions
ssl                               0.5.3  Bindings for OpenSSL
stringext                         1.4.3  Extra string functions for OCaml
topkg                             0.8.1  The transitory OCaml software packager
typerep                       113.24.00  typerep is a library for runtime types.
tyxml                             4.0.1  TyXML is a library for building statically correct HTML5 and SVG documents
uchar                             0.0.1  Compatibility library for OCaml's Uchar module
uint                              1.2.0  Unsigned ints for OCaml
uri                               1.9.2  RFC3986 URI/URL parsing library
utop                             1.19.3  Universal toplevel for OCaml
uutf                              0.9.4  Non-blocking streaming Unicode codec for OCaml
variantslib                   113.24.00  Part of Jane Street's Core library
yojson                            1.3.3  Yojson is an optimized parsing and printing library for the JSON format
zarith                            1.4.1  Implements arithmetic and logical operations over arbitrary-precision integers
zed                                 1.4  Abstract engine for text edition in OCaml

This gives me confidence that nothing is obviously broken, including in usage with corebuild, oasis or topkg.

[gasche]

Thanks a lot, this is very helpful.

[gasche]

I propose the Changes change in #178 (usually I would just commit Changes directly, but this one is arguably a bit more delicate, and I am also crediting @whitequark for some of the recent work, so I would like to make sure that this is right). Then we need to converge on #160 and merge it, maybe let @whitequark do some -toolchain testing if that makes sense, and as far as I am concerned we are good to go for the release process.

[whitequark]

@gasche Can you go through https://github.com/ocaml/ocamlbuild/issues?q=is%3Aissue+is%3Aopen+label%3Atriage and state your opinion there? (The idea of the "triage" tag is that an issue for which a decision can and should be made, as opposed to implementing something, is marked with it so that it is not lost.)

[gasche]

I (reluctantly) went through them. Thank you for forcing me to face a disconcerting bugtracker once in a while. Yet I think that we don't need to consider more issues before deciding to release 0.11; we have enough features/changes on our plate already, so why wait or increase the risk?

[gasche]

I went for #202 as it looked like a low-hanging fruit (with no regression risk whatsoever), but after that I think we should just release. (Some missing Menhir flags are on my mind, but oh well.)

[whitequark]

Sure, sounds good.

[gasche]

I pushed the release and sent it to opam-repository: https://github.com/ocaml/opam-repository/pull/8625

[fdopen]

One not longer compiling program: opam install obus lablgtk 0install (0install with all depopts).

It did compile with ocamlbuild.0.9.3 (and passed all test cases), it doesn't even compile with ocamlbuild.0.11.0

### stdout ###
# [...]
# touch gui_gtk.mli  ; if  ocamlfind ocamlopt -pack -g -bin-annot -I gui_gtk gui_gtk/gtk_common.cmx gui_gtk/alert_box.cmx gui_gtk/gtk_utils.cmx gui_gtk/add_box.cmx gui_gtk/help_box.cmx gui_gtk/unsorted_list.cmx gui_gtk/cache_explorer_box.cmx gui_gtk/app_list_box.cmx gui_gtk/bug_report_box.cmx gui_gtk/limiter.cmx gui_gtk/preferences_box.cmx gui_gtk/component_box.cmx gui_gtk/component_tree.cmx gui_gtk/icon_cache.cmx gui_gtk/tray_icon.cmx gui_gtk/solver_box.cmx gui_gtk/trust_box.cmx gui_gtk/gui_progress.cmx gui_gtk/gui_gtk_plugin.cmx -o gui_gtk.cmx  ; then  rm -f gui_gtk.mli  ; else  rm -f gui_gtk.mli  ; exit 1; fi
# ocamlfind ocamlopt -a -thread -package curl.lwt -thread /home/andreas/.opam/4.03.0/lib/lablgtk2/lablgtk.cmxa /home/andreas/.opam/4.03.0/lib/lwt/lwt-glib.cmxa -package sha -package dynlink -package obus,obus.notification,obus.network-manager -package lablgtk2 -package yojson,xmlm,str,lwt,lwt.unix,lwt.react,lwt.preemptive,extlib,curl,dynlink gui_gtk.cmx -o gui_gtk.cmxa
# + ocamlfind ocamlopt -a -thread -package curl.lwt -thread /home/andreas/.opam/4.03.0/lib/lablgtk2/lablgtk.cmxa /home/andreas/.opam/4.03.0/lib/lwt/lwt-glib.cmxa -package sha -package dynlink -package obus,obus.notification,obus.network-manager -package lablgtk2 -package yojson,xmlm,str,lwt,lwt.unix,lwt.react,lwt.preemptive,extlib,curl,dynlink gui_gtk.cmx -o gui_gtk.cmxa
# Option -a cannot be used with .cmxa input files.
# Command exited with code 2.
# Makefile:33: recipe for target 'ocaml' failed
# make[2]: Leaving directory '/home/andreas/.opam/4.03.0/build/0install.2.12/ocaml'
# /home/andreas/.opam/4.03.0/build/0install.2.12/Makefile.build:20: recipe for target 'all' failed
# make[1]: Leaving directory '/home/andreas/.opam/4.03.0/build/0install.2.12/build'
# Makefile:53: recipe for target 'all' failed
### stderr ###
# make[2]: *** [ocaml] Error 10
# make[1]: *** [all] Error 2
# make: *** [all] Error 2

[dbuenzli]

@fdopen could you please provide a larger build log. I tried to follow a bit the build system but got stuck at finding all-${TYPE}.itarget files in the repo. Are you positive the above failure is an ocamlbuild invocation ?

/cc @talex5

[dbuenzli]

Ah they are defined as rules here : https://github.com/0install/0install/blob/master/ocaml/myocamlbuild.ml

I suspect something related to tag definitions that injects archives on the cli in the wrong context.

[fdopen]

Yes, I think it is ocamlbuild. While I didn't look at the build instructions in detail, I had to add -classic-display in order to get a decent log on the screen 😛

Logs: https://gist.github.com/fdopen/297aca821e5c727c371bf06660b8d6cb

[dbuenzli]

Thanks. I suspect these two flag lines are too general.

[gasche]

(Thanks, @dbuenzli, for your help with the release!)

This looks similar to the ppx_deriving failure in https://github.com/ocaml/ocamlbuild/issues/176#issuecomment-282903942 : reordering the .mllib rules means that now the .mllib is taken into acount, so different build rules apply, and this reveals bugs in part of the build system that were previously ignored.

[gasche]

Thanks. I suspect these two flag lines are too general.

This was, of course, exactly right, and I fixed them in https://github.com/0install/0install/pull/57 .