Closed aantron closed 6 years ago
Hi! Random new-OCaml-learner here, coming from the JavaScript/Node world.
Just came by to say this careful, premeditated handling of breaking changes is extremely heartening, and makes me 10x more excited to join the OCaml community.
Keep up the good work! <3
Well, thanks :)
Here are additional breaking changes, also planned for the future 4.0.0, announced and prepared by the recent 3.2.0 release. To give proper time to adapt, 4.0.0 will be released in three months, around March 2018.
Each breaking change is detailed below, with instructions on how to prepare for it. I will cc the maintainers of affected opam packages after 3.2.0 becomes installable. I've listed the names of opam packages potentially affected under each notice. See if your package is in the lists.
The scheduled breaking changes are still proposals. If you think anything below is a really bad idea, please don't hesitate to object or otherwise comment :) The three-month window is for adaptation and discussion :)
Package lwt.ppx
(#338):
The PPX is being factored out into its own opam package, called lwt_ppx
. This new opam package is available now, as of Lwt 3.2.0.
In Lwt 4.0.0, the old method of linking with the PPX, which is to install package lwt
and refer to the PPX as lwt.ppx
, will no longer work. The PPX will be available only as Findlib package lwt_ppx
in the new opam package of the same name.
To adapt, if you use the PPX, depend on the new opam package lwt_ppx
now, and replace lwt.ppx
by lwt_ppx
in your build system.
lwt.ppx
is used in packages: bitcoin
bookaml
brozip
cstruct
devkit
elasticsearch-cli
eliom
future
gamepad
gdb
gdbprofiler
hardcaml
i3ipc
imaplet-lwt
jsoo_router
jupyter
lambdoc
litiom
maki
multipart-form-data
ocsigen-start
ocurl
ojs-base
otetris
qfs
rawlink
rdf
slacko
socket-daemon
spotify-cli
sqlexpr
stog
testrunner
usbmux
vhd-format
Package lwt.syntax
(#370):
As with the PPX, the Camlp4 syntax is being factored out into its own opam package, lwt_camlp4
. This new opam package is available now, as of Lwt 3.2.0.
In Lwt 4.0.0, it will no longer be possible to use the Camlp4 syntax by installing opam packages camlp4
and lwt
, and referring to the syntax as lwt.syntax
. The Camlp4 syntax will be available only as Findlib package lwt_camlp4
, in the new opam package of the same name.
The Camlp4 syntax is deprecated, and we will likely delete it completely from the Lwt repository after the 4.0.0 release (#511).
To adapt, if you use the Camlp4 syntax, we recommend replacing it by the PPX. However, if you don't wish to do that, depend on the new opam package lwt_camlp4
, and replace lwt.syntax
by lwt_camlp4
in your build system.
lwt.syntax
is used in packages: aws
biocaml
bitcoin
eliom
facebook-sdk
hardcaml
imaplet-lwt
iocaml
iocaml-kernel
iocamljs-kernel
jitsu
joolog
js_of_ocaml
lambdoc
mirage-http-xen
mirari
mongo
ocsigenserver
ojquery
ojwidgets
rdf
release
riak
sqlexpr
themoviedb
tls
usb
xe
xen-disk
xenctrl
xentropyd
Package lwt.log
, modules Lwt_log
and Lwt_daemon
(#484):
The Lwt logging library, consisting of modules Lwt_log
, Lwt_log_core
, Lwt_log_rules
, and Lwt_daemon
, is being deprecated in favor of the logs library, which includes an Lwt-aware module Logs_lwt
.
As part of this, the Lwt logging library is being factored out into its own opam package, lwt_log
. This new opam package is available now, as of Lwt 3.2.0.
In Lwt 4.0.0, it will no longer be possible to use the Lwt logging library by installing opam package lwt
, and linking with Findlib packages lwt.log
and/or lwt.unix
.
To adapt, if you use the logging library, we recommend switching to Logs_lwt
. Otherwise, depend on the new opam package lwt_log
, and resolve any linking problems by adding dependencies on the Findlib package lwt_log
in your build system. If you use Lwt_daemon
, it is also included in the new opam package lwt_log
, but see the discussion starting from this comment.
Lwt_log
or Lwt_daemon
is used in packages: 0install
amqp-client
arakoon
bap-server
bt
camltc
charrua-client
charrua-*
cohttp
conduit
distributed
eliom
ezirmin
gdbprofiler
joolog
js_of_ocaml
jupyter
kinetic-client
lambda-term
lwt-parallel
message-switch
mirage-conduit
mirage-console
mirage-http
nbd
nsq
ocsigen-start
ocsigenserver
ojs-base
opium
ordma
release
shared-block-ring
skkserv-lite
usbmux
utop
uwt
websocket
xentropyd
xen-disk
yurt
>>
syntax in the PPX (#471, #495):
The >>
construct will be removed from the PPX in 4.0.0.
This was originally added to ease porting to the PPX from the Camlp4 syntax. However, it is has proved troublesome – see #495.
To adapt, we recommend replacing all instances of foo >> bar
with foo >>= fun () -> bar
. Otherwise, constrain your project with {"lwt" < "4.0.0"}
.
>>
is used in packages: hardcaml
imaplet-lwt
ocsigenserver
ocurl
release
sqlexpr
usbmux
xen-api-client
Package lwt.preemptive
(#487):
Findlib package lwt.preemptive
is being merged into lwt.unix
in Lwt 4.0.0, and the name lwt.preemptive
is being retired.
In Lwt 3.2.0, lwt.preemptive
is already just an alias for lwt.unix
, and the actual content has been moved into lwt.unix
. So, to adapt, delete all references to lwt.preemptive
in your build system, replacing them with lwt.unix
in places where lwt.unix
is not already listed as a Findlib dependency. This requires that you constrain your project with {"lwt" >= "3.2.0"}
.
lwt.preemptive
is used in packages: 0install
arakoon
baardskeerder
bistro
camltc
ctypes
debian-formats
flow
future
hvsock
imaplet-lwt
maki
ocsigenserver
ordma
osx-cf
osx-fsevents
pvem_lwt_unix
stog
trakeva
unix-dirent
unix-fcntl
unix-sys-stat
unix-unistd
usb
usbmux
vhd-format
vmnet
Lwt.waiter_of_wakener
(#458):
This function should not be used, as it makes contravariance of resolvers ('a Lwt.u
) unsound. To adapt, explicitly retain references to promises that correspond to resolvers.
Module Lwt_sequence
(#361):
Lwt_sequence
, an implementation of ordinary mutable doubly-linked lists, is deprecated (#361). To adapt, switch to an implementation from Containers or Core.
If you need to use Lwt_sequence
because you are using some of the APIs in Lwt_main
, you can suppress deprecation warnings in one of two ways:
e
referring to Lwt_sequence
, wrap it in (e) [@ocaml.warning "-3"]
.To suppress the warning in an entire file, create a new, non-deprecated alias for Lwt_sequence
:
[@@@ocaml.warning "-3"]
module Lwt_sequence = Lwt_sequence
[@@@ocaml.warning "+3"]
Lwt_sequence
is used in packages: arakoon
bt
dns-forward
lambda-term
mirage-net-*
mirage-solo5
mirage-tcpip-*
mirage-unix
mirage-xen
netchannel
nocrypto
notty
ordma
release
shared-memory-ring
sqlexpr
tcpip
usb
utop
utp
vmnet
xen-evtchn
xen-gnt
Rare linking bug (#482):
The layout of the Lwt installation directory will change in 4.0.0. While this is not part of the Lwt API, a few packages depend (or depended) on lwt.unix
incorrectly, by linking only with Findlib package lwt
. Because the library files for lwt
and lwt.unix
are being installed in the same directory in the old layout, Findlib is still able to locate and link with them. This will no longer be the case in 4.0.0.
Some of the affected packages are: biocaml
dns
irmin_watcher
mirage-profile
See https://discuss.ocaml.org/t/1337/13Lwt.bind
(#500):
The semantics of Lwt.bind
, and most other Lwt functions, are being adjusted for better exception and stack safety. Specifically, in 4.0.0, Lwt.bind
on resolved promises will sometimes defer callbacks, instead of always running them immediately.
This can change the order in which callbacks are executed in your program.
The 3.2.0 release retains the "old" semantics. To test your code against the future (4.0.0) semantics, pin Lwt to the safer-semantics
branch:
opam pin add lwt https://github.com/ocsigen/lwt.git\#safer-semantics
Then, rebuild your projects, and run their tests.
If your project does not work with both Lwt 3.x.x and safer-semantics
(4.0.0), the simplest thing to do is to constrain it by adding {"lwt" < "4.0.0"}
to its opam
file. However, we encourage adjusting it, so that it works with both 3.x.x and safer-semantics
(4.0.0).
Since this potentially affects all packages, we are announcing this change on forums, mailing lists, etc., and adding a message to Lwt's opam
file.
cc @agarwal @akabe @andersfugmann @andrenth @andrewray @anyakichi @Armael @avsm @balat @c-cube @copy @cyberhuman @darioteixeira @delamonpansie @diml @djs55 @dominicjprice @domsj @dsheets @edwintorok @emillon @essdotteedot @fdopen @fxfactorial @gaborigloi @gildor478 @gregtatcam @haesbaert @hannesm @hhugo @ivg @johnelse @jrochel @kayceesrk @Leonidas-from-XIV @lindig @MassD @mato @metadave @mfp @mseri @msimon
cc @nojb @pqwy @pveber @rgrinberg @ryanslade @samoht @seliopou @smondet @talex5 @toolslive @vasilisp @vbmithr @vouillon @vzaliva @xvw @yallop @ygrek @yomimono @zoggy @zshipko
Please see the above comment, with instructions on how to adapt to future changes in Lwt 4.0.0, which will be made in three months. Mostly, adapting will involve adding new dependencies to opam files, due to changed packaging of Lwt helper libraries.
@aantron can you list the affected packages?
What command do you use to estimate what packages are impacted by this change?
Is this something like:
$> opam list --depends-on lwt
@gildor478 The affected packages for what? Each specific change lists the affected packages under the description already. Are you referring to something else?
The command is
opam list --unavailable --depends-on lwt --depopts --recursive --short
@aantron Thanks. The list of affected packages by breaking changes is perfect.
I have received first the comment where I was CCed and I was wondering how to act on it. It looked like a long list of sorted names. You may want to improve the CC message with something like
@gildor478 for debian-formats, @mfp for sqlexpr, @agarwal for biocaml, [...]
This can help people to know what they should change.
Anyway, thanks. I'll change ocaml-debian-formats accordingly.
Thanks. I'll probably have to do that for the next time we have breaking changes.
The Lwt.bind
change will most likely not happen. Please see https://discuss.ocaml.org/t/1337/13 for details. Hopefully, this makes maintenance easier for everyone, and thanks for all feedback :)
That's a shame. Sounds like this change would have made bind
behave consistently and correctly. Plus, bumping the major version implies API changes that may not be backward-compatible. If people are worried, they can continue to use the 3.2.x branch until they've tested thoroughly against 4.0.x. After all, OPAM reports available versions all the way back to 2.3.2 so 3.2.x isn't going away soon.
That's a shame. Sounds like this change would have made bind behave consistently and correctly. Plus, bumping the major version implies API changes that may not be backward-compatible. If people are worried, they can continue to use the 3.2.x branch until they've tested thoroughly against 4.0.x. After all, OPAM reports available versions all the way back to 2.3.2 so 3.2.x isn't going away soon.
The majority of big projects, that are serious stakeholders of Lwt, will then may choose to continue to use Lwt 3.x, that will eventually lead to a nasty schism, like gnome 2.x vs gnome 3.x, etc. This will, in the best case scenario, double the support burden, and, in the worst one will make many packages uninstallable, or create yet another branch in the already too complex "which async library to use" decision tree. Given that in the real life, the noted problem with the bind semantics, is not often observable (if at all), I don't think that we should make this step.
Also, I would suggest everyone to refrain yourself from using exceptions together with Lwt (or any other async computations) and use variants to denote failed computations. Under such model, the bind function has perfectly fine and sound semantics.
We may still be able to make the Lwt.bind
change, but with more preparation, to make it easier for users to adapt. For example, we could add a testing mode to Lwt that randomizes the deferral strategy to help discover mistaken assumptions about it.
However, I'm not sure what the ultimate value of this would be, given it's still work for users, it can't be flagged statically, and the semantic problem is not a problem for users in practice.
All the semantic problem means is for developers of Lwt, we have to do extra work when editing any core Lwt API, to ensure that it interacts safely with bind
. But we almost never change the core Lwt API anyway. Fixing Lwt.bind
would make it possible to analyze its behavior in isolation from other APIs, which is worthwhile, but probably not worth breaking all users for.
I agree strongly with @ivg about avoiding exceptions. Reflecting a lot on experience with Lwt and JS promises, my overall opinion is that rejections and asynchronous exceptions are both pretty ill-conceived mechanisms, almost as poor as cancelation. While it would be nice if these things worked well, because exceptions need to go somewhere and we don't want to handle them manually each time, each actual design that I've seen or come up with so far seems to produce undesirable consequences.
As @ivg points out, even though Lwt supports rejection, a user that is aware of its problems is likely to be using explicit error handling, in which case the Lwt.bind
problem "accidentally" becomes irrelevant.
Lwt_result
can help with this.
Async provides an Eager_deferred
module that can be opened to shadow bind
and friends with their eager (lwt-like) versions. Maybe a similar solution could be provided by lwt to opt-in for safe monads. That is assuming it is possible to have both styles of promises in the same process and the implementation complexity is reasonable.
We could eventually do that. It should be very easy to get both styles of API exposed. The changes from #500 support this.
Closing this, as 4.0.0 is now merged into opam (ocaml/opam-repository#11708). See also the main announcement on Discourse.
Further discussion, questions, etc., are welcome, of course :)
Great work! However, the documentation and tutorials should reflect the new project structure as well, especially with regard to lwt.ppx versus lwt_ppx.
@vog thanks for catching and reporting it! I (think I) fixed all remaining lwt.ppx
in 0a1863d just now. The docs site is a bit behind on picking up Lwt changes. That is being tracked in #576 for now.
Lwt 4.0.0 will be released
towards the end of 2017in March 2018 with minor breaking changes to packaging, and perhaps several APIs. Hopefully, there will be less breakage than in 3.0.0.The current release, 3.1.0, prepares to delete some modules. Another planned release, 3.2.0, will schedule additional breaking changes for 4.0.0. A comment will be added to this issue when 3.2.0 is released, with details and instructions about those additional plans.
lwt.simple-top
(#371). This has been deprecated for years, in favor ofutop
. It has no users in OPAM, and users found in a GitHub search have already been notified.Lwt_chan
(packagelwt.unix
) (#441). This has been deprecated for nine years in favor ofLwt_io
, which subsumes it completely. GitHub and OPAM package maintainers have been notified in the linked issue.