handle poor network performance better

[mor1]

(carving up the issues i raised on opam-devel)

when trying to reinstall opam on a sequence of poor connections (on-train wifi etc) i hit timeouts that stopped package installs partway through so had to be restarted, sometimes after having removed earlier versions of the packages (leaving me in an increasingly broken state until i could get sufficiently through the install).

[mor1]

don't know what happened -- i was doing this on my home broadband which is usually pretty reliable -- but had a similar experience just now. a clean opam init, followed by setting a remote and installing mirage. installing io-page fails for some unknown network reason, apparently can't be downloaded. but when i retry with --verbose, it downloads but then fails to install because it's already installed...?!

: mort@greyjay:~$; opam remote
   0 [http]     default     https://opam.ocaml.org
: mort@greyjay:~$; opam remote add -k git mirage-split     https://github.com/mirage/opam-repository#mirage-1.1.0
mirage-split Fetching https://github.com/mirage/opam-repository#mirage-1.1.0
^CUpdating ~/.opam/repo/compiler-index ...
Updating ~/.opam/compilers/ ...
Updating ~/.opam/repo/package-index ...
Updating ~/.opam/packages/ ...
no-proxyUpdating the cache of metadata (~/.opam/state.cache) ...
'opam remote add -k git mirage-split https://github.com/mirage/opam-repository#mirage-1.1.0' failed.
: mort@greyjay:~$; no-proxy
: mort@greyjay:~$; opam remote add -k git mirage-split     https://github.com/mirage/opam-repository#mirage-1.1.0
mirage-split Fetching https://github.com/mirage/opam-repository#mirage-1.1.0
Updating ~/.opam/repo/compiler-index ...
Updating ~/.opam/compilers/ ...
Updating ~/.opam/repo/package-index ...
Updating ~/.opam/packages/ ...
Updating the cache of metadata (~/.opam/state.cache) ...
: mort@greyjay:~$; opam upgrade -u
opam: unknown option `-u'
Usage: opam upgrade [OPTION]... [PACKAGES]...
Try `opam upgrade --help' or `opam --help' for more information.
: mort@greyjay:~$; opam update -u
mirage-split Fetching https://github.com/mirage/opam-repository#mirage-1.1.0
default    Downloading https://opam.ocaml.org/urls.txt
Downloading https://opam.ocaml.org/repo
Updating ~/.opam/repo/compiler-index ...
Updating ~/.opam/compilers/ ...
Updating ~/.opam/repo/package-index ...
Updating ~/.opam/packages/ ...
Updating the cache of metadata (~/.opam/state.cache) ...
Everything is up-to-date.
Already up-to-date.
: mort@greyjay:~$; opam install mirage
The following actions will be performed:
 - install ocamlfind.1.4.0 [required by mirage]
 - install cmdliner.0.9.2 [required by mirage]
 - install ipaddr.2.2.0 [required by mirage]
 - install lwt.2.4.4 [required by mirage]
 - install optcomp.1.5 [required by mirage]
 - install ounit.2.0.0 [required by mirage]
 - install re.1.2.1 [required by mirage]
 - install ocplib-endian.0.4 [required by mirage]
 - install cstruct.1.0.1 [required by mirage]
 - install io-page.1.1.0 [required by mirage]
 - install mirage-types.1.1.0 [required by mirage]
 - install mirage.1.1.0
12 to install | 0 to reinstall | 0 to upgrade | 0 to downgrade | 0 to remove
Do you want to continue ? [Y/n]

=-=-= Installing ocamlfind.1.4.0 =-=-=
ocamlfind.1.4.0 Downloading http://pkgs.fedoraproject.org/lookaside/extras/ocaml-findlib/findlib-1.4.tar.gz/5d1f8238c53964fdd14387b87b48b5d9/findlib-1.4.tar.gz
Copying ~/.opam/repo/mirage-split/packages/ocamlfind/ocamlfind.1.4.0/files/ocamlfind.install to ~/.opam/system/build/ocamlfind.1.4.0/
Building ocamlfind.1.4.0:
  ./configure -bindir /Users/mort/.opam/system/bin -sitelib /Users/mort/.opam/system/lib -mandir /Users/mort/.opam/system/man -config /Users/mort/.opam/system/lib/findlib.conf -no-topfind
  make all
  make opt
  make install
Installing ocamlfind.1.4.0.

=-=-= Installing cmdliner.0.9.2 =-=-=
cmdliner.0.9.2 Downloading http://erratique.ch/software/cmdliner/releases/cmdliner-0.9.2.tbz
Building cmdliner.0.9.2:
  ocaml setup.ml -configure --prefix /Users/mort/.opam/system
  ocaml setup.ml -build
  ocaml setup.ml -install
Installing cmdliner.0.9.2.

=-=-= Installing ipaddr.2.2.0 =-=-=
ipaddr.2.2.0 Downloading https://github.com/mirage/ocaml-ipaddr/archive/2.2.0.tar.gz
Building ipaddr.2.2.0:
  ocaml setup.ml -configure --prefix /Users/mort/.opam/system
  make build
  make install
Installing ipaddr.2.2.0.

=-=-= Installing lwt.2.4.4 =-=-=
lwt.2.4.4  Downloading https://github.com/ocsigen/lwt/archive/2.4.4.tar.gz
Building lwt.2.4.4:
  ./configure --disable-libev --disable-react --disable-ssl --enable-unix --enable-extra --enable-preemptive --disable-glib
  make build
  make install
Installing lwt.2.4.4.

=-=-= Installing optcomp.1.5 =-=-=
optcomp.1.5 Downloading https://github.com/diml/optcomp/archive/1.5.tar.gz
Building optcomp.1.5:
  ./configure --prefix /Users/mort/.opam/system
  make
  make install
Installing optcomp.1.5.

=-=-= Installing ounit.2.0.0 =-=-=
ounit.2.0.0 Downloading http://opam.ocamlpro.com/archives/ounit.2.0.0+opam.tar.gz
Building ounit.2.0.0:
  make build
  make install
Installing ounit.2.0.0.

=-=-= Installing re.1.2.1 =-=-=
re.1.2.1   Downloading https://github.com/ocaml/ocaml-re/archive/ocaml-re-1.2.1.tar.gz
Copying ~/.opam/repo/mirage-split/packages/re/re.1.2.1/files/re.install to ~/.opam/system/build/re.1.2.1/
Building re.1.2.1:
  ocaml setup.ml -configure --prefix /Users/mort/.opam/system
  ocaml setup.ml -build
  ocaml setup.ml -install
Installing re.1.2.1.

=-=-= Installing ocplib-endian.0.4 =-=-=
ocplib-endian.0.4 Downloading https://github.com/OCamlPro/ocplib-endian/archive/0.4.tar.gz
Building ocplib-endian.0.4:
  ocaml setup.ml -configure --disable-debug --prefix /Users/mort/.opam/system
  ocaml setup.ml -build
  ocaml setup.ml -install
Installing ocplib-endian.0.4.

=-=-= Installing cstruct.1.0.1 =-=-=
cstruct.1.0.1 Downloading https://github.com/mirage/ocaml-cstruct/archive/v1.0.1.tar.gz
Building cstruct.1.0.1:
  make
  make install
Installing cstruct.1.0.1.

=-=-= Installing io-page.1.1.0 =-=-=
io-page.1.1.0 Downloading https://github.com/mirage/io-page/archive/v1.1.0.tar.gz
[ERROR] https://github.com/mirage/io-page/archive/v1.1.0.tar.gz is not available

[ERROR] Due to some errors while processing io-page.1.1.0, the following actions will NOT proceed:
 - install mirage.1.1.0
 - install mirage-types.1.1.0

===== ERROR while installing io-page.1.1.0 =====
Could not get the source for io-page.1.1.0:
OpamGlobals.Exit(66)

The former state can be restored with opam switch import -f "/Users/mort/.opam/system/backup/state-20140103221845.export"
'opam install mirage' failed.
: mort@greyjay:~$; opam install --verbose mirage
^C'opam install --verbose mirage' failed.
: mort@greyjay:~$;
: mort@greyjay:~$; no-proxy
: mort@greyjay:~$; opam install --verbose mirage
The following actions will be performed:
 - install io-page.1.1.0
1 to install | 0 to reinstall | 0 to upgrade | 0 to downgrade | 0 to remove

=-=-= Installing io-page.1.1.0 =-=-=
io-page.1.1.0 Downloading https://github.com/mirage/io-page/archive/v1.1.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122  100   122    0     0    183      0 --:--:-- --:--:-- --:--:--   183
100 48181  100 48181    0     0  30094      0  0:00:01  0:00:01 --:--:--  143k
200
Building io-page.1.1.0:
  make
  make install
ocaml setup.ml -configure --enable-tests

Configuration:
ocamlfind: ........................................... /Users/mort/.opam/system/bin/ocamlfind
ocamlc: .............................................. /usr/local/bin/ocamlc.opt
ocamlopt: ............................................ /usr/local/bin/ocamlopt.opt
ocamlbuild: .......................................... /usr/local/bin/ocamlbuild
Package name: ........................................ io-page
Package version: ..................................... 1.1.0
os_type: ............................................. Unix
system: .............................................. macosx
architecture: ........................................ amd64
ccomp_type: .......................................... cc
ocaml_version: ....................................... 4.01.0
standard_library_default: ............................ /usr/local/lib/ocaml
standard_library: .................................... /usr/local/lib/ocaml
standard_runtime: .................................... /usr/local/bin/ocamlrun
bytecomp_c_compiler: ................................. clang -fno-defer-pop -Wall -D_FILE_OFFSET_BITS=64 -D_REENTRANT
native_c_compiler: ................................... clang -D_FILE_OFFSET_BITS=64 -D_REENTRANT
model: ............................................... default
ext_obj: ............................................. .o
ext_asm: ............................................. .s
ext_lib: ............................................. .a
ext_dll: ............................................. .so
default_executable_name: ............................. a.out
systhread_supported: ................................. true
Install architecture-independent files dir: .......... /usr/local
Install architecture-dependent files in dir: ......... $prefix
User executables: .................................... $exec_prefix/bin
System admin executables: ............................ $exec_prefix/sbin
Program executables: ................................. $exec_prefix/libexec
Read-only single-machine data: ....................... $prefix/etc
Modifiable architecture-independent data: ............ $prefix/com
Modifiable single-machine data: ...................... $prefix/var
Object code libraries: ............................... $exec_prefix/lib
Read-only arch-independent data root: ................ $prefix/share
Read-only architecture-independent data: ............. $datarootdir
Info documentation: .................................. $datarootdir/info
Locale-dependent data: ............................... $datarootdir/locale
Man documentation: ................................... $datarootdir/man
Documentation root: .................................. $datarootdir/doc/$pkg_name
HTML documentation: .................................. $docdir
DVI documentation: ................................... $docdir
PDF documentation: ................................... $docdir
PS documentation: .................................... $docdir
findlib_version: ..................................... 1.4
is_native: ........................................... true
suffix_program: ......................................
Remove a file.: ...................................... rm -f
Remove a directory.: ................................. rm -rf
Turn ocaml debug flag on: ............................ true
Turn ocaml profile flag on: .......................... false
Compiler support generation of .cmxs.: ............... true
OCamlbuild additional flags: .........................
build the unix library: .............................. true
Create documentations: ............................... true
Compile tests executable and library and run them: ... true
pkg_cstruct: ......................................... /Users/mort/.opam/system/lib/cstruct
pkg_bigarray: ........................................ /usr/local/lib/ocaml

ocaml setup.ml -build
/usr/local/bin/ocamlopt.opt unix.cmxa -I /usr/local/lib/ocaml/ocamlbuild /usr/local/lib/ocaml/ocamlbuild/ocamlbuildlib.cmxa myocamlbuild.ml /usr/local/lib/ocaml/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
/Users/mort/.opam/system/bin/ocamlfind ocamldep -package bigarray -package cstruct -modules lib/io_page.mli > lib/io_page.mli.depends
/Users/mort/.opam/system/bin/ocamlfind ocamlc -c -g -package bigarray -package cstruct -I lib -o lib/io_page.cmi lib/io_page.mli
/Users/mort/.opam/system/bin/ocamlfind ocamldep -package bigarray -package cstruct -modules lib/io_page.ml > lib/io_page.ml.depends
/Users/mort/.opam/system/bin/ocamlfind ocamlc -c -g -package bigarray -package cstruct -I lib -o lib/io_page.cmo lib/io_page.ml
/Users/mort/.opam/system/bin/ocamlfind ocamlc -a lib/io_page.cmo -o lib/io_page.cma
/Users/mort/.opam/system/bin/ocamlfind ocamlopt -c -g -package bigarray -package cstruct -I lib -o lib/io_page.cmx lib/io_page.ml
/Users/mort/.opam/system/bin/ocamlfind ocamlopt -a lib/io_page.cmx -o lib/io_page.cmxa
/Users/mort/.opam/system/bin/ocamlfind ocamlopt -shared lib/io_page.cmx -o lib/io_page.cmxs
/Users/mort/.opam/system/bin/ocamlfind ocamlc -c lib/stub_alloc_pages.c
+ /Users/mort/.opam/system/bin/ocamlfind ocamlc -c lib/stub_alloc_pages.c
clang: warning: argument unused during compilation: '-fno-defer-pop'
mv stub_alloc_pages.o lib/stub_alloc_pages.o
/Users/mort/.opam/system/bin/ocamlfind ocamlmklib -o lib/io_page_unix_stubs lib/stub_alloc_pages.o
/Users/mort/.opam/system/bin/ocamlfind ocamldep -package bigarray -package cstruct -modules lib/io_page_unix.ml > lib/io_page_unix.ml.depends
/Users/mort/.opam/system/bin/ocamlfind ocamlc -c -g -package bigarray -package cstruct -I lib -o lib/io_page_unix.cmo lib/io_page_unix.ml
/Users/mort/.opam/system/bin/ocamlfind ocamlc -a -dllib -lio_page_unix_stubs -cclib -lio_page_unix_stubs lib/io_page_unix.cmo -o lib/io_page_unix.cma
/Users/mort/.opam/system/bin/ocamlfind ocamlopt -c -g -package bigarray -package cstruct -I lib -o lib/io_page_unix.cmx lib/io_page_unix.ml
/Users/mort/.opam/system/bin/ocamlfind ocamlopt -a -cclib -lio_page_unix_stubs lib/io_page_unix.cmx -o lib/io_page_unix.cmxa
/Users/mort/.opam/system/bin/ocamlfind ocamlopt -shared -I lib lib/io_page_unix.cmx -o lib/io_page_unix.cmxs
ocaml setup.ml -install
Installed /Users/mort/.opam/system/lib/io-page/io_page.mli
Installed /Users/mort/.opam/system/lib/io-page/io_page.cma
Installed /Users/mort/.opam/system/lib/io-page/io_page.cmxa
Installed /Users/mort/.opam/system/lib/io-page/io_page.a
Installed /Users/mort/.opam/system/lib/io-page/io_page.cmxs
Installed /Users/mort/.opam/system/lib/io-page/io_page.cmi
Installed /Users/mort/.opam/system/lib/io-page/io_page.cmx
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.ml
Installed /Users/mort/.opam/system/lib/io-page/libio_page_unix_stubs.a
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.cma
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.cmxa
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.a
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.cmxs
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.cmi
Installed /Users/mort/.opam/system/lib/io-page/io_page_unix.cmx
Installed /Users/mort/.opam/system/lib/io-page/META
Installed /Users/mort/.opam/system/lib/stublibs/dllio_page_unix_stubs.so
Installed /Users/mort/.opam/system/lib/stublibs/dllio_page_unix_stubs.so.owner
ocamlfind: [WARNING] You have installed DLLs but the directory /Users/mort/.opam/system/lib/stublibs is not mentioned in ld.conf
Installing io-page.1.1.0.
: mort@greyjay:~$; opam --version
1.1.1

[AltGr]

About the latest issue: your log fails to show that it fails to install because it's already installed ; if you got a bug where it's OPAM telling you that you are trying to install two different versions at the same time, I'd be very interested: that may be a severe bug.

About the first issue, there are a few features that could help in that case:

• download everything before starting to uninstall
• provide another .import file that you could use to retry instead of revert ?
• snapshot the directory before starting to process, to get a real revert (a bit heavy-weight !)

Any opinions / other ideas ?

[mor1]

sorry-- i must've cut'n'pasted the wrong console output (or just misread it).

downloading everything first is probably the first & easiest thing to do here.

not sure what the .import file you propose is or what it's for?

directory snapshot via hardlinks perhaps? though filesystems that don't support them might have trouble (but do you have stats on which systems opam is used on -- if there's a very small number that can't support hardlinks, might be acceptable to add a feature that's not supported on every platform).

[AltGr]

| downloading everything first is probably the first & easiest thing to do here. Yes, I'm implementing that, it's cleanest and possibly faster (you get somehow finer grained parallelism)

| not sure what the .import file you propose is or what it's for? Not so closely related, but on failure OPAM now asks you:

The former package state can be restored with `opam switch import -f ...`

The idea was to add:

Or you can try to replay the same request with `opam switch import -f ...`

Not sure if it's that useful though

[AltGr]

Fixed by 1c03455989e81c4b60aa9095b230e296258bd703 :)