search

ocaml / opam

opam is a source-based package manager. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.
https://opam.ocaml.org
Other
1.25k stars 359 forks source link

The internal solver is broken on some corner cases #1371

Closed samoht closed 10 years ago

samoht commented 10 years ago
rdicosmo commented 10 years ago

Now that we have a solver farm available, would'nt it be simpler to just integrate a call to the farm by default inside opam itself?

There are many clear advantages:

Without an internet connection, well... opam is not that useful, and I can think on only two cases where one might want to run without internet

Roberto

On Fri, May 02, 2014 at 09:11:14AM -0700, Thomas Gazagnaire wrote:

• if you have an installed package which is not in the universe anymore, it will not propose to uninstall it

• @avsm reported that some base packages where not installed properly

— Reply to this email directly or view it on GitHub.*

dbuenzli commented 10 years ago

Le vendredi, 2 mai 2014 à 18:33, Roberto Di Cosmo a écrit :

Without an internet connection, well... opam is not that useful, and I can think on only two cases where one might want to run without internet

git allows you to work offline, pins can be local git repositories. I'd like to be able to work with local pins (update/reinstall) when I'm disconnected.

Daniel

samoht commented 10 years ago

And we have no proof currently than the solver farm is reliable under load, we need to worry about servers availability, ie. all the boring but very important features offered by cloud services.

The server farm is a great idea for some use-cases, but I don't think it's a good idea to make it a default.

Regarding the internal solver: I have no intention to maintain it forever, but that's currently the simplest/portable thing we have at hand. Any other pure-ocaml solution is more than welcome.

rdicosmo commented 10 years ago

On Fri, May 02, 2014 at 09:46:16AM -0700, Daniel Bünzli wrote:

git allows you to work offline, pins can be local git repositories. I'd like to be able to work with local pins (update/reinstall) when I'm disconnected.

Sure, for power users, taking the time of installing a properly working version of aspcud is surely justified.

But I wonder what % of the total user base we are targeting this represents.

And, as I clearly stated in my original mail

Roberto

rdicosmo commented 10 years ago

Several interesting points here too... let me try to offer my opinion

On Fri, May 02, 2014 at 09:51:13AM -0700, Thomas Gazagnaire wrote:

And we have no proof currently than the solver farm is reliable under load, we need to worry about servers availability, ie. all the boring but very important features offered by cloud services.

True, but these are quite commonplace and boring issues one can solve easily today. We also need to worry about github availability, if we start going that way.

The server farm is a great idea for some use-cases, but I don't think it's a good idea to make it a default.

You are right, I did not make my thought clear enough: the idea would be to do the following

IF a local external server is available, use it ELIF the solver farm (or one of several solver farms) is available, use it ELSE use the internal solver and accept its limitations

Regarding the internal solver: I have no intention to maintain it forever, but that's currently the simplest/portable thing we have at hand. Any other pure-ocaml solution is more than welcome.

Here, I do apologize for resorting to pedantic mode, but as much as an OCaml fun I am, let me state yet again that properly dependency solving is a difficult separate concern.

We need several solvers, properly maintained and evolved by other research teams (like aspcud or packup), and rewriting them in OCaml is simply not an option.

samoht commented 10 years ago

We also need to worry about github availability, if we start going that way.

well, that's their business, they have people working full-time on that and they proved to be good at it, so yes, I'm ready to rely on them.

We need several solvers, properly maintained and evolved by other research teams (like aspcud or packup), and rewriting them in OCaml is simply not an option.

That's true. I just meant we need a portable solution (with people ready to invest in the maintenance cost).

rdicosmo commented 10 years ago

Out of curiosity, is this just an hypothetical argument, or do you really do this in your day to day life?

The reason why I am asking is that maintaining a copy of the opam repository is indeed easy (using git), but also useless unless you have all the source files.

Indeed, these sources need to be dowloaded over the network following the links in the url file contained inside each opam package.

And even if you know about the little known (and not recommended) sync-archives option, it is not a good idea to sync all the archives every time you run opam update: it eats up your disk space, it takes a lot of time and it is not as up-to-date as you might think.

To criticise my original post, it seems to me that the only use case where you might meaningfully use opam without the network is in an "opam-in-a-box" scenario, which is a pretty limited scenario indeed for a developer.

If you believe I am missing something here, I'd be glad to know more.

On Fri, May 02, 2014 at 09:46:16AM -0700, Daniel Bünzli wrote:

Le vendredi, 2 mai 2014 à 18:33, Roberto Di Cosmo a écrit :

Without an internet connection, well... opam is not that useful, and I can think on only two cases where one might want to run without internet

git allows you to work offline, pins can be local git repositories. I'd like to be able to work with local pins (update/reinstall) when I'm disconnected.

Daniel

— Reply to this email directly or view it on GitHub.*

dbuenzli commented 10 years ago

Le vendredi, 2 mai 2014 à 23:16, Roberto Di Cosmo a écrit :

Out of curiosity, is this just an hypothetical argument, or do you really do this in your day to day life?

When I travel I'm not always connected and/or the connection is get can be spotty. Besides I only live with the 3g connection of my phone that I plug through usb and I'm not always connected to it.

The reason why I am asking is that maintaining a copy of the opam repository is indeed easy (using git), but also useless unless you have all the source files.

Obviously I don't want opam to be able to install things I don't have installed yet without the network, that would be absurd. However I would like opam to be able to work with what I have locally. Since it keeps an archive of what has been installed, reinstalling these things should be possible. I want to be able to work on my pinned packages (let's say on react which may retrigger the installation of lwt) while being disconnected.

(Incidentally this happened to me just the other day, had already disconnected my phone to go to sleep, just had an idea, implemented it wanted to reinstall the pin to but opam failed).

Daniel

rdicosmo commented 10 years ago

On Fri, May 02, 2014 at 10:39:59AM -0700, Thomas Gazagnaire wrote:

We need several solvers, properly maintained and evolved by other research
teams
(like aspcud or packup), and rewriting them in OCaml is simply not an
option.

That's true. I just meant we need a portable solution (with people ready to invest in the maintenance cost).

I agree, but let's be pragmatic and look at the current state of affairs:

Hence this simple proposal for a new workflow for dependency solving in Opam

IF a local external solver is available, use it (1) ELSE IF one (out of several) solver farm is available, use it (2) ELSE use the internal solver and accept its limitations (3)

This is an approach that will make most users happy, allowing everybody to leverage the new preference combinators available in recent versions of aspcud, while preserving exactly the previous behaviour (1)+(3) in case the network fails or the solver farm is unavailable.

I cant see any drawback, but I'd glad to hear if I'm missing something

Roberto

rdicosmo commented 10 years ago

Ok, I see... so yours is really a feature whish, not something you already do.

Currently, opam does not keep a copy of the source files of the packages that are/have been installed. That's why I was saying that you need the network to use opam, and using the solver farm is a no brainer.

Anyway, I think you can try the following solution for your very special needs:

Hope this helps

Roberto

On Fri, May 02, 2014 at 02:38:47PM -0700, Daniel Bünzli wrote:

Le vendredi, 2 mai 2014 à 23:16, Roberto Di Cosmo a écrit :

Out of curiosity, is this just an hypothetical argument, or do you really do this in your day to day life?

When I travel I'm not always connected and/or the connection is get can be spotty. Besides I only live with the 3g connection of my phone that I plug through usb and I'm not always connected to it.

The reason why I am asking is that maintaining a copy of the opam repository is indeed easy (using git), but also useless unless you have all the source files.

Obviously I don't want opam to be able to install things I don't have installed yet without the network, that would be absurd. However I would like opam to be able to work with what I have locally. Since it keeps an archive of what has been installed, reinstalling these things should be possible. I want to be able to work on my pinned packages (let's say on react which may retrigger the installation of lwt) while being disconnected.

(Incidentally this happened to me just the other day, had already disconnected my phone to go to sleep, just had an idea, implemented it wanted to reinstall the pin to but opam failed).

Daniel

— Reply to this email directly or view it on GitHub.*

dbuenzli commented 10 years ago

Le samedi, 3 mai 2014 à 01:17, Roberto Di Cosmo a écrit :

Ok, I see... so yours is really a feature whish, not something you already do.

That may be a feature wish but otherwise it's not about something I do or not, it's about expectations. The day it comes I want to do something and that by chance I have no network and this thing can be done without network I expect the program to do it without the program asking for the network.

Currently, opam does not keep a copy of the source files of the packages that are/have been installed. That's why I was saying that you need the network to use opam, and using the solver farm is a no brainer.

I thought that this was what is in .opam/archive but I now see that this seems incomplete. So what's in .opam/archive exactly ?

Daniel

avsm commented 10 years ago

OPAM was designed to work without requiring Internet access, which doesn't necessarily mean network access. For instance, it can be used within a company's build environment with a custom remote, fetching sources from internal git trees.

I wouldn't be opposed to better support for a hosted remote solver, but at a minimum the scripts and other setup need to be made available so that a copy could be hosted alongside the rest of the OPAM infrastructure on opam.ocaml.org.

I'm also not sure why this cannot always just be a shell script. Wouldn't it be better to give OPAM a solver search path (in ~/.opam/solver.d or somewhere else similar) and let anyone drop in solver scripts there?

-anil

AltGr commented 10 years ago

Let's not enter a philosophical debate here, and focus on providing the best features in the most convenient possible way.

None of these solutions is perfect, but the solver farm does in my opinion fill a role in here. For these to play nicely, it would be best to be able to choose transparently, possibly by checking in advance the availability of the farm, so that there is no additional polling delay. This is a little bit more difficult to do with a script, and that also makes the basic opam installation require more than a single file -- but on the other hand I'm not totally happy with including that part into OPAM either.

Just a random additional remark you even need the solver to remove packages

samoht commented 10 years ago

Currently, opam does not keep a copy of the source files of the packages that are/have been installed. That's why I was saying that you need the network to use opam, and using the solver farm is a no brainer.

Hum ? just noticed that now ... this claim is not true. Opam keeps a cache of archives which have been dowloaded, so if you install a package twice, it will only download the archive once.

rdicosmo commented 10 years ago

On Mon, May 05, 2014 at 10:24:35AM -0700, Thomas Gazagnaire wrote:

Currently, opam does not keep a copy of the source files of the packages
that are/have been installed.

Hum ? just noticed that now ... this claim is not true. Opam keeps a cache of archives which have been dowloaded, so if you install a package twice, it will only download the archive once.

Sorry @samoht, you are right on this, and I stand corrected!

AltGr commented 10 years ago

if you have an installed package which is not in the universe anymore, it will not propose to uninstall it

Can you confirm that this is the case even when doing a simple opam upgrade ? It's not supposed to be removed otherwise, unless needed, and this should be handled by the preprocess_request function.

samoht commented 10 years ago

I certainly was able to be in a situation where OPAM was trying to install a new version of a package without uninstalling the other one, when using --no-aspcud. Using the external solver gave the good result, so I assume something is wrong in the internal solver somehow.

AltGr commented 10 years ago

@samoht I believe this has been fixed when the 'pinned' version was removed, please reopen if it has happened again !