Deprecate package: put an upper bound of 0.6- on the Julia version

[timholy]

Note to any watchers: this package is now officially deprecated. Will be supported through Julia 0.5, but that's the end. Start making plans to migrate to Interpolations.

If you use the restrict/prolong functionality, let's talk about moving that elsewhere.

[tkelman]

This would need to be done on all existing tags in metadata too.

[coveralls]

Coverage remained the same at 80.071% when pulling 8b6a2e26b699252e109fdb6ff8164877c2dabb93 on teh/deprecate into 292e43b7337a235710a20830133a26fa85fcd291 on master.

[timholy]

Yep, preparing that now. 0.0.0 through 0.2.x don't specify a julia version. I'm planning on simply deleting those versions from METADATA, and putting upper bounds on the rest. OK?

[tkelman]

No deleting of past versions. Add a julia 0.0 0.6- line to their requires (create it if it doesn't exist yet)

[timholy]

For the record, 0.3.0 was tagged in April of 2014, when julia 0.3 was still being developed.

[timholy]

OK.

[timholy]

Our poor version resolver is going to crash and burn eventually if we don't delete old versions.

[tkelman]

The most archival-friendly way to do that would be a clean-start new branch of metadata. Which is going to be a hassle, but probably necessary for Pkg3 restructuring anyway.

[timholy]

Once we get Pkg solid enough that there's unlikely to be a need for future reboots, we might need to have a policy for trimming the number of versions, unless we're planning on migrating away from treating it like an optimization problem.

[tkelman]

I think the filesystem access and our current layout of multiple files per version causes more of a problem than the cost of the resolver algorithm itself. @carlobaldassi has done more detailed profiling. If we don't move to an actual database in Pkg3, we should at least move to a single version-listing file per package, since I suspect parsing something like TOML or JSON will actually be less expensive than reading hundreds of files.

[timholy]

(As someone who grudgingly uses kmail and resents akonadi's troubles with mysql and has noticed that they're working on migrating to a custom solution, I would be nervous about using an "actual" database. But maybe that's just me.)

[tkelman]

haven't played with it yet, but see https://github.com/ararslan/JuliaPkgDb if you're curious

[tkelman]

And right as you make that prediction, it looks like https://github.com/JuliaLang/PkgDev.jl/issues/56 is actually a resolver-accuracy issue! Setting JULIA_PKGRESOLVE_ACCURACY to 2 makes it go away! Now who's the annoyingly good one? Let's see if deprecating old versions of some packages with upper julia version bounds helps the resolver out...

[carlobaldassi]

I fixed the solver issue of https://github.com/JuliaLang/PkgDev.jl/issues/56 in https://github.com/JuliaLang/julia/pull/17709.

However, regarding the policy about old package versions: adding upper bounds to the julia requirements in old versions works perfectly fine for the time being, since those versions are filtered out in a preliminary step before the resolve machinery is invoked. However, this strategy is not going to work unless julia versions keep coming out at the current pace, i.e. after julia 1.0 one would not expect that old package versions should be deprecated in this way. At that point, I think that coming up with some strategy for retiring old versions will be required anyway, whatever solver is used (i.e. even without solving an optimization problem there is still a satisfiability problem to solve, which is hard, meaning that for sufficiently complex graphs it will either fail or take a long time — for example, with the current solver, we could decide to restart it with a higher accuracy if it fails, with a computational cost penalty).

One relatively easy thing to do could be to try running the solver without the oldest versions, and only bring them back in case of failure. We could also try to be smarter in PkgDev.publish() in detecting what tests really need to be performed.

It is also telling that the solver was starting having trouble when trying to pin some very old versions of a package. The dependency graph for those versions was very large and quite nasty, mostly because for many packages there were no upper bounds set. Somehow, I doubt that the resulting installation would even actually work. I don't think it would be impossible to set up some periodic task to try and detect automatically the upper bounds for some packages if they don't declare them, based on their tests.

I also don't rule out the possibility that some even more aggressive pruning of the dependency graph could be performed.

[carlobaldassi]

Oh and about performance: yes, during a normal Pkg.update() the resolver time is mostly negligible compared to the rest (just think of fetching the new METADATA, or precompilation), at least it was whenever I measured it. The resolve time becomes significant for PkgDev.publish and METADATA tests because resolve is called repeatedly, almost for each and every version in METADATA (it simulates a blank state and the addition of a single package at a specified version, and checks if the requirements are satisfiable).

[timholy]

Personally, I think it's quite reasonable to have a policy of deleting all but the last 0.1.x release when you come out with 0.3.0 (or perhaps 0.3.n where n is high enough to be confident things are working). That would eliminate the large majority of versions from consideration.

[tkelman]

For reproducibility I don't think any old versions should be deleted. Upper julia version bounds that rule out their installation on versions of Julia that they will never work on are fine though.