Pkg3: conservative compatibility will make it harder to access new features

[tkelman]

[sorry for wall of text - leaving town this weekend, want to write initial reactions down while fresh]

A package should not declare compatibility with a minor version series unless some version in that series has actually been published – this guarantees that compatibility can (and should) be tested. If a new compatible major or minor version of a package is released, this should be reflected by publishing a new patch that expands the compatibility claims.

So this is requiring upper bounds, and in a strict way where the bound must already exist (though unlike current upper bounds, these would be inclusive). While I like that idea in theory - only allowing package dependency versions to be installed that are known to work - it trades the "the new release of package A broke package B" problem for a smaller feasible set of allowed versions. Users of widely depended-on packages will be held back to old versions if they also want to use any dependent packages that update slowly. If both package B and package C depend on package A, package B hasn't tagged since A released version 1.4, and package C depends on a new feature in package A that was first released in version 1.8, you won't be able to use both package B and package C in the same environment until package B has tested and tagged a 1.8-compatible version.

Automatic testing of reverse dependencies and making an automatic set of new downstream tags with wider bounds if tests pass could help here, but we don't have that infrastructure yet (and requiring such infrastructure in order for a set of packages to progress cohesively may be a burden to place on organizations that want to run their own registry). For packages that can't be tested on CI, or start failing in the automatic test results, then you start needing to involve the authors of even sporadically-developed packages any time their dependencies put out new feature releases that people want to use.

We'd also need much better error messages and suggested fixes when dependency resolution fails to find a feasible set of versions. Resolution failure is luckily pretty rare right now, but can be very confusing when it happens. Downgrading or being held back to old versions does happen now with upper bounds, and being stricter about them would make that more common. If a set of versions that are known to work can be installed we should do that, but I fear the choice will often be between allowing untested versions to be installed or erroring when the user tries to do so. When those are the only choices, I think the former has a higher chance of allowing the user to get things done (or figure out how to fix the problems).

https://www.well-typed.com/blog/2014/09/how-we-might-abolish-cabal-hell-part-1/ is worth at least skimming, as Haskell's ecosystem has gone through many similar issues. By going from the current scheme of only applying upper bounds when problems are already known or the package author is choosing to be conservative, to a scheme where strict tested bounds are the only thing that's allowed to be released, we're moving from the Julia equivalent of "Type errors" or "Compile errors" to the equivalent of "solver failure."

Lastly, this makes package authors' adherence to semver (or lack thereof) much more consequential. We do want to encourage these processes and get people thinking more about them, but I think social expectations and more thorough documentation are safer ways to get there right now than baking them into the behavior of the package manager.

[ChrisRackauckas]

While I was thinking this on my first read, I actually think this won't be an issue, at least within orgs. What I think you'll see is a lot of orgs which tag a bunch of things together, where package A has a new minor release, with B,C,D along for the ride with a patch which just update the version requirements to say "we can use the new A". So within orgs, I think the updates can happen seamlessly.

Outside of orgs, I think it's fine because if you didn't know a package got an update, the conservative option is probably better (the current way this is handled, by me at least is, wake up in the morning an see an issue about how X no longer works and go, huh, _____ must've updated). With the different environments, maximum version requirements will no longer constrain the user's entire system from getting the new release (if I understand it correctly), so to me it seems acceptable.

[tkelman]

Project-specific environments mean you don't have to worry about packages getting held back by things you aren't currently using. But unless we somehow enable simultaneous loading of multiple versions of the same package without them stomping on each other, you would need packages to be able to coexist in the same environment if you want to use them at the same time.

[ChrisRackauckas]

Thanks for the clarification. Then yes, I see big problems here. A good example is Compat. Lots of libraries depend on it, and so you wouldn't actually get the new version until every library tags to add compatibility. Since I usually don't know when that happens, you would probably be waiting on people like me for two weeks to find out I need to update that maximum version requirement and tag. Even worse, a developer who just disappears for a month would have a similar effect. So you'd have to work on the assumption that, widely used packages don't actually update for a week or so.

One way to help this is to make it automatically open an issue on the dependent package's repository to inform them of the update. That sounds like it can get out of control really fast though.

[tkelman]

Yeah upper bounds on Compat would cause things to go slightly haywire. Every Compat tag would have to be followed by a few hundred upper bound bumps.

[nalimilan]

I'm also concerned this is going to be a nightmare except for very actively maintained packages from code organizations. The generally retained solution used by the SONAME mechanism is to have major versions be parallel-installable, and require minor versions to be backward-compatible.

Maybe we should use a similar approach by allowing loading two major versions of the same package at the same time? This would be more or less equivalent to appending the major version to the module name (but transparently). Of course the user wouldn't be able to use them both at the REPL, but each package would still use its own version internally. This would work well e.g. for Compat.

Then we should just stress to package authors that minor versions should never break existing code, and that new major versions should be tagged instead. Systematic testing of dependencies would help checking that.

[StefanKarpinski]

There are two major issues:

1. The compatibility in the source of a project should be considered definitive – having to look in multiple places based on complex criteria is a bad situation.
2. Compatibility constraints shouldn't be wrong by design – and currently they are. This is compensated for by updating them afte the fact currently, but that's completely at odds with point 1.

If registries have the ability to tag new versions of packages, then issuing a lot of patches of packages wouldn't be so bad, although it would want automation very much.

I would point out that the API idea that I've floated would solve this problem since it pushes the reasoning about compatibility with a library onto the library itself, rather than its dependents. I can open an issue about the API idea when I have some time.

[tkelman]

I don't think it's that bad to have the registry responsible for compatibility information, since that's the living changing source of versions. Compatibility constraints are often correct when created and only become incorrect later due to the registry changing with new versions of other things. The criteria for which information to use isn't that complicated now, it's if you're on a tagged version of something registered then you use METADATA and otherwise you use the package REQUIRE. The "otherwise" will happen a lot less if Pkg3 isn't responsible for managing anything that doesn't come from a registry.

[StefanKarpinski]

So you're basically saying "it's fine, I like it the way it is"?

[StefanKarpinski]

Having registries in charge of compatibility makes working with unregistered packages impractical / hard. I'm not sure if you've tried explaining where the definitive compatibility information is to someone, but it really fails the "sane explanation" criterion. Having package source be definitive also isn't what causes a problem – it's the fact that this proposal only allows minor version compatibility declarations. Could Compat version bumps be patches? I'm a lot less concerned about other packages. It's also possible that the way Compat works needs to be reconsidered. We could do something more like Python with "from future import".

[nalimilan]

It's not just about Compat: several packages have many reverse dependencies, like DataFrames.

[tkelman]

Enforcing upper bounds is the issue here. They can be recommended, but upper bounds lead to disallowing combinations of packages from being installed together unless one or both authors have recorded that those versions work together. Is disallowing installation when information is not yet known preferable to allowing installation and documenting how to change which versions are being used?

Making registries or developers of widely used packages have to then tag many of their reverse dependencies could get pretty confusing for the authors of those packages that got tagged without their involvement.

[tkelman]

Making the definitive source of compatibility information come from something immutable, the original released package source, is the other issue here. If multiple registries is a thing that works fine, then you have a better more uniform place to put compatibility information for packages that we would treat as unregistered right now.

Compat implements new features often, those probably shouldn't be released as patches.

[tkelman]

There are 3 possible states for any particular version of a dependency:

1. known to be compatible (aka has been tested)
2. known not to be compatible
3. not yet known

We need to record enough information in the system to make known incompatible dependency versions not installable. Your proposal says state 3 should not be installable either (so new versions in state 3 will constantly have to be tested to be recategorized), I'm suggesting it should. Otherwise you have to work outside of Pkg3 to do the testing of new versions, or create a tag before being able to test it.

We probably want a way of recording at tagging time (edit: or after, if the compatibility records are mutable) which dependency versions have been tested, so there's enough information to tell apart known-compatible versions from unknown. But I'm not sure how we should use that information - preventing installation is IMO too strict. We could warn or make the distinction more discoverable in other ways maybe?

The constant arrival of new versions, and the fact that action is required to categorize each of them, suggests to me that compatibility records should be mutable.

[StefanKarpinski]

That's a good breakdown – my proposal does lump 2 & 3 together (under the premise that a new patch will be used to expand the sphere of 1 to include the appropriate regions of 3). Different end-users may have different tolerance here: some users will only want known-good combinations and keep potentially surprises to a minimum; others will prefer to have fresher versions at the risk of things breaking sometimes. There's also a distinction to be made between minor version updates and major ones: minor updates should continue to work most of the time; major ones will often cause things to break.

[StefanKarpinski]

Keeping a database of version sets and systems where tests have passed or failed would be useful. This could be submitted by people's actual systems if they opt into it, as well as test bots. I'd be very reluctant to have that entirely replace explicit compatibility claims, however.

[simonbyrne]

In addition to requiring authors to be strict about semver of their own package, it does also place significant onus on them to be accurate about their dependency requirements.

Suppose that packages B and C both depend on v1.2 of package A, and a new v1.3 of package A has recently been tagged.

• Package B is updated, and also uses some new features of package A, but the author forgets to bump the version requirement.
  • Any user of Packages B and C would be stuck with A v1.2, which would break B
• Package B is updated, not breaking compatibility with v1.2, but the author can't be bothered checking all checking all versions of A to see which are compatible, so just sets v1.3 as the requirement.
  • Any user of Packages B and C would not be able to upgrade B.

Such cases don't seem too uncommon, so we would need sufficient tooling to check for these sorts of errors. Perhaps it is worth setting out exactly what this would entail.

[StefanKarpinski]

One possible approach is to automatically pull up-to-date compatibility info into packages when they're installed and when you build/test them. That implies including the additional allowed versions in files that are in .gitignore by default – e.g. in Local.toml. That could be merged with what's in Config.toml easily enough by concatenating compatibility arrays.

I still think that it makes sense for the "atoms" of compatibility to be minor versions with exclusions for specific broken patches. I can't think of a legitimate situation where that's not sufficient to express a compatibility relationship.

[simonbyrne]

I guess one lesson from our current situation is that it is useful to be able to make post hoc modifications to dependency version constraints: e.g. in my first example above, tagging a new version of package B with updated constraints would not solve the problem, as the resolution mechanism would still stick with the older broken version set. In other words, bugs in dependency versions are the only bugs that can't be fixed by tagging new versions, so we need an alternative mechanism to do that.

[StefanKarpinski]

This is exactly why I originally made dependencies mutable in METADATA and take precedence over what's in the package source, but that's confusing and doesn't interact well with package development. If the updated dependencies are automatically injected into the package source in a way that's development friendly (which is what the above comment aims to achieve), then we could have both.

[tkelman]

I still think that it makes sense for the "atoms" of compatibility to be minor versions with exclusions for specific broken patches. I can't think of a legitimate situation where that's not sufficient to express a compatibility relationship.

You're being charitable to people's ability/discipline in following semver. We can't really enforce that, and while it may be "illegitimate" in a way, that doesn't mean someone won't tag a version 0.5.10 that both breaks other packages that worked on 0.5.0-0.5.9, and introduces features that other packages will need to depend on. Downstream packages are going to end up having to list out every single patch version that won't work individually. Our version ranges have an unintuitive syntax right now, but I don't think allowing patch endpoints is a problem.

[StefanKarpinski]

Whether people follow semver or not is partly due to the way the package manager works. If someone introduces 0.5.10 like that, then, when there are complaints, the right response is to give feedback on why that's bad, advise them to retag 0.5.10 as 0.6.0 and then other packages can include [0.5, 0.6] if they work with the new feature, or indicate [0.5, !0.5.10] if not. The fact that this process is a bit awkward provides important feedback to package authors about how packages should be versioned.

[tkelman]

Quite a few people, including some package authors, have no idea what semver is, and it's not like learning about it is a required prerequisite for using Julia or publishing a package - keep in mind the percentage of Julia users who are not software engineers. People aren't intentionally ignoring semver because REQUIRE allows them to use point releases, people forget about it because they don't know any better (the packages documentation page doesn't even say anything about semver being recommended right now) or they just use the default PkgDev.tag behavior out of habit. We maybe should think about deprecating single-argument tagging, with a brief explanation about when to use patch vs minor vs major, to mitigate this.

Forcing inconvenience by removing a currently-possible level of specificity from dependency version requirements doesn't seem desirable to me. We should aim to make the requirements syntax more intuitive, explicit, and easier to understand, but removing a capability from version resolution isn't required for that. It seems more coupled to your proposed "version resolution will favor the most recent patch very strongly" change than anything else. To draw an analogy to other packaging systems, you're effectively proposing demoting what we currently use as patch versions from a full-fledged separate version to something more like a build revision number in deb/rpm/conda terminology.

[StefanKarpinski]

Expecting package authors to roughly follow semver is in no way unreasonable a huge burden. The basic idea can be explain in a few sentences. Are we, as a community, not going to expect packages to have tests and pass them? That's a much more "software engineering" high bar than very roughly following semantic versioning.

[ChrisRackauckas]

I agree with Stefan here. I mean, it was only a few months ago that @tkelman pointed me to CI testing and semvar before registering packages, and it was fairly easy to pick up. I think an expectation of semvar, docs, and CI is fine.

However, I think if semvar is to be expected, it should be baked into the way things work. Basically, if this is how we are actually doing things, I would like a notification of some sorts of a package I have a dependency on has a new minor version, expected that there might be a breaking change that just occured. This is so I can it to the list of things I need to update. Otherwise I won't know to do so, and it will sit until I try to tag and @tkelman points it out to me that my version requirements are low and blocking other people.

Honestly, I only check the versions of packages outside my orgs when I am tagging and @tkelman asks for adding minimum version requirements. Since many packages are interconnected, that tells you how long it will take before a new minor version of something like Compat or Dataframes will actually be available: months after the tag. With how agile and fast moving the package ecosystem is, that seems like a major problem.

[tkelman]

Expectation is fine, encoding the behavior into the package manager is another matter. What's the technical reason to engineer the system to work intentionally poorly when semver isn't followed? Things will break for people until action gets taken from social pressure, but it's not the package authors responsible for the mistake who will feel the pain directly - only downstream packages will notice. Some package authors have a hard enough time making tiny changes to version dependencies on tags now. The consequences of not following semver won't be noticed until dependencies start trying to use new versions, there's a delay and fixing the versioning after the fact can end up breaking other packages if new dependencies have been declared in the interim.

The lack of notifications and visibility if upper bounds are default behavior is why I opened this. The REQUIRE syntax needs revising and there needs to be more visible feedback when bounds are making a difference, but I don't think the way in which dependency version resolution functions between packages is all that broken. The bullet point about global resolution is an issue, but that doesn't require changing the granularity of dependency specification to fix. Is there precedent for an existing package manager that does not use ranges as the most common form of dependency specification, or treats different parts of the version numbers differently in expressing requirements?

The closest thing I can think of is build revisions in deb, rpm, and conda, but these are usually "this was a mistake" or "changed build configuration" or "rebuilt same sources against updated dependency versions." We may want to allow replacement-republishing this kind of thing for requirements/tests/doc-only changes, but if anything under src changes then it shouldn't be treated as the same version number, either in publishing or in dependency resolution.

There's also the not-totally-implausible scenario of a bug that gets fixed in, say, Images version 0.4.50. If another package needs to depend on that bugfix, is the only way to express that by writing out every single one of 0.4.0 through 0.4.49 as negated patches?

[StefanKarpinski]

If a package can't use versions 0.4.0 through 0.4.49 I think it's a stretch to claim that it "works" with 0.4 in general. In that frankly rather unlikely scenario, I would wait until 0.5 and then assert compatibility with that.

In general, I'm not sure what harm there is in starting out with the most limited form of dependency expression that satisfies the kind of things that we know packages need to do. If that turns out to be too limited in practice, we can always safely add expressiveness to it, but you can't take expressiveness away without breaking things. Constraining the kinds of things people can express to the kinds of thing they should express has, in my experience, been one of the only effective ways to keep them from expressing things that they shouldn't.

[tkelman]

If the same person were responsible for the choice of version tag and needing to depend on that version tag downstream in every case, I'd agree with you. But these are being done by different people on different time scales. You're constraining what downstream can express, and hoping the upstream behavior to make it work correctly will be followed, without any immediate method of enforcing that at tag time. Past versioning mistakes will inconvenience downstream users indefinitely in this scheme, to the point where upstream may no longer be paying attention, or may have moved on to later versions such that the mistake can't be fixed by a new tag any more. In a post-1.0 ecosystem, there will be users who are concerned with specifying older known working versions, and there will be other users who want the bleeding edge latest releases.

You're proposing taking away expressiveness vs what we have right now, and I'm not convinced that the current expressiveness is harmful in any way. It's more flexible at retroactively fixing past mistakes than the proposal here, and that would be a painful capability to lose.

[tbreloff]

I want to make the point that perhaps we should separate the responsibilities and expectations for "package authors/maintainers" and "release/registry maintainers". I know that, for me personally, I'd be very happy if there was a group of people that managed version dependencies, tagging releases, etc so that I could focus on the actual development of features.

This isn't possible today because Pkg2 is tightly coupled to the git repo (and the git tags). If a "tag" was just an entry in some list/registry, then one wouldn't need commit access to a package repo to be able to manage its releases.

It might be that organizations take on a shared role for maintaining cross-package compatibility requirements and for tagging point releases following semvar in a consistent manner. I bet overwhelmingly individual package authors would jump at the chance to add their package to a "managed pool" of packages. The upside is that packages can be tested and tagged as large groups, and it will be incredibly simple to resolve issues as compared to expecting every package author to play well together at all times.

The other advantage to shared responsibilities is that we aren't waiting on one or two people to fix a critical version incompatibility. (this will happen a lot if upper bounds are forced... something that I think is a good idea, but requires the right social structure to work well)

[andyferris]

Maybe I missed or misunderstood something fundamental, but I'm seriously worried by something, and I was hoping for an explanation. SemVer states that minor releases are to be backward compatible. Why would you simultaneously require package maintainers to learn and follow semver while not maximizing the upgrade possibilities at the same time?

If someone makes a backward-incompatible change to a minor patch, people will quickly ask them to (a) patch the minor release to be backward compatible, and (b) make a new major version wherever that is necessary (i.e. the improvement required actual breakage).

Of course, in the end if we are following SemVer as a community, we will see a huge increase in major releases, and for each and every one of these we could end in dependency gridlock. A tool the lets you see the registered dependencies for your package and perhaps even email the developers that a new major release of your package is available would help somewhat.

However, there's no sense in making gridlock more likely by gridlocking on minor releases, and doing so (in what seems to me) unnecessarily.

[StefanKarpinski]

@andyferris: I'm not entirely clear what you're arguing for here? Compatibility declarations at major version granularity? I.e. assume, by default, that new minor versions don't break compatibility?

[andyferris]

Yes, precisely!

Is there an argument against this?

[StefanKarpinski]

Even if everyone follows SemVer perfectly, lower bounds on compatibility relations need to be a the minor version level since new features are introduced at that level. It's also not uncommon for new features to break existing code, at the very least because of the way using works. If everyone explicitly imports everything they're using, that shouldn't be a problem, but then we need a way of enforcing that. I also find it more plausible that people can be strict about point releases not breaking things than they can about minor releases, which introduce new functionality, not breaking things.

[ChrisRackauckas]

At least currently with how the ecosystem is growing, more people tend to use minor releases as "breaking change releases" than major releases (which are under-utilized right now, likely because of the state of development for most packages). Enforcing at the minor level makes more sense for how this is currently done. This means that many people are adding features in patches (myself included) since we're avoiding major releases: should this be a practice which we shouldn't do anymore?

[StefanKarpinski]

0.x => 0.y is a major release; 1.x => 1.y is minor. This isn't official in SemVer, but that's how we've been doing it. w.x.y => w.x.z is a patch (bugfix only) regardless of whether w == 0 or not.

[tbreloff]

Yes with 0.x.x there really isn't such a thing as a minor version (or no such thing as a patch??) 0.1 --> 0.2 means any sort of breakage is allowed. 0.1.1 --> 0.1.2 means... well hopefully things still work :)

[andyferris]

Yes, ignoring 0.x.y, a minor release will mostly be non-breaking, and package maintainers can add exceptions at a minor granularity. @StefanKarpinski you foresaw the possibility of requiring a minimum patch and excluding some patches otherwise - I think doing the same at the minor (but not major) level makes sense.

I also find it more plausible that people can be strict about point releases not breaking things than they can about minor releases, which introduce new functionality, not breaking things.

Sure, but breakage will occur (much) less often than compatibility. Thus it would be a lesser burden for package maintainers, while simultaneously reducing the possibility of "deadlock" between packages which aren't frequently maintained. Furthermore, a maintainer will feel more pressure to fix a problem when upstream changes occur, just like now, so thus we are all much more likely to continue moving forward together. Seems win-win to me.

[tkelman]

Depending on a feature in a lower bound should be at the minor level most of the time, but sometimes you need to depend on a bugfix as a lower bound. Or people don't follow semver and add features in patch versions.

[Spencerx]

Maintaining different snapshots in multiple instances of build environments usually makes it easier for me to manage broken builds. I just revert back to a previous build and offload processes to an appropriate environment new or old. Counting what works and were, sticking with it were it counts moving or fixing it when it's a must. Not sure this applies everywhere especially in a production environment. Depends on your process and goals.