Release Candidate plans for 0.2

[staticfloat]

Where better to discuss RC policy that on an issue tracker?

After seeing https://github.com/JuliaLang/julia/issues/2597#issuecomment-21534868 and almost replying to @ViralBShah right there in the issue, I thought it might be useful to hash out how we plan to do RC's and the eventual release of 0.2. All my experience with RC's has been with the blender project, which has a pretty intense set of policies regarding when certain pieces of code can get merged into trunk, but I think we can adapt some of their practices to help our releases to go as smoothly as possible.

I'm thinking that we should tag some commit as 0.2.0-rc1, and then push that out to the community at large to see if they can break it. The key here is as widespread testing as possible, followed by bugfixes only making their way into master, everything else happens in a topic branch. Once a predetermined amount of time goes by (for the blender heads it's on the order of a week, I believe) where no new issues come up and we're able to squash all the showstoppers, a release is made. There are usually a few bugs that are found afterward, but unless they're something really terrible a bugfix build is usually not released.

The most important thing here is getting the community to test the RC builds on all their very inventive setups, which means probably announcing things like this on julia-dev, julia-users, etc.... The bugfix week (or however long it is) is also a really good time to focus in on documentation, but let's be honest, when is a time when we WOULDN'T say it's a good time to focus in on the docs?

Discuss!

[ViralBShah]

One immediate task that comes to mind is the preparation of Release Notes, and much of the material is in #2581.

We certainly should go through release candidates. We may end up doing more than one, preparing binaries for each release candidate also, so that those who are not building from source can also help test. Your proposed method seems quite reasonable, although we should hear from @StefanKarpinski on tagging and branching.

Certainly, all doc issues should be addressed, and some kind of a triage of the open issues also needs to be done to separate out what will happen in a future release. Only 2 out of the existing 24 doc issues are marked for 0.2, which does not sound right.

[mlubin]

+1

[StefanKarpinski]

Jeff and I were just talking about this. We absolutely have to have a freeze on master. Now that people are using Pkg2 we should do it soon and hold off on anything but bugfixes on master.

[staticfloat]

Great. I think it would be good for the powers that be to mandate a time and duration for this to happen, so the rest of us can plan our efforts accordingly.

[IainNZ]

A thought on the wider package ecosystem and how it relates to releases - we should really emphasize that package owners should try to be Julia version-aware and be more specific with specifying which version they support. Version 0.1 rapidly became obsolete and many packages gave up on supporting that very quickly. I've noticed a tendency for packages to not specify a dependency on 0.2, which we should not actively discourage in the future. I have a feeling that many of the packages listed under master on METADATA will not work with the 0.1 binary - makes for a bad first impression.

Related to that, I think 0.2 is going to be (and should be) supported for far longer than 0.1 was by package owners, so maybe including some guidance on best practices with regards branches/tagging on packages should be encouraged.

This ties into @johnmyleswhite noble intentions with institutionalizing pan-METADATA testing of packages as well - we'll be able to identify "naughty" packages which say they run on Julia HEAD but are actually really 0.2 dependent, and notify those owners.

[ViralBShah]

The lack of unit testing in many packages would make this difficult. Such an exercise would certainly make packages higher quality and we should probably do CI in METADATA where the updated package's tests are run.

Cc @aviks

[aviks]

To be successful in doing this, we really need to get a fix on @johnmyleswhite Julep on package testing. Once that is done, getting travis to to run all tests will be easy. Currently every package has its own way of running tests. which makes running common testing platform difficult.

[IainNZ]

Agreed. I'm going to throw up a Gist, and lets hash this out.

[IainNZ]

https://gist.github.com/IainNZ/6086173 A Julep for both package testing and quality

[JeffBezanson]

Given that it fixes a bug that came up a fair amount, maybe we should switch to LLVM 3.3 for 0.2? I believe everything works with both 3.2 and 3.3 at the moment, so distro packagers can pick one as necessary.

[nolta]

What is the plan for 0.2? Can we pick a date? It's been over 6 months since 0.1 was tagged.

[ViralBShah]

The current blocker is a couple of Pkg2 things that @StefanKarpinski should talk about. The rough process that I am thinking of is:

• [ ] Declare feature freeze (immediately, say T)
• [ ] Anything marked as 0.2 needs to either be fixed or assigned to 0.3. All of us should get on the task.
• [ ] No new issues to be assigned to 0.2, unless they are bugs or doc updates. Only bugfixes to be committed in this period.
• [ ] RC1 when no more 0.2 issues open and widespread testing of binaries (T+2 weeks). Ensure key packages are tested and updated too.
• [ ] RC2, which hopefully also becomes 0.2, if there are no more reported issues (T+3 weeks)

We may need to move to LLVM 3.3 as discussed above, and given that it will take about 3-4 weeks to release, I am ok with updating it, although I know others have reservations. We almost certainly need to have an openblas upgrade as well, which is due in the next few days.

[staticfloat]

LLVM 3.3 works perfectly for me, the only caveat is that I haven't managed to get it into my PPA yet. It seems to rely on newer build tools than the typical 13.04 buildd bot has.

Rather than pick a date for 0.2, can we settle on a feature set and enter feature freeze? It seems like we've already got a large number of features waiting to land, and a focus on bugfixing and documentation couldn't go over badly.

From what I hear, Pkg2 is working well on Julia. I'm not sure if we want to try and finish up this package testing stuff before then, because anything we decide in that Julep is going to have to be propagated to all the packages that are currently released. That seems like more than a week or two's worth of work.

EDIT: Viral just ninja'd me! I agree with his procedure.

[ViralBShah]

Also worth noting that the MIT summer crew will largely disband in the above timeframe, and it would be good to get 0.2 out in the next 3-4 weeks.

Totally agree that package testing framework should be a 0.3 issue.

I am in favour of an immediate feature freeze. Let's just give some time for others to voice any opinions.

[IainNZ]

I disagree the package testing framework should be 0.3 - I don't consider it a feature of Julia. It will touch the main repo in the sense that the documentation will change, and Pkg2 will display messages/run the tests but its mainly setting a consistent style for packages. I think it'd be shame to put it off to 0.3.

[Keno]

Whatever we do we should create a release branch, so we can start putting some of the breaking changes we have for 0.3 on master.

[staticfloat]

Whatever we do we should create a release branch, so we can start putting some of the breaking changes we have for 0.3 on master.

Agreed, but we should reach a consensus on the further breaking changes we need (if any) that should be applied to master first. I'm mostly hoping that Jeff/Stefan will jump in here and either agree with Viral that we should freeze now, or tell us that there's something really important to include in 0.2 first.

I think it'd be shame to put it off to 0.3.

What I don't want is for us to release 0.2 with a bunch of packages that are failing their new tests, don't yet have tests defined, or are half-conforming to a new standard and half not. I would rather release now with a packaging ecosystem that largely works, than impose new regulations on the packaging ecosystem that it might take a while to adapt to. Especially since a large number of packages aren't even under the control of the people commenting on that gist.

I might be overstating the difficulty, it just sounds like there is a sudden interest in getting a stable 0.2 out for release quickly.

[ViralBShah]

@IainNZ You are right that the package testing framework does not have a dependency on base, except documentation and messages. It would be great if it is ready and adopted by the commonly used packages in the 0.2 release timeframe. That said, many of the same people are likely to be working on getting 0.2 stabilized. It is worth pointing out that this can be put in place even after the 0.2 release, and it would still work out for the 0.2 release, so long as the packages follow the convention.

[IainNZ]

Yes @staticfloat's argument is quite compelling. The package system does, in fact, work as is. I now think we should not block 0.2 on this package ecosystem 'robustness' work. This certainly gives us more time to think about how to do it right and make sure its a long-term solution.

[JeffBezanson]

I agree with Viral's proposed process. My last major task for 0.2 is #1571. I would also kind of like #3385 in 0.2 but I understand if others think it's impractical at this point. Also as many API changes like #3405 as possible.

[StefanKarpinski]

I'm also down with this release process. The only things I feel need to go into 0.2 still are the API changes that @JeffBezanson mentioned and additional API changes or features for Pkg2 – and, of course, deleting Pkg and renaming Pkg2 to Pkg.

[mlubin]

Have you guys though about how to structure METADATA? Will 0.2 still point to the devel branch?

[IainNZ]

@StefanKarpinski "additional API changes or features for Pkg2" would this include #3540?

[StefanKarpinski]

I would consider breaking package support for 0.1 by renaming devel to master, but it's a bit unclear to me how many people out there are actually using 0.1.x with packages that this would have an impact on. We could also maintain a release-0.1 branch of METADATA and give people instructions for using that.

[StefanKarpinski]

Sure, #3540 could be considered. Since adding new functions to the Pkg2 API doesn't break anything, we could consider it fair game for 0.2.x, however (since we haven't yet reached a fully stable version).

[IainNZ]

I personally think 0.2.x would be best, i.e. not making it a blocking thing on 0.2, for that reason. I also think it could be good to "break" 0.1 packages by changing METADATA - I have a feeling most of the package ecosystem is broken for 0.1 anyway, due to people not maintaining 0.1 versions (I know I haven't) and not specifying a Julia version dependency (part of my crusade).

[mlubin]

I also don't have any issues with breaking METADATA for 0.1, but it should at least be structured going forward so that breakage doesn't happen again. Should 0.2 look for a release-0.2 branch instead of master?

[IainNZ]

If we only accept METADATA pull requests that specify a Julia version in there REQUIRE, we should be ok - short of some fundamental re-think of how METADATA/Pkg work. Specifying a branch would be the most robust approach though, at the cost of maintenance/confusion

[StefanKarpinski]

The fundamental issue with 0.1 was that there was that 0.1's Pkg didn't know how to take into account dependencies on the version of Julia itself. Now that we have that, such future breakage won't happen. If a version of a package doesn't work with with certain versions of Julia, just indicate that the version's requires file in METADATA. As long as we maintain compatibility of the structure of METADATA going forward, things will be fine and we won't have to maintain a branch of the repo per Julia version (I'd really like to avoid that). We can add new files that older versions don't know to look at, but we need to maintain the previous ones as they are – which is one of the reasons I've been a bit slow to add new things to METADATA.

[stevengj]

Let me make a plea for #3932 to make the feature freeze. Without this it will be very hard for packages to support multimedia output in IJulia.

[dcjones]

Let me make a plea for #3932 to make the feature freeze. Without this it will be very hard for packages to support multimedia output in IJulia.

Seconded. I'm eager to use this.

[StefanKarpinski]

New variable bindings on each for loop and comprehension iteration: #1571.

[Keno]

@StefanKarpinski Did you mean to post this in https://github.com/JuliaLang/julia/issues/2581?

[StefanKarpinski]

Uh, yeah. Sorry.

[stevengj]

Is there any update on this? It would be nice to get this out before the semester starts, and the major issues listed above seem to be fixed.

[staticfloat]

@StefanKarpinski, will we remove Pkg1 completely before 0.2-rc1?

[StefanKarpinski]

yes, that's the idea.

[stevengj]

I'm going crazy depending on 0.2 prerelease binaries, because we are constantly getting regressions. The binaries work one day and break the next, and I can never tell the students it is working. Can we get something that verifiably works for an IJulia binary install on virgin Mac/Windows machines and release it?

Can we at least get a list of release-blocking issues?

[StefanKarpinski]

The only holdup at this point is OS-specific dependencies in Pkg. I'm on my way from speaking at Strange Loop to a wedding in SF, but I should have some time tomorrow – or maybe today on the flight I'm about to take – and can try to get that working. Then we're good to release as far as I can tell, except for documentation, which can be finished after releasing a binary.

[mlubin]

I'd suggest officially tagging, releasing, and announcing a 0.2-rc1 binary and letting that settle for a few days to make sure there's no more show-stopping issues for users. Package maintainers should also test their packages on as many platforms as possible and clean up deprecation warnings.

[StefanKarpinski]

Yes, that's a very good idea. Let's do it.

[stevengj]

(I assumed that was standard practice.) And we should be sure to test the binaries on some virgin machines, since a lot of the regressions have been due to accidental dependencies (e.g. assuming the user has git, or gfortran, or ...) But obviously releasing rc binaries has to wait until we actually have a release candidate.

[staticfloat]

Note; to get the version numbers to align properly, you need to edit VERSION to contain 0.2.0-rc1, commit that, and then git tag v0.2.0-rc1. That will get the splash screen to look like this:

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0-rc1
 _/ |\__'_|_|_|\__'_|  |  Commit 7b2c1d3 2013-09-21 19:16:44 UTC
|__/                   |  x86_64-linux-gnu

[Keno]

May I suggest that we just cut a release branch now, make master target 0.3, and do the few things that are missing for 0.2 on both branches?

[ViralBShah]

I am all for it, but someone needs to convince @StefanKarpinski :-)

[mlubin]

I'm worried this could distract developer attention from releasing 0.2.

[IainNZ]

Would be good to cut an RC this weekend at least because the Forio guys and @mlubin and I are (seperately) presenting Julia stuff at INFORMS2013 to an audience that is perhaps a bit more conservative when it comes to trying pre-release stuff compared to CS types - if only for the psychological factor of having "RC" instead of "pre".

[ViralBShah]

Sorry about not being able to get an RC out earlier, but we are trying to get it out today if some of the Windows issues can be sorted out.

[staticfloat]

Awesome! Once it's tagged I'll generate 10.6 OSX, 10.7+ OSX, and Ubuntu packages.

[JeffBezanson]

OK, let's make commit a963262672fdb79a90f2bf455fc34dab235fbb50 v0.2-rc1.