conditionally parsing code via the `%if feature` pragma

[StefanKarpinski]

When one changes the language, it forces packages to choose between using a new feature and only working on new versions of Julia or not using the feature but supporting new and old versions. For non-syntactic changes, you can often work around this with fairly simple conditional definitions. For syntax changes (which are blessedly rare), this doesn't work. To allow code to gracefully handle new syntax while still parsing in older versions of Julia, we're going to introduce, ironically, a new syntax feature: pragmas. The first pragma will be the feature pragma, used as follows:

%if feature conditional_modules
# write code that uses the conditional_modules feature
%else
# fallback code that doesn't use the conditional_module feature
%end

A Julia parser will have a list of feature names. If the parser recognizes a feature name, it parses the file as if the source were the code between the %if and %else or if there is no %else until the %end. If the parser doesn't recognize the feature name, it parses the file as if the source were the code between the %else and the %end or as if the source were empty if there is no %else.

In general, %if pragmas nest:

%if A
  %if B
    # A and B
  %else
    # A but not B
  %end
%else
  # neither A nor B
%end

There should probably also be a %ifelse to handle the classic ambiguity here. Other pragmas, should we introduce them are single-line.

[vtjnash]

I'm not sure this is really necessary or a good idea. It reminds me too much of the bad parts of C. We can already do conditional include. Perhaps we should just have a basic feature test check function (ala python's future class), rather than depending on VERSION.

[tkelman]

Or feature test macro...

@supports_foo begin
...
end

no, yeah, nevermind, still needs to parse if it's a macro. ignore me.

[JeffBezanson]

The problem is that this is the only way I can see to skip invalid syntax. If you use conditional include, you might have to copy code into both files. On Jun 28, 2014 5:17 PM, "Tony Kelman" notifications@github.com wrote:

Or feature test macro...

@supports_foo begin ... end

— Reply to this email directly or view it on GitHub https://github.com/JuliaLang/julia/issues/7449#issuecomment-47440030.

[StefanKarpinski]

@tkelman – macros don't help if there is new syntax that was previously a parser error.

@vtjnash, how do you propose to support both new and old versions across syntax changes without forking your code entirely? I know this is ugly but maintaining compatibility is an ugly business.

[ivarne]

We can also document this feature to be removed before Julia 1.0, if we don't like the look of it.

[mlubin]

We should be making it easy for package authors to maintain compatibility with the latest stable release, so I'd say this is a reasonable approach even if it's a bit ugly.

[vtjnash]

@StefanKarpinski and Jeff just use more include files!

Honestly, I just don't think this syntax will help improve compatibility, but I do think it will significantly impact readability

[StefanKarpinski]

This is only something to be used in dire need. It's not like this is meant to be used in any kind of normal programs so I don't see how it can affect readability. You use it when you absolutely have to.

[vtjnash]

I guess I don't see how that is much of an argument. Saying that this will be a rarely used feature doesn't make me feel that that this is a must have language feature. (And if the % character is actually up for grabs, I'm sure we could find better used for it).

I would rather not have this hastily implemented for 0.3, and just not have conditional modules for a while longer (or add them in some later RC, since they are a non-breaking change)

[ViralBShah]

I have been thinking about this since yesterday, and I would rather not do this feature hastily this late into the release cycle. I realize the importance, but in balance, I think we are ok breaking some eggs here.

[StefanKarpinski]

I'm not sure how much you guys have tried to maintain installations of packages and code bases using them, especially ones that are used by people on multiple different Julia versions, but this is a crucial feature. We are not all Julia developers who get to live with a bleeding edge Julia version. I'm still maintaining a setup with a 0.3-prerelease from two months ago. I had to patch several packages to make it work. Conditional modules don't break backwards compatibility but they completely wreck forwards compatibility, which is exactly the point of this feature.

[vtjnash]

As author and maintainer of Gtk, Glob, Polynomial, WinRPM (among others) -- all of which work on 0.2 and 0.3 AFAIK -- I don't see a strong benefit to this. It makes more sense if we intend to move to rolling release. I think we should just instead try to have more frequent release milestones.

[lindahua]

I have no strong opinion of this feature.

However, I am not sure this should be 0.3 stuff. There has been a long waiting of the 0.3 release, and it seems that there remains a lot of blocking issues. We should be cautious of adding more to that list.

Things that do not require a breaking decision and that are likely to run into long debates may be deferred to 0.4 cycle.

[staticfloat]

I think we do need good, solid ways of attacking this problem as it's going to continue to be a problem, but I'm with Jameson on this one, I don't really like this, as it feels too much like the #ifdef soup you get in C code when using autoconf, debug flags, etc...

Since we're specifically talking about syntax changes such that older parsers would get confused, I think using conditional includes is a better way to control what gets passed to the parser; no new syntax needed, identical functionality is preserved, and the only downside is that authors will need to write (N + 1) files instead of just one, where N is the number of mutually exclusive julia syntax subsets. As Stefan states, this is hopefully going to be a rare occurrence, so this shouldn't happen too much.

[tkelman]

Having to split things out into separate little files for every different piece of syntax also feels a little too reminiscent of old-style-Matlab-OOP. #ifdefs are ugly and I don't think anyone likes them, but they do serve a valid purpose. Another alternative to conditionally including snippet files without requiring new syntax would be conditionally eval(parse( on a here-string, but that sounds even worse.

[ViralBShah]

I fully support the idea. I am just not comfortable trying it this late in the release process, as doing it hastily just means that we will not get it right.

[tknopp]

While I do not really have an opinion on the proposed idea yet I would like to second @vtjnash opinion that it is a good idea to have more frequent releases. I can absolutely understand that in the open source world releases are something that are "evolving" as it was hard 9 month ago to predict all the great changes that made it into master until now. Nevertheless, it might be a good idea to say that we make e.g. a release every 6 month and the features that are not in after 5 month (feature freeze) have to wait for the next release cylce.

[ViralBShah]

Originally we talked about for 3 month releases which is clearly not practical. 6 months should be OK. Let's move this discussion to a different place though.

[tknopp]

Yes sorry this is of course a little off topic.

[quinnj]

@ViralBShah, I think the idea is this needs to be in 0.3 so that packages developed in the 0.4 cycle will be able to have backwards compatibility with a single package.

I'd be willing to try this out. I say we do it now and try it out during 0.4, with maybe an asterisk attached that says this is a very experimental feature that may not survive if it proves too much a hassle. This should be pretty developer focused anyway.

[StefanKarpinski]

@ViralBShah, I think the idea is this needs to be in 0.3 so that packages developed in the 0.4 cycle will be able to have backwards compatibility with a single package.

This. The key aspect of this feature is that not much needs to be decided – the above description is a complete design. If we don't include this in 0.3 then there will be no way aside from conditional includes to support both 0.3 and 0.4 with the same code base. Maybe that's ok, but maybe not. Regarding conditional includes, can all top-level constructs be done in a conditional include? I'm not certain that they can. If they can't then it's actually not a sufficient solution to this problem.

Another significant point is that #ifdef is so common in C because it's the only conditional compilation feature in C. We have lots of conditional compilation features and once the language stabilizes, this will become almost entirely unnecessary. But while we're developing the language, this is needed.

[vtjnash]

I would like to see a practical example of how this feature would help compatibility. The whole point of adding new syntax is enabling things that couldn't be done before, not giving simple alternatives. For example, conditional modules allows structuring code with dependencies in a completely different (and this less coupled) architecture. For another example, immutable types allowed completely different optimizations which were not previously achievable without significant effort via bitstypes. The common trait though is that a %if block does nothing to minimize code duplication or maintenance for these situations.

(All constructs can be done in a conditional include, or base wouldn't be able to compile -- the parser doesn't care if a file was conditionally or directly included, although in the future, code caching might)

[StefanKarpinski]

Simple example based on @tknopp's example:

module A
    export myf
    myf() = 1
%if feature conditional_modules
    module AB when B
%else
    module AB
%end
         importall ..A
         using B
         myf(::B) = 2
         # lots of other code to support using A and B together
    end
end

Writing module AB when B is a syntax error in an older Julia version. To work around this using conditional includes, you have to have two very slightly different copies of the entire module AB block – one with the when B and the other without it – everything else is identical.

If this facility isn't added to 0.3 then any usage of new syntax in 0.4 will force the packages using the new syntax to be 0.4-only. If we actually mean it when we say that we're going to maintain support for 0.3 better than we did with 0.2 then we need this feature in 0.3.

[tknopp]

I have a little mixed feelings about this and can understand both arguments. One real issue is that package authors use master features in released package versions. I.e. when conditional modules hit master in the 0.4 development phase, IMHO no package should be released with that feature until 0.4 julia is release. It can be in the package master version but not in released versions for 0.3 julia. And to make this more practical so that we don't have to wait too long that feature lands in packages, shorter release cycles would be handy.

[StefanKarpinski]

If packages don't use features during the release cycle, how can we know if the features work?

[tknopp]

Isn't there a possibility to specify that a certain package release is only available for julia master? Sorry if I got this wrong. I am still a newbe in the Pkg arena.

[lindahua]

Maybe, packages that use this can require Julia 0.4 pre-release?

[StefanKarpinski]

No worries. There is no way to specify only supporting Julia master and I'm not sure what that would mean since master is constantly changing – how would you know if the copy of master you have is new enough? One of the nice things about the parser including a list of features is that you don't really care about which version you're using, just if it supports a syntax feature or not.

[vtjnash]

Right, as I mentioned, this starts to become necessary if we want to change to a rolling release without version numbers. Although we could also just start making minor version increments of master at certain times -- for example: even/odd alternation of the minor number for stable/unstable, like the Linux kernel.

[StefanKarpinski]

This does help with rolling releases but the problem exists even with the current release process. Using conditional includes is a work-around but potentially requires repeating a lot of code.

[JeffBezanson]

What if 0.4 breaks a bunch of other things too, like checking integer conversions? It may not be realistic to support both versions with the same code.

[mbauman]

Is it too ridiculously complicated to allow specifying hashes in the REQUIRE version? E.g.,

julia 0.4-SHA_HASH # Require commit that adds conditional modules
MATLAB 0.1.2+f3a5f56 # Require a bugfix I committed and need but hasn't been tagged yet

[lindahua]

Checking whether a commit is strictly behind another is nontrivial.

Better way is to ask the packager maintainer to bump. (I will bump MATLAB later).

[lindahua]

The %if %end really reminds me the unpleasant mess of #ifdef ... #endif in C/C++.

I am afraid that if we introduce this, we are opening Pandora's box. Frankly, @StefanKarpinski's example above begins to hurt readability ...

If a package depends on certain set of features available after the latest release, we may allow richer format of the REQUIRE file, such as

julia 0.x: conditional_module, ...

[mbauman]

Hah, didn't mean to call you out, @lindahua! I don't think it's necessary to tag every bugfix. There's no rush here.

This is a common thing I run into. I often find myself on master versions of packages when I'd really be happier on a known-good commit until a new version is tagged. Every now and again, I find myself going through packages trying to figure out if I can free them yet. I simply suggest it to see if this could be a solution for the bigger problem here. But, yes, I'm well aware that git commit ordering can be a nightmare.

[tknopp]

@StefanKarpinski just to clarify: I meant 0.4 prerelease when I said "julia master". So new feature should use an appropriate REQUIRE where the 0.4- is specified (not 100% sure if this is the right terminology).

[StefanKarpinski]

If @aviks's experience, which matches mine, is any indication, it's likely that many people will use periodic snapshots of Julia rather than just staying on the previous stable release.

[mlubin]

@StefanKarpinski, that's likely due to the slow release cycle and lack of support for the latest stable on the part of package authors. Package authors can't be expected to support multiple arbitrary snapshots of master, and this feature only protects against one particular kind of breakage.

[StefanKarpinski]

It's true, but I've found it quite easy to submit patches to maintain support for the particular snapshot I'm using, which alleviates the package maintainer's responsibility.

[JeffBezanson]

Maybe we need a policy of releasing a new minor version immediately after any breaking syntax change.

I also see people using periodic snapshots from master rather than the last stable release. Late in a release cycle this tends to work fine, which indicates that there should have been a tagged version at some point.

[StefanKarpinski]

But people are using pre-release snapshots. What we'd need is something like 0.3-dev+1 etc.

[JeffBezanson]

What I'm saying is, do all breaking changes within 3 months, then release 0.4. 0.4 would not be expected to be highly stable until 0.4.x, and we would keep working on the 0.4.x series until we feel it is stable enough. Then we branch, and do breaking changes for 0.5.

The idea is to do releases once breaking changes are in, not just after they're stable.

[StefanKarpinski]

Ok, that's a bit weird though and not quite how these things are traditionally done. Maybe we need to start doing 0.4-alpha, 0.4-alpha+1, then 0.4-beta, etc. until it seems stable and then release 0.4.

[StefanKarpinski]

This is also all getting rather far afield from the original issue.

[mlubin]

:+1: for -alpha and -beta tagged releases.

[tknopp]

I like Jeffs proposal although I also would call them alpha/beta or just pre-releases. 3 months to get features in, then the tagged prerelease. And then one month stabilizing. release. (maybe 3 months are too short though)

[JeffBezanson]

Yes, it seems like it would be very useful to have alpha releases, to identify the point where major breaking changes happened.

[vtjnash]

not quite how these things are traditionally done.

Libuv has been doing this via 0.1.1-unstable, 0.1.2-unstable, 0.1.3-stable, 0.2.1-unstable etc.

The Linux kernel has been doing this with 3.3 (unstable), 3.4 (stable), 3.5 (unstable), 3.6 (stable), etc.

So it's not without precedent to tag more than just the final release series

[pao]

The Linux kernel has been doing this with 3.3 (unstable), 3.4 (stable), 3.5 (unstable), 3.6 (stable), etc.

The kernel hasn't been developed this way since the 2.6 series (which continues to this day) started. Right now on kernel.org, there's longterm stable support for 3.2, 3.4, 3.10, 3.12, and short-term stable support for both 3.14 and 3.15. Whether a kernel is maintained as a -stable tree or not has more to do with how many major vendors used it and invest in that maintenance. Note the skip from 3.4 to 3.10, for instance.

[IainNZ]

To add to the cacophony: I too have the concerns about adding language features, but less strongly than @vtjnash. I think they'd be OK because I don't think they'd be used too much, and where they are used it'd be clear why. Having said that, I like the idea of more frequent snapshotting even more, especially because it formalizes something people are effectively already doing in practice. Something 3 months after 0.3 release with big stuff merged in, like a 0.4.0-alpha, which I recall @JeffBezanson referring to as his preference, sounds really nice.