A new category of standard features: experimental

[rouson]

Motivation

Forall and do concurrent are commonly assumed to offer the potential for better performance than do loops, but both were ultimately found to have design problems that preclude important optimizations, notably parallelization, in important use cases. Fortran 2018 declares forall obsolescent but attempts to improve upon do concurrent despite known standard-conforming use cases that a compiler cannot safely, or even in principle, parallelize. A proposal was submitted to J3 to address do concurrent parallelization, but the committee did not adopt the proposal, presumably because the proposal changes the semantics of existing, standard-conforming code.

The value that the committee places on not breaking existing codes makes a replacement feature attractive. The problems found in forall and do concurrent suggest anticipating the likelihood that any replacement will need future revisions to ensure the desired aim even if those revisions break existing code.

Proposal

Annex B of the Fortran 2018 standard lists the first two categories of features enumerated below. I propose to add the third:

1. Deleted features were "redundant and considered largely unused in Fortran 90 or 2008."
2. Obsolescent features are those which "were redundant and for which better methods were available in Fortran 90."
3. Experimental features could change in a future standard in ways that are incompatible with the current standard.

Experimental features that survive 2 or 3 standards without incompatible changes could be removed from the list of experimental features.

Use Case

An experimental new feature designed to be very similar to do concurrent but with parallelizable semantics could minimize the work of the committee, the compiler developers, and the user community wishing to refactor codes to leverage the new feature. Most importantly, with users forewarned, the committee could update experimental features in subsequent standards even if the updates break existing code.

[certik]

I support this 100%, thank you Damian for opening it. I would go even further and suggest any bigger feature to go into experimental first, such as any feature that does not yet have a prior compiler implementation and hasn't been tested in code by the wider community. The "do concurrent" is a prime example. As we experiment with ways to efficiently parallelize on GPUs, there might be more additions (perhaps specifying array memory layout, or specifying where an array should live) that I would not like in the "Main", but would be great to have in "Experimental", so that we can experiment with it.

[FortranFan]

When it comes to DO PARALLEL or any such new facilities, my personal opinion is to lead R&D - thus Experimental - effort(s) to prototype the feature in some open-source compiler platform(s) such as LFortran and/or LLVM, and/or GCC/gfortran and converge on the optimal solution. And then strive to standardize the features based on results and lessons learned.

With all due respect, I do not think a 3rd category in the standard is viable or even advisable.

[tskeith]

The Java standard has something similar, preview features: https://openjdk.java.net/jeps/12

A preview feature is a new feature of the Java language, Java Virtual Machine, or Java SE API that is fully specified, fully implemented, and yet impermanent. It is available in a JDK feature release to provoke developer feedback based on real world use; this may lead to it becoming permanent in a future Java SE Platform.

Of course, Java is not precisely comparable to Fortran because all new features are implemented before being standardized.

[everythingfunctional]

I am in favor of this idea. We beta test everything else in this industry, why not features of the language themselves. As @tskeith pointed out, there is some precedence for this in at least one other standardized language. Rust is another example of a language with this property (see their "Unstable Features"), even if it isn't standardized.

But I'm not sure this is either sufficient, or maybe even necessary to get what you actually want; for compiler writers to implement new features to be tested out before they get permanently etched into the standard. Compiler writers already have enough trouble keeping up with the new standards. I worry this may give them an excuse to put off implementing these features. On the whole I think it does put us in a better position though, because at least then we don't have to individually convince them to implement features that aren't in the standard.

[klausler]

I agree that the "rank agnostic array addressing" feature is a candidate for this category; it's a trainwreck, and its purpose could be more easily attained with far simpler language tweaks. But that's its own discussion.

Here's a problematic objection that will be raised to quash this idea: language features are usually interdependent, and often in subtle ways. Adding a "feature" to the standard often involves edits to many clauses and new constraints in nonobvious places. Some features, like coarrays, become pervasive throughout the document.

So, suppose one wants to add a feature in this new experimental category. Where do you describe its interactions with all of the other aspects of the language? You can't really just package up all of the necessary edits for a nontrivial feature into a "patch" that then becomes a subclause, but you really really can't distribute the changes over the rest of the document either. This gets harder with experimental features that have unavoidable interactions with each other.

I think this idea will get shot down quickly unless you guys have a good description of how experimental features would appear in an appendix (or whatever) and be able to address all of the interactions with the standard language and other experiments. It may be that the descriptions of these features would be more useful if they remained outside the document entirely until proven (or not) in implementations. You probably also want to be able to edit/fix/extend these descriptions at a much faster cadence than the committee process allows, too; more iterations will generate better results.

[rouson]

Thanks to all who have provided feedback so far. My primary concern is that the committee's understandable reluctance to break existing codes and/or delete features leaves us stuck for long periods with a feature that is known to be broken (forall) or known to have significant problems that cannot be addressed without breaking existing codes (do concurrent).

@everythingfunctional great point. The wide variability in implementation speeds makes me wonder whether the determining factor is something other than what's in the standard. We have at least one Fortran 2018 compiler while there are several that aren't even 2008-compliant yet. I don't think it's the standard itself that makes the key difference in implementation speed.

@klausler I'm imagining that experimental features get integrated into the standard in the same way as any other feature. The only change to the standard would be listing experimental features in Annex B as a way to signal that the committee will be willing to make changes that are incompatible with previous standards if it becomes clear that the feature has significant problems that need to be addressed.