Reorder [basic] before [lex]

[zygoloid]

[basic] would make more sense if it appeared prior to [lex] rather than after. Conversely, [lex.separate] and [lex.phases] really belong in [basic], not [lex]. (We should also consider moving [cpp] to immediately after [lex] so that the top-level clauses are in a more logical phase-of-translation order, but keeping it in exile -- as an appendix of sorts -- has some merit too.)

[jensmaurer]

Literally minutes after releasing a draft with the large C++20 renumbering applied do we come up with even further large-scale rearrangements.

[tkoeppe]

Well, there's always force-pushing :-)

[jensmaurer]

Maybe also move the preprocessor section near lex or merge both into a new "lexical processing" clause.

[basic] has some details that need to move later, e.g. allocation/deallocation function details should go to "Declarations".

Operator overloading [over.oper] should partially move to [expr] and to "Declarations".

[jensmaurer]

I have reservations about moving [basic] before [lex]; this needs more internal restructuring.

[AlisdairM]

I was working on a PR/paper that would perform the suggested restructuring, and it quickly ended up redistributing most of [lex] across [basic] and [cpp]. In hindsight, the contents of [lex] truly should precede the contents of [basic].

One reorganisation that I believe did help was moving [cpp] to directly follow [lex], per the phases of translation, and then moving [modules] directly after the relocated [cpp] to put all the program composition and source transformation together, before starting on trying to interpret its actual contents. That involved also creating and deploying \firstcoreclause and \lastcoreclause to match \firstlibraryclause and \lastlibraryclause in config.tex.

[cor3ntin]

lex.phases and lex.separate don't belong in lex, but lex really is better before basic so I don't see a good solution. I really would be against splitting the rest of lex. lex is starting to be in good shape.

[modules] is in sort of a weird place - but it needs to be after [dcl] to make sense (or at least after basic). [cpp] does belong after [lex]. I think it was pushed to the end because well, it's [cpp].

We could do either of these somewhat targeted changes:

• Move [modules] right before [cpp]
• Move [cpp] right after [lex]
• Move [cpp] right after [lex] and [modules] right after [basic] - giving us lex/cpp/basic/modules - which kind of make sense

The status quo does not prevent me to sleep, this would be a marginal improvement

[zygoloid]

There has historically been resistance to moving [cpp] -- some have said they like having [cpp] as a neutral zone / buffer between core and library :) -- but I think logically [cpp] belongs towards the start of the standard, next to [lex] (actually in the middle of [lex] as it stands...).

There are some other issues in the presentation of [lex], mostly stemming from preprocessing token concerns getting mixed up with token concerns and phase 7 concerns. For example, most of [lex.ext] is talking about phase 7 things, and various other sections on literals talk about the meaning of expressions rather than anything to do with lexing.

If I were doing things over from scratch, I'd have a section describing phases 1-3 (and only phases 1-3), then a section describing phase 4, and I'd remove phases 5 and 6 (and instead specify that certain phase 7 grammar productions consume a sequence of string-literal tokens, not just one). And then a bunch of phase 7 stuff.

[AlisdairM]

I have been working to address this issue in the background, and finally pushed my suggested refactoring in #7153.

This is NOT mergeable as it is based off something closer to N4981, the pre-St Louis draft. It is a vision of changes I would like to incrementally apply for the next mailing.

In particular, it gives up on merging parts of [lex] and [basic] as I could not find a satisfying merge, and instead focused on doing its best job on putting up front the parts that specify how to arrange source text to build a program.

The most obvious change is merging [cpp] into [lex], as both were small clauses the combined clause is closer to the average page count, but still low. I looked at merging the similarly small [modules], but did not find a good factoring at this time.

[AlisdairM]

I have withdrawn the previous PR in favor of the more ambitious #7272 that completes the refactoring of [lex] to become a full clause on program construction. It comprises a sequence of many changes, some smaller, some larger, but the PR is intended for a design review of direction rather than an actual attempt to merge.

[AlisdairM]

Now that the major clause cleaning has been done for C++26, should we defer this issue to C++29?

[jensmaurer]

We're certainly not doing this for C++26.

[AlisdairM]

https://wg21.link/cwg2670 might also relevant to any such reorganization.