n3031 - Restartable Functions for Efficient Character Conversions

[ThePhD]

There are a lot of functions that should be proposed for the C Committee's Unicode support. Specifically, we need:

• mbs[r]toc[8,16,32]s, 6 functions
• c[8,16,32]s[r]tombs, 6 functions
• wcs[r]toc[8,16,32]s, 6 functions
• c[8,16,32]s[r]towcs, 6 functions

This is too much to propose at first, and does not include the "single code unit" conversion functions for wchar_t. (Notably, because those are fundamentally broken on Windows and IBM machines).

[ThePhD]

The minimum we should propose is the first and second bullets. We should leave off everything else, and see if WG14 is interested. Asking for literally everything is a recipe for destroying the whole proposal in front of WG14.

[Morwenn]

Have you looked at N2282? http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2282.htm

Here is the feedback from the Pittsburgh 2018 meeting minutes (N2307):

Still require the proper proposal format (including prior art). It is listed in this paper that he has prior art in his compiler. Fred: Can one wide character be more than one c16? Rajan: Yes in general, but the wide characters listed here are for c16’s if given in context. Jens: Would IBM be willing to provide the second implementation? Rajan: Can’t speak for my company right now. The committee likes the direction but wants to see other implementations of this before standardizing it.

EDIT: and I just found some SG16 notes about N2245, the previous version of the paper, so you probably already knew about it x)

[ThePhD]

I did know about it, and I met the author here. We coordinated. :D

[ThePhD]

N2431 - Latest

[ThePhD]

N2440 - Latest

[ThePhD]

N2500 - Latest

[ThePhD]

This entire paper is going to be reworked. The current facilities are insufficient due to their intrinsic problems and limitations. See this talk for details on how shitty it is:

https://www.youtube.com/watch?v=X-FLGsa8LVc

[ThePhD]

Posted: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2595.pdf

Latest: https://thephd.github.io/vendor/future_cxx/papers/C%20-%20Efficient%20Character%20Conversions.html

[ThePhD]

We finally have a form and wording that's ACTUALLY good.

Also, we got approved to do a UTF version of the paper too, so the identical wording is applied there too:

https://thephd.github.io/_vendor/future_cxx/papers/C%20-%20Efficient%20UTF%20Character%20Conversions.html

https://thephd.github.io/_vendor/future_cxx/papers/C%20-%20Efficient%20Character%20Conversions.html

[h-vetinari]

Is N2730 still in the cards for C23?

[ThePhD]

I need to publish by October 15th. Same with any other papers that need to make forward progress.

[h-vetinari]

I thought that as a proposal that was already seen, it would still have time until the end of the year? I mean, from a process perspective, earlier is always better, just trying to not put you under even more pressure than you put yourself under already 🙃

[ThePhD]

Until March, basically, is my understanding.

[ThePhD]

(March 2021.)

[h-vetinari]

N2595 (resp. N2730) should probably get a mention as "Continuing proposals" in the agenda document...?

@h-vetinari: I thought that as a proposal that was already seen, it would still have time until the end of the year?

@ThePhD: Until March, basically, is my understanding. (March 2021.)

I'm not sure I understand (as that's in the past; unless you mean 2022), but the schedule doesn't seem to allow anything but editorial changes after the Winter meeting, which has a paper deadline until 2021-12-31? That is, assuming that the paper needs to be voted in as usual.

PS. Just saw the following, which might help? 🙃

1.11 Note where we are in the C23 schedule [N 2864]

Note: Due to the large influx of last-minute documents, this schedule is now unlikely to be met. We should do our best but will probably need an extension.

[ThePhD]

We might schedule an extra meeting, but the point is to see all new material by the 2021-12-31 deadline. Given how much interest has been focused on C23, we might need to process papers better. We used to process papers really fast in face-to-face meetings but with the lack of those now thanks to COVID and us denying the Portland 2022 meeting, we've got to make do with remote meetings for now.

My understanding is that anything can be done up until:

2022-02-28 Editor revises draft 7 2022-03-07

So by February 28th. That's the last time I can edit the draft. I understand it says editorial but with the volume of changes to make I expect to be merging, fixing, etc. features until basically the last minute. Of course, if the C Committee controls that as a whole ultimately: if they stop working on papers and approving/disapproving things, then the work stops. I will merge every paper that goes through.

[ThePhD]

(Also, March 2021 is definitely a typo. It was March 2022.)

[h-vetinari]

We might schedule an extra meeting, but the point is to see all new material by the 2021-12-31 deadline.

Maybe we have a different interpretation of "new", but the schedule says quite explicitly:

Deadline for initial proposal of new features: 2021-10-15

I'm guessing this doesn't apply to N2595/N2730 because it had already been seen. To be honest I'm not nearly as deep in the process as you are, but just from reading the timeline, it sounds like "all papers still wanting to get into C23 need to be submitted by 2021-12-31", followed by one last meeting that votes in proposals (or not).

I get that the actual editing will take longer and has different deadlines, and if there's another meeting due to the volume, great! That's now a different topic though from my question above: whether N2730 still in the cards for C23 (I brought up the current schedule because - being past the "new features" deadline AFAIU, it would have to appear under "continuing proposals" to still have a chance procedurally)

[ThePhD]

N2730 is still on the agenda and still in the running. Success would be greatly increased if I released cuneicode and also put it into another stdlib before that time expires.

[ThePhD]

Hey, just want to confirm that:

• This paper is still on the agenda
• I am still working on it (and rolling a public implementation, see: https://github.com/soasis/cuneicode)
• The "final document deadline" has been pushed back towards February 2022 because there was an influx of papers we couldn't quite see just yet

[h-vetinari]

Yes I saw, congrats on the first public appearance of cuneicode! 🥳

The "final document deadline" has been pushed back towards February 2022 because there was an influx of papers we couldn't quite see just yet

Does that mean that there'll be an extra meeting too? Before Strasbourg? Or the whole C23 schedule is pushed back by 3-6 months?

[ThePhD]

Maybe an extra meeting, but also we might push the schedule.

https://twitter.com/__phantomderp/status/1462329618313224201

[h-vetinari]

All tests passed (1206695680 assertions in 2 test cases)

1.2 billion tests sounds like a casual weekend, I reckon. 🙃

(seriously, mad props for aaaaaaaall of this)

[ThePhD]

N2902.

[h-vetinari]

The paper is not unfortunately still not mentioned in the February agenda... Maybe this needs to be brought to someone's attention so it can still be scheduled/considered? (I'm aware of how silly it is to ask the C Editor that, but better safe than sorry...)

[ThePhD]

Shrug, guess I should e-mail the convener, though I think it's too late to change the schedule now. ¯\_(ツ)_/¯

I'll try to see if I can't add it, and if not then at least get it thrown in the "overflow" bin.

[h-vetinari]

There have been schedule corrections in the past, so 🤞

[ThePhD]

Further inspection reveals that all the papers listed in the WG14 document log are their old versions. For example, Modern Bit Utilities is r0 when there's an r1, the typeof paper is the older revision, etc. etc.

This is all likely due to the fact that I was slow on the draw. Papers have to be published a month in advance, to give adequate time for the reading period. I missed it, so only old stuff is on the agenda.

I'll still advocate for the new versions, but it might mean this stuff gets pushed back to later.

[h-vetinari]

(Please tell me to stop if this is annoying you) Just noticed that the updated agenda for the second half of the Jan/Feb meeting already refers to several new papers (most recent one being N2924 from end of Jan), but still does not mention this paper.

[h-vetinari]

I say this i.a. because I'm kinda-sorta expecting another agenda update, seeing that some scheduled papers in the tail end of the February half (including typeof() 🥳) have already been decided.

[ThePhD]

It's simply not scheduled for this meeting. There may be time at a later meeting given what we've voted to do with our upcoming meeting schedule. There's also no time to change the agenda now because people need 2 weeks prior notice to read a paper.

[ThePhD]

@h-vetinari This is more or less the final form. Let me know if you have any questions or comments. It's the last chance to make it for C23, I think, so it's gotta be as good as I can make it. This paper - and your wishes - are my priority here:

https://thephd.dev/_vendor/future_cxx/papers/C%20-%20Restartable%20and%20Non-Restartable%20Character%20Functions%20for%20Efficient%20Conversions.html

[h-vetinari]

I think, so it's gotta be as good as I can make it.

I think it's excellent, thank you so much for your work on this! Unfortunately (or fortunately?) I don't have much more design feedback to give, but 🤞🤞🤞🤞🤞🤞🤞 that this still makes C23!

The one thing from looking at it again is that I think that section 4.1 could use a more explicit conclusion subsubsection to explain which form was chosen. As in: remove "Therefore, we keep the double-pointer form to retain the information properly." in 4.1.1 and add 4.1.4 à la:

4.1.4 Proposed choice

Given the above considerations about consistency with other functionality (no struct returns), the downsides of function forms (2)-(4), and the benchmark indications that the impact of doubly-indirected pointers is negligible in bulk, we therefore propose the double-pointer form, in order to retain the requisite information properly.

Also, the addition of the restrict keyword to all the pointers could be mentioned in the changelog (and its use perhaps explained in a short paragraph...?).

[h-vetinari]

Was it intentional that the new batch of papers has a release date 2 days after the WG14 deadline (though they were all effectively ready before, i.e. modulo https://github.com/ThePhD/future_cxx/commit/5c7d1aaec788cf6ccdb6d8765acda038c3068ab7), or are they still going to make it for the July meeting?

PS. There's a bunch of C-issues on this tracker that can be closed by now 🥳 PPS. If you're willing to give me triaging rights, I could do the trivial house-keeping parts here (updating N-numbers, closing completed issues, moving milestones where necessary).

[ThePhD]

I posted it late because I'm tired and forgot the deadline, despite requesting the paper numbers in-time on the 4th of this month. I can only hope I'm allowed to carry it forward, because otherwise every single paper I've written won't be allowed. No unicode functions, no enum improvements, etc.

[h-vetinari]

I can only hope I'm allowed to carry it forward

🥳

[h-vetinari]

Infinite sadness. I have no idea what happened (and will hence refrain from colourful adjectives), but it does sound a lot like "oh, there's someone who spent years working out a solution to a decades-old problem that affects everyone - that's interesting, next." If that's even remotely close, then you truly didn't deserve that.

[ThePhD]

Easy, there. There were just a lot of wording suggestions made and doing them now might constitute too many changes to make it into C23, delaying it for the next revision.

I was still given the ability for a Homework Paper, so I need to re-do the bad wording bits and change it to be better. But, if the changes are too significant to process, it could still be rejected under procedural grounds.

[ThePhD]

If it's rejected procedurally, we fail. If it isn't rejected procedurally but the wording sucks, we also fail.

There's effectively a lot of room for failure, and this is me we're talking about here.

[h-vetinari]

I was gonna say:

There's effectively a lot of room for failure, and this is me we're talking about here.

But you're right. With you at the helm, this doesn't look impossible. 🤞 🙃

(Also, not sure how good the minutes will be to give guidance on the re-wording demands, but happy to help if I can)

[ThePhD]

If you would like to make critiques of the edits, now would be a good time to (important changes are highlighed specially, as shown in the changelog):

https://thephd.dev/_vendor/future_cxx/papers/C%20-%20Restartable%20and%20Non-Restartable%20Character%20Functions%20for%20Efficient%20Conversions.html

[h-vetinari]

Just checking back in - there were still two outstanding changes and two open (AFAICT) points from https://github.com/ThePhD/future_cxx/commit/8575bad3ad28478056f9fbb530fca84d5d41332a, and contrary to the enum papers, this isn't visible as published yet.

[ThePhD]

I think I addressed those in the next commit, no? And they're not published because I sent it to another person (who made the original request for changes) to see if they wouldn't mind checking it over for feedback.

[ThePhD]

Oh my god, the commit got lost in all my attempting to rig the submodule up correctly, AAAA!

[h-vetinari]

Oh my god, the commit got lost in all my attempting to rig the submodule up correctly, AAAA!

It's not the end of the world. Fixing the grammar can be done in an editorial fix I guess?

Also, since n3031 isn't published to the WG14 document log yet, I guess there's still a small window that a correction might arrive in time.

[h-vetinari]

Oh my god, the commit got lost in all my attempting to rig the submodule up correctly, AAAA!

It's not the end of the world.

Ah, even better - I had misread that you had already sent it for publication. So coming up with those changes again should definitely be in the realm of possibility. ;-)

[ThePhD]

Pushed again. Look it over if you like!

[h-vetinari]

Last (🤞) round of comments in one place:

• double "the the" 1, 2
• possibly missing footnote
• possible clarification in the change log
• "embodied in" nit

[h-vetinari]

💚 for the kind words and 🙏 for all your work on this. Now all that's left is

[h-vetinari]

I was still given the ability for a Homework Paper, so I need to re-do the bad wording bits and change it to be better. But, if the changes are too significant to process, it could still be rejected under procedural grounds.

BTW, will this still be decided during the July meeting?