Open jonwil opened 5 years ago
I think the best thing (and I believe they already plan to do it) would be to put the code into a separate branch here on github, but not to ship it in any official form.
The worst possible outcome would be both version being shipped and supported for quite some time in parallel and people to start #ifdef ing their code based on which version the code is built against and incompatible binaries floating around. Imo that would be a disservice to the eco system.
All that aside: I guess one of the problems may be that those patches no longer apply cleanly to the current master version of the code.
Releasing it as a separate branch is the answer for sure. People who want to ship it can grab that branch and get a set of binaries out of it knowing full well those dlls are ABI-breaking compared to the official binaries.
Of course it may not be possible to publish the code at all if the ABI-breaking branch would require changes to the compiler or changes or recompilation of other non-open binaries like vcruntime, vcstartup or concrt (that said, it looks like concrt is the only runtime library component shipped with the compiler that actually links to and pulls in the STL so its entirely possible the answer to any problems with an ABI incompatible STL solved by making concrt open source, something that would be nice to see anyway given the circular dependency between concrt and STL)
ABI breaking STL could be released on vcpkg. To avoid conflicts with the built-in STL, the headers could be located under another folder, like so: #include <stl/string>
We do plan to get the binary-breaking branch integrated here eventually; we effectively must do so to achieve our goal of primarily developing the STL out of this repository. I would in no way refer to those plans as "releasing" so much as "developing", we still have quite a few ABI-breaking changes that need to be implemented on that branch.
@MikeGitb's "I guess one of the problems may be that those patches no longer apply cleanly to the current master version of the code" could not be more spot on. We switched from TFS to Git a couple of years ago, and the ABI-breaking branch lies largely abandoned on our old TFS instance. There will be quite a bit of hand-porting involved, with the likely exception of the threading bits that were completely rewritten. We did pull in the <atomic>
rewrite a few months ago, for example, and that wasn't too bad except for a few days spent to maintain the old ABI.
Are there any cases of abi-breaking changes that need to be made to vcruntime, vcstartup. concrt or other closed modules or any cases where abi-breaking changes require closed modules to be recompiled to work?
I recall some changes being made to vcruntime, in addition to a new "vcplatform" layer being added. I don't recall any ConcRT changes; we'll be able to drop the ConcRT link along with our XP targeting support. We'll need to rethink vcplatform as we reapply the changes to GitHub. Among other things, we clang-formatted the STL's sources in the meantime and changed other conventions, so all of the changes will need to be handled line-by-line.
I thought VS 2019 dropped XP support (or are there enough users out there that still need XP support that bringing it back to VS 2019 is necessary?)
Probably just would break the ABI to drop all the XP support bits
It did drop XP targeting support (in the IDE, etc.) but unfortunately we can’t remove it from our DLLs, as that would damage any XP machines if the redist were to be installed there.
I forgot that the 2019 dlls are also the dlls for versions of VS where the XP targeting support is still actively supported.
Would it make sense to create an entirely separate branch here on github that is just a copy-paste version of you internal TFS branch just to give it exposure to the community (there are tools that would preserve history, so you can actually see the diffs that where applied relative to the old master and not just compared to the current one). The actual integration work could then either be done in yet another branch, or by adding the changes directly into master but hidden behind appropriate #ifdefs
.
Creating a separate branch also creates many process challenges. You'd probably only want breaking changes to go into this STL Next branch and all other changes to go to master. For every change to master, you need to cherry-pick the changes and merge them into the STL Next branch which can be problematic. Just before master is released you probably want to lock it down and only accept high priority bug fixes to it as final testing is done. At this point, the STL Next branch can take non-breaking changes. When master is released you now have a problem of merging STL Next back into master. There are many ways to deal with the myriad problems with managing multiple branches but all of them have many pain points. And not to mention the issue associated with managing separate CI systems for multiple branches. It's not just a matter of creating an STL Next branch and going wild on it. That's a recipe for disaster.
@MikeGitb
Would it make sense to create an entirely separate branch here on github that is just a copy-paste version of you internal TFS branch just to give it exposure to the community (there are tools that would preserve history, so you can actually see the diffs that where applied relative to the old master and not just compared to the current one).
Unfortunately, that will not be possible. We spent several months cleaning up our product and test code for public release (notably, overhauling all of the copyright/license banners, and properly handling all non-Microsoft-written code). That's why this repo's GitHub history begins in September 2019. The vNext TFS branch predates all of that work, so we can't publicly share it as-is.
The history is also a complete mess, partially due to how difficult merging was before git. We don't have a clean "rebased" record of the changes relative to where vNext diverged from v19. We need to extract the parts we want so we can have a clean record.
@SunnyWar
Creating a separate branch also creates many process challenges.
Yes, we will definitely need to figure out how to handle vNext logistically. It would certainly be simplest if v19 development stopped (except for critical fixes) and vNext development started with a purely linear history, but we may end up needing to merge features from v19 to vNext periodically.
@SunnyWar: This wasn't meant as a suggestion to have two parallel development branches. I was just asking if existing work could be made visible to the community, which could then help with the integration process in whatever way the dev team seems fit. I'm not surprised that the code is not fit for the public, but I thought I'd ask.
Working on a new major release while still supporting the old once is certainly not a new or uncommon problem, so there exist a lot of existing practice around this. Personally I'd develop vNext on master when and if that actually becomes a thing and keep a branch for v19, that receives bugfixes and backports via cherry picking, when possible, but is not otherwise kept in sync with master one way or the other.
Of course, I'm not involved in this library and know far too little about the development and testing process of msvc (or Microsoft for that matter) to know if that is actually a sensible strategy here.
Are there any updates on this or plans? Some of the bugs that are tagged vNext are really showstoppers in some cases. Perhaps Visual Studio 2021 with binary incompatible STL to VS2015-2019?
We're currently planning to finish C++20 (hopefully in 2020) before working on vNext (hopefully in H1 2021).
@StephanTLavavej Thanks for the info! But perhaps it would be a nice idea to publish the vNext branch here, even now, so at least the community will start solving stuff, while you are working on C++20? :)
The issues with that are:
The community is absolutely welcome to start working on binary-incompatible overhauls (e.g. regex
would be amazing), but we really can't review them or provide more than the most superficial guidance until next year.
Ah, I see, very unfortunate... Buy anyway, thanks for the honest update and keep up the good work! We've waited 5 years, so we can wait a few more months. :)
At the same time it would be quite valuable to know what you already did.
I think that there is a lot of interest in the community to work on this so the question is whether it is wise for a maintainer to spend a week porting some feature when he could review what a contributor did porting the same feature.
Maybe once there is some more time to breath for you it would make sense to create a vNext project with the different things you already did and the things you plan. I believe letting the community start beforehand is much better than just locking yourself into a 6 month window
Yeah, we can do that once one or more maintainers are available. From memory, the work we accumulated and will need to port is:
vector<bool>
and deque
in time./clr:pure
.<hash_set>
and <hash_map>
.More things that I would like to do, that haven't been done yet:
/Zc:wchar_t-
TRANSITION, ABI
issues throughout the codebase<experimental/filesystem>
and ideally the rest of the experimental machinery that has been superseded<regex>
<random>
and its accumulated layers of TR1 machineryunordered_meow
deque
tuple
- specifically implementing compression and eliminating the recursive inheritance<cvt/meow>
before v19 ends so we can remove it in vNext_HAS_EXCEPTIONS=0
.It would be amazing if we could drop C++14 mode but it would also be amazing if I got a pony and a unicorn.
Just for me, is it possible that we support concepts in all language modes once EDG supports them (Looking at tuple especially)
We'll be able to use concepts in C++20 mode, but not in C++14/17 mode - i.e. it's not like if constexpr
or explicit(bool)
.
😿
Hey again @StephanTLavavej
Time passed, C++20 implementation of MS STL is completed(still waiting for a few fixes from WG, but nothing major). Do you have more clear idea when the work on vNext can be started? :)
@TheStormN Unfortunately, while we originally hoped that we could finish C++20 and switch to vNext, the compiler front-end team is currently too busy to work on vNext changes. :crying_cat_face: While it isn't absolutely required for the compiler to break ABI at the same time as the libraries, it would be sad if only the libraries used this once-in-a-decade opportunity without simultaneous FE changes to fix the empty base class optimization, improve RTTI layout, etc. At this time we don't have an ETA for when the FE and STL will be able to start vNext - I hope it'll be the Dev18 cycle (i.e. next major version after VS 2022) but no guarantees.
I seem to recall statements in the past that the next release of VS after VS 2019 was going to be an ABI breaking release. What happened to that?
- I hope it'll be the Dev18 cycle (i.e. next major version after VS 2022) but no guarantees.
:(
And here I thought this was only about delaying the start for a couple of minor releases. Not a whole nother VS cycle.
@jonwil Long ago, we originally hoped that vNext would be ready for production at the same time as 17.0, but then it became clear that finishing C++20 was going to require a ton of work from both contributors and maintainers so we'd be lucky to reach completeness in 16.10 - which we did! So until recently, we hoped that we'd be able to start vNext now, and have it ready for release after a few updates of Dev17 / VS 2022. What happened is that the FE team had to commit to too many high-priority tasks to have capacity for vNext. I know it's disappointing news, and I am :crying_cat_face: to relay it, as I want vNext more than anyone - I'm the only current maintainer who worked on the STL during the break-ABI-every-release era (which was as glorious for development as it was problematic for customers). However, because plans have a tendency to change, if you look at our previous statements you should find that they were all expressed as hopes/expectations and not promises. We try to be open about our tentative plans for the future, while also trying really hard not to promise anything that isn't certain.
@MikeGitb It's possible that things will change (e.g. some other project is deferred or cut) and we'll be able to start vNext some time in the middle of the Dev17 cycle - I don't think it's likely at all, but it's possible. It's also possible that we might decide to do a library-only vNext release - again, low probability, but I would start arguing for that if it looks like the FE won't ever have the bandwidth for the foreseeable future.
In all fairness - while there are various ABI-related things I'd love to see fixed/improved as soon as possible, I couldn't even begin to quantify how much of a impact this would make for day to day work. Especially compared to other improvement X that probably don't require ABI breaks.
@StephanTLavavej Sad to hear(as you are). I don't know if there is a good way to relay the community feedback on this stuff, but while I think keeping ABI compatibility can ease up migration for large and conservative corporations, it draws back the community in general.
In recent years I think the world became much more agile compared to 10-15 years ago, so keeping ABI for 10 years sounds like a lot of legacy approaches which are hurting more, than helping. Also, while I'm here, providing feedback, I can tell you that I know people around me which are just monitoring silently and the disappointment just motivates them to take the work necessary to move to another platforms/compilers.
Also, I don't know what is actually happening with FE team, but looks like they are well out of numbers to keep up with bug fixing and new features. Perhaps making the FE open source is the next logical step in order to make sure that this compiler will still be used in the next 10 years.
@StephanTLavavej would it be possible to introduce vNext as #ifdef on main branch? What do you think about dual ABI like GCC did it for C++11?
Sounds messy, considering in some cases entire classes are rewritten
@sylveon
#ifdef VNEXT
#include <future>
#else
#include <old_but_gold>
#endif
@OwnageIsMagic While that would be technically possible, our concern is that an #ifdef VNEXT
mode would lead to mix-and-match ODR violations. There's a secondary concern around maintainability, since it would essentially lead to having two implementations in one codebase, both of which need to be tested. We already have ABI-affecting modes, so we're not eager to add more.
@StephanTLavavej ODR can be handled by inline namespaces and it intended only for static linking due to it's unstable nature, so no runtime mismatch. You already have 2 implementations which are diverge more and more with each commit on main
branch. I'm will be ok if it will be provided “as is”, without warranty of any kind
, as all other my dependencies.
Unfortunately, inline namespaces only solve one level of the ODR problem - they don't help if you have a struct Cat { string name; };
where the Cat
won't be placed into an inline namespace. There are also issues (thanks @barcharcraz for mentioning them to me) with passing Standard Library types in third-party library interfaces; if we released an #ifdef VNEXT
then it could cause lots of confusion if third-party libraries started using it. Not saying that it would be technically impossible, just that the complexity concerns have led us to avoid that approach so far.
Hey @StephanTLavavej ,
Sorry to bother you again on this issue, but as time passes, it would be nice to get a bit of information how the things are now and where are they going. :)
So VS 17.2 LTSC was released with completed C++20 support. Also this is the second LTSC release of the 17.x series. Lots of work was done, lots of bugs were fixed, lots of new stuff was added and of that is great. My thanks to you, your team, compiler devs and everyone else dedicated to C++ ecosystem in MS!
Since we are here, I would like to know(I guess not only I), how the things are going with the vNext project(both STD library and compiler). Do you now have a bit more light, when this project can actually be started, do we really need to wait 2-3 years for VS 18.x? Can you make a new branch from 'main', where ABI stable work can continue and start using 'main' for breaking changes? I guess all the work that would not break the ABI can be backported to the 'abi-stable' branch and so on...
So anyway, these are just a few thoughts, but I would like to hear from you. :)
I'm glad you like 17.2!
We haven't done any work on vNext, and currently there is no ETA for when work will start.
Can you make a new branch from 'main', where ABI stable work can continue and start using 'main' for breaking changes? I guess all the work that would not break the ABI can be backported to the 'abi-stable' branch and so on...
We tried maintaining two branches simultaneously and it was unworkable - that's a big reason why our initial vNext changes are stranded in a long-unused TFS branch.
I strongly believe that in order for vNext to work, we need a clean switchover, where new work stops flowing into the ABI-stable branch and starts flowing into the vNext branch. Perhaps the completion of C++23 will be a good time, but that's far from certain yet.
The problem as I understand it is that the ABI-break needs not just STL and runtime library work but also (in order to deliver the most benefit) compiler work and with so many other things on the table (newer C++ standards, compile time improvements, workflow improvements etc) that were deemed more important, the compiler team hasn't have the time to work on the ABI-break work (which is why the plans to make VS2022 ABI-breaking were dropped).
Won't you need at least a couple of month or longer to perform all the ABI-breaking work? Not to mention that you probably want 1-2 iterations of real-world feedback, before freezing the new ABI again. Is that really possible without having (at least for some time) two branches in parallel?
I've had to work on a bigger v1 - v2 transition only once so far and of course with no where near as many users as the STL, but what did work for us was
a) Daily merge any change from v1 to v2
b) Keep v2 in a "merge/diff friendly" state for the period of transition. E.g. large removals where achived by a simple #if 0
rather than physically removing the LOCs and we also tried to avoid shuffling code around.
Of course, maintaining two branches is strictly more work than one, but if the alternative would be much more time pressure and less chance for feedback, I'd at least consider it, as long as there is a clear goal/time frame and the switch over doesn't get pushed further indefinetly.
Won't you need at least a couple of month or longer to perform all the ABI-breaking work?
I believe it would be at least a year of work.
Is that really possible without having (at least for some time) two branches in parallel?
Sure.
Note that we inherently have release branches - e.g. VS 2019 16.11.x has an internal release branch, alongside the branch for VS 2022 17.x. The question is where arbitrary development work (new features and fixes) flows into. For the STL, that is always into the latest branch, with very rare exceptions (backporting C++20 DRs to 16.11.x being the largest).
A clean switchover would simply mean that we'd declare that no new features and only critical fixes would flow into the ABI-stable branch, with all new work going into the vNext branch.
For the STL, maintaining two branches simultaneously (i.e. adding features to both) would be a massive amount of additional work(), and we need to make major* overhauls that are extremely unfriendly to diffs (and I can speak from experience, since we tried parallel branches already, in the abandoned effort).
* The only way we can keep up with Standardization is by prioritizing development throughput and avoiding wasted work - this explains a lot of our processes that might otherwise seem strange (e.g. extreme effort spent on code review to minimize effort spent on bugfixing).
@StephanTLavavej Thanks for the detailed explanations. I see the big picture now.
Perhaps the completion of C++23 will be a good time, but that's far from certain yet.
And this is where I think we are stuck in infinite recursion. Always trying to catch up with latest standards and no time to pick up the tech debts. This can go forever, if a plan is not made soon...
Perhaps take a look how other vendors keep ABI compat and still fixing stuff. For example GCC STD library is really not compatible both ways. For example, if I build something on Ubuntu 16.04, it will work on Ubuntu 20.04, but not vise versa(given that only dependency is the g++ lib). The runtime linker complains that it cannot find functions with names, which are tagged to the specific newer gcc runtime. If not enough people are available for the STD library and compiler FE, perhaps convince bosses to hire more? These are just a few thoughts, but I think we have to start thinking of something.
A clean switchover would simply mean that we'd declare that no new features and only critical fixes would flow into the ABI-stable branch, with all new work going into the vNext branch.
Do I understand correctly that you would then not ship any new features or improvements to an ABI-stable version for at least a year or so? Not that I would complain.
@TheStormN
And this is where I think we are stuck in infinite recursion. Always trying to catch up with latest standards and no time to pick up the tech debts. This can go forever, if a plan is not made soon...
Two notes:
<charconv>
), to C++20 (completed in 2021 - that is, before the DRs landed and we completed it again).Keeping up with Standardization is not the primary reason that vNext has been delayed. It's mostly a matter of:
@MikeGitb
Do I understand correctly that you would then not ship any new features or improvements to an ABI-stable version for at least a year or so? Not that I would complain.
Yes. To clarify: My preference for a clean switchover would be to declare that the v19-compatible libraries are permanently finished, and that no new features would flow into them (only critical bugfixes). If the switchover hypothetically happened between C++23 and C++26, the v19 toolset would remain available for "side by side" use (supporting C++23), but anyone wanting new features from the latest Standards (C++26 and beyond) would need to switch to the vNext toolset. After "completing" vNext, it would then become the new ABI-stable toolset, with features being added to it.
I would have no particular objection to the compiler deciding to add features to the v19 toolset (as it experiences ABI in a very different way from the libraries), but attempting to backport library changes continuously would be expensive and error-prone. As I mentioned, it took @CaseyCarter's exceptional skill and quite a lot of time to backport C++20 DRs to 16.11.x, and that was with relatively little source divergence (just a couple of 17.x point releases, and a few relevant compiler changes). The optimal thing to do would be to focus all of our effort into getting vNext right (since it will be our one chance to fix this stuff for probably a decade until vNextNext), and then resume keeping up with the Standardization treadmill.
(General reminder: nobody wants vNext more than I do, and I am continually pushing to make it happen. Hopefully we'll have good news to share someday.)
To clarify: What I had in mind was not to backport anything to old stable. Rather to implement new features in old stable and merge them forward to vNext and only do abi breaking work on vNext. Also, old stable would stay the only official supported version and vNext would be hidden behind a /experimenral:abi2 flag or some such until it is "done" (and maybe the next version of VS is released).
But of course: If you can take the proper time and exclusively work on vNext that is even better.
That aaide, I know I'm repeating myself, but I want to point out again that I (and I guess many others) very much appreciate the fact that you people share your thoughts on these matters here in the open, even if they are speculative and things turn out differently.
@StephanTLavavej My apologies, I guess my last post sounded really ungrateful, underestimating all the work you've all done. I've been with the MS compiler since VS 6.0(skipping VS 2002) and saw all the good and the bad. Since project rejuvenation stared long time ago I really saw how with each year not only the products, but the company transformed from disregarding(and even hating) open-source to loving it.
The design work behind the current stable ABI was vary brave(breaking the runtime to many satellite DLLs - api-ms-win-*.dll) in order to add new functionality and keep compatible ABI, although this ended up not really great and the idea behind it was to satisfy both regular users and corporate customers and I do appreciate all of that. I also saw how quickly you've reacted to my bug report about std::function
capture in lambda and fix was implemented just a few days after the reporting. So by NO means I would want to suggest that you(all) are not addressing tech depts or not hearing the community.
Now back on the main topic a bit. :) As far as I understand, keeping ABI between major releases is to allow corporate customers to make more timely migrations to the new versions and that you have tremendous positive feedback about that. However I would also like to share my years of experience in large corporations regarding upgrades. All of the ones that I've been working for are still migrating slow. For example in my current company(40k+ people) are using VS 2019 v16.9.x, because it is still supported and they are now looking forward to the next Long-Term supported version in order to avoid updating as much as they can and while the upgrade will be easier, because of the ABI compatibility(they are not going to recompile some of the libs), this does not mean that if there were an ABI break, they would reconsider. That's because of ISO policies which require only supported software to be used with or without ABI compatibility. So while in case of ABI break, the work would've been a bit more, it was not going to be avoided.
I guess my point is that freezing an ABI for 10+ years in general would lead to positive testimonials(because they've saved some time recompiling, but the time for testing the product is still the same), it is not worth it. :) I know you don't have much saying regarding the matter and if it's up to you, you will break the ABI on every major release, but perhaps you can push for the company to make a bit different surveys regarding corporations upgrading to newer version(for example breakdown of the time spent on compilation and testing). My guts tell me that you may get surprised.
@MikeGitb
To clarify: What I had in mind was not to backport anything to old stable. Rather to implement new features in old stable and merge them forward to vNext and only do abi breaking work on vNext.
I see. That's effectively the same amount of work, though - we'd still have to ensure that features can work in both "universes". Some would be easy (those with few dependencies on existing code), some would be hard, but the amount of extra work would be basically independent of where the feature was originally developed.
Essentially the reason I'm so allergic to the idea of adding features to two diverging branches simultaneously is that it takes all of our effort and skill just to keep a single branch up to date with the Standard and always shipping at production quality. We have what I believe is the largest single team of Standard Library maintainers at a single company, working alongside an awesome group of contributors here, but our capacity is still finite and we have to be careful what we spend time on.
I (and I guess many others) very much appreciate the fact that you people share your thoughts on these matters here in the open, even if they are speculative and things turn out differently.
:heart_eyes_cat:
@TheStormN
Since project rejuvenation stared long time ago I really saw how with each year not only the products, but the company transformed from disregarding(and even hating) open-source to loving it.
:heart_eyes_cat: :smile_cat:
I also saw how quickly you've reacted to my bug report about std::function capture in lambda and fix was implemented just a few days after the reporting.
Yeah, that one was real interesting - JonCaves explained to me that it had always been a compiler bug, where the lambda's "secret" constructors (that the compiler uses to initially construct the object, but which should never be visible afterwards) were still visible to SFINAE, and the new changes in std::function
constraints revealed this issue which had lurked for 10+ years!
For example in my current company(40k+ people) are using VS 2019 v16.9.x, because it is still supported and they are now looking forward to the next Long-Term supported version in order to avoid updating as much as they can and while the upgrade will be easier, because of the ABI compatibility(they are not going to recompile some of the libs), this does not mean that if there were an ABI break, they would reconsider.
This is very useful information, thanks for explaining. It's things like this that might help convince our bosses and boss-like entities!
perhaps you can push for the company to make a bit different surveys regarding corporations upgrading to newer version(for example breakdown of the time spent on compilation and testing). My guts tell me that you may get surprised.
Yeah, that's a good idea - we do such research from time to time, we could ask questions along those lines.
Last 10 years we had a new major Visual Studio update roughly each 2 years. VS2022 is soon 2 years. Any news or rumors on the next major update? Is Dev18 announcement right after the corner? =)
@vrubleg We try to be very open about our plans for the STL in this repo, but there are a few things we can't ever comment on - unannounced releases and release dates are major examples.
There are a number of issues out there (both listed here on the STL repository and also things like https://developercommunity.visualstudio.com/content/problem/193041/the-visual-c-2017-condition-variable-implementatio.html listed on the Visual C++ bug tracker) that are listed as being fixable but only in whatever the next release is that breaks ABI. There are likely people who want those fixes and would be quite happy using their own STL build that broke the ABI if it meant getting those fixes sooner.
I suggest you consider releasing the ABI breaking changes in a way that those who know what they are doing and want something that breaks ABI can get them. (but where anyone who doesn't explicitly download the ABI-breaking-changes branch wont have to worry) Or if its not possible to publish this stuff (either because there are also ABI changes in vcruntime, vcstartup, concrt or other non-open bits or because vcruntime, vcstartup, concrt or other non-open bits need recompiling with the new ABI-breaking code to pick up the new ABI) then at least consider publishing a comprenensive accurate list of the things that have/can be fixed but only by breaking ABI (so people know where the STL has bugs (or differences with the C++ standards and why they can't be fixed at this time)