Rewinding this repo and encouraging community

att / ast

AST - AT&T Software Technology

Eclipse Public License 1.0

557 stars 152 forks source link

Rewinding this repo and encouraging community #1466

Closed gordonwoodhull closed 4 years ago

gordonwoodhull commented 4 years ago

I work with @lkoutsofios at AT&T Labs Research and volunteered to see if I could help restore the repo to a state the community is comfortable with. This was last discussed in https://github.com/att/ast/issues/1464#issuecomment-581159703 and onward.

I am not an expert in ksh or the technical issues, and I don't know the entire history, but my understanding is that

ksh93u+ is the final official release from AT&T. This can be found in the branch 2012-08-01-master
A later version, ksh93v-, was contributed by the original authors after they left AT&T, but this version is not considered stable. This can be found in the tag ksh93v
@krader1961 has been maintaining the master branch for some years. He has tried to streamline the codebase (and only support ksh, not the rest of ast), but this has caused compatibility issues.

The original authors of ksh have all left AT&T. We still have plenty of users inside the company, but no one who can maintain and manage the project.

My proposal is:

[x] Move the current master to a branch "ksh2020" (or whatever is appropriate).
[x] Clearly mark README.md and CHANGELOG.md on that branch as unmaintained, point to #1464 and this issue for further info
[x] Reset the master branch to 2012-08-01-master
[x] Remove external collaborators from the repo. Since no one in AT&T is maintaining this, it doesn't make sense to have this fork be the active one. Git and Github allow anyone to take the code in whatever direction they want, in their own space.
[x] Clean up README.md to reflect the current state: "This repo contains the AT&T ksh93u+ and experimental ksh93v- versions of AST. There may be later forks of AST and ksh out there."
[x] Change CONTRIBUTING.md and issue template to be clear we are not actively maintaining this fork and do not promise to review anything.
[x] Shorten the repo description to "AST - AT&T Software Technology" (#1441)

After this is complete, we at AT&T may contribute patches to ksh93u+ in the master branch. We probably won't review or accept contributions unless they are trivial (build issues, etc.) Eventually, we will archive this repo.

Currently I don't think anyone in the company plans to create a repo just for ksh. However, I think this is fairly straightforward and can be taken up by the community.

If anyone wants to create a community fork, I'd suggest creating a new GitHub Organization. This is free for open source projects. The bigger commitment is time, making contribution and maintenance guidelines clear, possibly creating a code of conduct, etc. Of course if anyone wants to fork this repo to an existing organization, they are welcome to do that too.

jghub commented 4 years ago

@gorodnwoodhull: my most recent comments are here: https://github.com/att/ast/issues/1464#issuecomment-581893584

sorry for overlooking this (better) place. will take note from now on...

gordonwoodhull commented 4 years ago

I went ahead with removing the external collaborators and fixing the description. I'll wait to see if there is further discussion about the other points before continuing with those.

lkoutsofios commented 4 years ago

thanks @gordonwoodhull for helping out.

my original motivation for going along with @krader1961's proposal was to keep at least KSH alive since no one else was volunteering to work on the AST project as a whole. but I think we were all clear that backwards compatibility was mandatory. KSH's strongest advantage over other SH/KSH clones was its programming interface. if the new version breaks old scripts, there's no point in using it. it seems from the recent issues on this repo that this isn't working out. so @gordonwoodhull's changes seem like the way to go. we'll see what happens with KSH. I do have some fixes for the u+ version that should get it to build and work on all recent distros of redhat, ubuntu, and opensuse, that I can push out. unfortunately, I don't have the time to be more involved.

DavidMorano commented 4 years ago

Just some friendly observational comments:

I have tried to follow both the recent discussion about rewinding the repository, et cetera, and the progress on this project since @siteshwar and then @krader1961 started making major contributions back in 2017. For the record, my own experience w/ KSH dates from its very (albeit public) inception, back in 1983 (I was at Bell Labs at the time and following). I have used KSH almost entirely exclusively and continuously since that time to the present (including very recently on ksh2020). Some items:

this code is very complicated and somewhat messy by the best modern coding practices (that may be something of a gross understatement); it is NOT suited for the casual programmer -- or team of programmers -- to dive into; despite people possibly feeling otherwise, I think that @krader1961 is-was fairly competent as a programmer and developer; where @krader1961 seemed to possibly lack something was his understanding of the various historical customer uses of KSH over the last 37 years -- being much more intensive and far-reaching than he seemed to realize; I will not dwell on this now, but KSH was Perl before Perl was, and it was Python before Python was; it had substantial enterprise applications written in it before the end of the 1980s (and the first book only came out in 1988)
the code creates a program that is substantially defined by three standards: 1) POSIX, 2) the KSH book (by David Korn), and 3) the existing behavior of the ksh93u+ branch; this is not something to be trifled with lightly; extreme care needs to be exercised to not break any existing documented or expected behavior (this may be an area were @krader1961 might have appeared to not be as sensitive as people would have expected)
despite some possible (arguably relatively minor) personality issues, @krader1961 has put an exceptionally enormous amount of work into fixing problems that even existed in ksh93u+ that were not always readily observed or complained about; to replicate @krader1961's work starting from scratch at the ksh93u+ branch will be extremely non-trivial (please, @siteshwar give your feeling on this if you would)
I am aware that even ksh93u+ had severe bugs and problems that needed to be eventually fixed; many of these did not manifest readily for most users, but they needed to be addressed eventually none-the-less; these are things such as strays writes, buffer overflows, and the like; there were a variety of minor (and not so minor) problems also that had been in there for about 20+ years (also not readily complained about by most regular uses); @krader1961 had made a very substantial start at getting some of the more serious of these (the memory related stuff) fixed (with the help of, among other things, ASAN and Valgrind); starting again from ksh93u+ means a huge amount of dog-work in fixing up again what @krader1961 has already done
not all distros are equal! some distros are more important and have more say in the world than others! sadly for some, the Red-Hat distro is more important now-a-days than pretty much all other distros combined (yes my opinion!); and I am not one who even likes Linux derived distros in the first place! ; Linux itself is very much inferior in many ways to older non-Linux distros (yes that is also my opinion); so the short of it is, that I think that @siteshwar should have more say in how KSH proceeds into the future than all the rest of us (distro maintainers or not) combined! I am sorry, but that is what I feel should happen; @siteshwar is not a dumb guy; he knows 'the score"; he knows full well that maintaining KSH is like managing fire in your hands; he works for Red Hat!!! Red Hat understands managing real (hard core, porn) enterprise (and other) customers and their requirements and demands; I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke
my fear is that if the ksh2020 branch is essentially abandoned in favor of starting from ksh93u+ again, a huge amount of time and work will have been forfeited; I want to hear from @siteshwar more of this point; this kind of decision should not be taken lightly
even separating out KSH from the rest of the AST code was not a totally trivial endeavor; @siteshwar and @krader1961 did this w/ ksh2020 at no little (no little) effort in itself; although I had some trepidation at the time, I feel that this was a useful and important thing to do to modernize the KSH code base in particular; this is just one example of something that would have to be done again if starting from scratch
my own preference going forward would be something along the lines of the following: 1) bring the ksh93u+ branch to a new state (calling it something else, ksh93uu+ whatever) so that it can be easily compiled by the community (easily, maybe using make, autotools, CMake, whatever) so that distros have a fairly reliable and readily accessible base to port from (in the near term), but then 2) continue w/ the ksh2020 branch and make it more conforming to historical and expected behavior; ksh2020 would be the base for future -- possibly feature enhanced -- versions going forward
and finally, for those who think that just using ksh93u+ is some sort of magical cookie to the golden land, it is very likely that making any fixes to ksh93+ (which will likely be required) will break some conforming behavior in unexpected ways; this is likely part of what happened with ksh2020 -- the act of fixing some previous bad stuff, exposed some bad stuff that was still in there already from the past, but not previously readily observable by most customers; this happens all of the time with old code, but to get through it one has to often just confront bug after bug and plow ahead through the hard times; for myself, I feel that ksh2020 has started down that path (as hard as it may be); starting fresh from ksh93u+ may never get past the hump of fixing some of its worst bugs lurking within it
KSH is a complicated and still -- in many quarters -- a mission critical program; in my opinion, it needs the attention and contributions from the entire world to focus on a single main code base branch in order for it to flourish into the future, rather than just staying "useable" at the ksh93u+ level; ksh93u+ had (has) many bugs that need to be eventually fixed if there is any hope for real future feature enhancements to likely work; the code base needs to be modernized also (not a trifle in itself)
just my two cents on this whole thing!

jelmd commented 4 years ago

@gordonwoodhull, I think you got it right and your proposal is good.

However, some nits:

* [ ]  Move the current master to a branch "ksh2020" (or whatever is appropriate).

Yes. But to avoid confusion [1] [2], that people use it as an official or stable version (actually what happened with version v-), README.md as well as CHANGELOG.md in this branch should make it very clear, that this is an experimental branch (e.g. by adding an appropriate level 1 header at the top) and the mentioned articles refer to an experimental release (or that there is no ksh2020.*, etc.).

* [ ]  Rewind the master branch to 2012-08-01-master

Adding @lkoutsofios patches is a good idea - thanks @lkoutsofios for spending your time! Direct merge, separate *.patch files or PR, whatever you like is probably ok.

* [ ]  Clean up README.md to reflect the current state, and point to any active forks from there. If desired, transfer relevant issues to child forks.

If no-one at AT&T has time to maintain it, or prefers one fork over another, it should be kept neutral and just mention, that there are forks which are not AT&T driven. I think, if there is a valuable, active fork, people interested in ksh93 will find it by themselves or by getting hints from others...

gordonwoodhull commented 4 years ago

@DavidMorano thanks for your perspective. I don't know the history but I agree in principle with what you're saying.

I will second that I hope ksh2020 starts an organization, perhaps led by @siteshwar and @krader1961, starting from what is currently on master.

gordonwoodhull commented 4 years ago

@jelmd, sure, sounds good - I've edited the tasks above.

jghub commented 4 years ago

@DavidMorano, I appreciate your perspective, especially concerning the "KSH was Perl before Perl etc" aspect and its implications.

regarding what has been achieved (or not...) in ksh2020 I respect your assessment, although it seems too "understanding" in my view. I would readily agree that I cannot really judge the technical difficulties involved when rewriting/rephrasing the C-code . I am in the position of a "typical" KSH user who uses the KornShell language for scripting w/o knowing the C innards of the ksh program .

let's say I cannot build a car but I can drive it and if I drive ksh2020 around the block I notice strange noises from the motor department and assorted malfunctions. it's slower, too... before the overhaul the car was just fine, only had some peculiarities the driver knew (mostly) to avoid. so while the overhaul obviously involved technical skills (at the C-level) beyond my own I can judge that the job has not produced something better but something worse.

I maintain that is the current state of ksh2020. choose any metric you like (except better formatted C-code and "tidiness").

as you explained to me elsewhere (I forgot so far to again thank you for this!) https://github.com/att/ast/issues/1449#issuecomment-581252797, you especially value the malloc vs. vmalloc changes for reasons of your special interests although you also agree that vmalloc might be uncritical for the 99.9% crowd not writing their own (multi-threaded) builtins. of course, if it turns out, that indeed replacing vmalloc is clearly better (but for whom and according to what measures would have to be decided) one might consider to do that to 93u+ at some point down the road. I am not sure that there is clear evidence that this is worth it -- but not for me to decide....

and it sure would have been the responsibility of @krader1961 and @siteshwar to listen to people like you and many others (I was a latecomer in the queue of people raising concerns...) from the start and adjust their actions how to deal with the code, the features etc. I feel they (at least krader) had the wrong agenda. and the decision to start from the 93v- beta is not comprehensible to me, given the circumstances (known buggy beta state, no advice from dgk etc.).

I do not know of any "problem/bug driven" TODO list with which ksh2020 was started. there was the "we must first overhaul the code to adhere to modern standards" and the fuzzy "keep ksh relevant" approach and that without any adequate understanding of either the code or KornShell. the issue history is there for everybody to go through (it is not that extremely long): a clear story evolves of misconceptions, false claims and -- and that was the central problem throughout -- an incomprehensible rigidity and refusal to adjust course and actions in the small or in the big issues on the side of krader.

so I believe your rather benevolent description of krader's "possible (arguably relative minor) personality issues" is objectively not accurate (and this is not irrelevant since it crucially has determined what has happened in ksh2020 -- it could have been a different project -- and is also crucial for deciding how to proceed from here).

for me this implies that ksh2020 should (no: must!), indeed pursue its goals independently of ksh93. I also insist that terminology-wise there is need for a strict and hopefully agreed-upon "branding". it was proposed to call ksh2020 "krsh" for obvious reasons, but I really believe the "ksh2020" name is more sensible and "catchy" and accurate enough, so why not call it that for the next 25 years just like ksh93 still is called that...

altogether I am still not quite sure, what exactly the goals of ksh2020 are: so far it has eliminated most or all of the new features of 93v- since they could not debug those (being, so far, not yet or not fully functional in 93v-) while incompatibilities and regressions and new bugs crept in and performance went southward. maybe, that all can be fixed and at some point ksh2020 is better than a (patched) 93u+ but I really do not see that right now.

to reiterate: in my view, ksh93 and ksh2020 development should not proceed in a common repository. this can not work out. what could work out in my view is a friendly coexistence (in the long run, at least, when potential hard feelings have waned) and, hopefully, bug fix exchange (the latter for the time being could very well be unidirectional from ksh2020 to ksh93u+ for the rather few cases (AFAICS) where ksh2020 has fixed things that apply to 93u+ and not only to 93v-).

I am looking forward to implementation of the strategy outlined by @gordonwoodhull: even if ksh2020 could proof at some point to be the "future", I believe making ksh93u+ easily available again for everybody ASAP is much more important right now. this requires separate projects as the history of the ksh2020 project proofs beyond reasonable doubt in my view.

gordonwoodhull commented 4 years ago

I clarified that we won't immediately archive the repo, and we'll contribute what patches we have, but we probably won't accept contributions unless they are trivial.

I'll proceed with the plan in the next week.

DavidMorano commented 4 years ago

@jelmd I think that we agree (and most everyone agrees) with the growing consensus around the following:

that a new workable version of KSH be made out of the ksh93u+ branch as its base, possibly replacing NMAKE w/ something else; this would serve the needs of distros that need a very reliable (as much as can be expected given everything) and behaviorally compliant KSH
further, I have no objection to VMALLOC staying in the ksh93u+ follow-on product; if I ever want to use this new KSH product for myself, I will just replace VMALLOC w/ a regular system one as I did in the past
neither do I take any issue w/ trying to bring the new (ksh93u+ derived) product up-to-date w/ any patches or fixes that can readily be put into it (the issue here is the time to do this at the level that ksh2020 has already achieved)
I do not want to waste ksh2020, and neither do @krader1961 and @siteshwar as far as I would guess; but it seems that the consensus is that they open a new unique repo for its development, separate from this ATT/AST repo
my guess is that @krader1961 and @siteshwar will likely continue to develop and orient ksh2020 as a possible next generally-released KSH also, and I encourage that work; but my preference would be that it try to adhere more to the standardized and expected behavior of prior KSHs -- circumscribed as I stated previously around 1) POSIX, 2) the KSH book (Korn), and 3) ksh93u+ behavior

For myself, I admit that I am optimistic that the benefits that @krader1961 has put into ksh2020 (much of it quite subtle, like cleaning up the various NV preprocessor flags mess, to name just one) more than offsets the regressions that people have complained about. But, yes, this is opinion on my part. But based on following many or almost all of @krader1961 commits. I have seen regressions in ksh2020 myself (like in the form of unexpected crashes). But in fairness, there were some unexpected crashes in many of the production KSHs over the last couple decades also. My hope is that, accidentally or not, @krader1961 might have actually fixed some of these (again, perhaps even by accident as it were).

Having a follow-on version of ksh93u+ available for distros (whatever its name is, I do not care about any names really) will take the pressure off of the ksh2020 development, so that they can concentrate on cleaning it up so that it appears to perform at least as reliably and conformally as the old ksh93u+ did (or its follow-on replacement).

I also have to admit that I hope that ksh2020 succeeds totally eventually. I have seen, and sometimes tried to understand, the code in ksh93u+ and it is often quite difficult (and I look at tons of bad code all of the time for my living). I tend to get spooked by and biased against code that has memory buffer overflows and the like, and I very much welcomed @krader1961 and other's initial attempts to clean some or much of it up -- as much as is exposed by bad behavior or by development tools (like ASAN or Valgrind, et cetra). I do not feel that the code base of ksh93u+ is really fit for the future. I would like to see the code come to a place where both fixes and feature enhancements become at some point much easier. But I think that the right path for this (following the project as I have) is with ksh2020, simply due to the amount of cleanup put into it (being perceived as useful or not to the end user). I could be wrong, but we will now see. Neither do I discourage future major fixes or enhancements to the new ksh93u+ branch. Let the race begin, as it were. The end code-builders or end-users can decide for themselves which one is the real future (or not).

jelmd commented 4 years ago

@DavidMorano, I totally understand your concerns. But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

FWIW, I'm basically repeating, what has been already said:

At the moment all distros ship a ksh93 package, have certain bugs already fixed with their own patches and the process to build the package, or apply a small patch from time to time is almost a no-brainer. Vendors/developers/users may even have a knowledge base, FAQ, ... wrt. workarounds for existing bugs/corner cases so that even patching is not really needed, as @jghub already pointed out. So on the vendor side there is no need to invest a lot of/any time and they can continue to ship it until a new star appears on the sky, which makes it redundant.

Redundant means for me full compatibility to ksh93u+. E.g. the getopts self documenting as well as the time conversion and formatter feature are essential and unique (skim over acme-ksh scripts to get some more details if you like). Last but not least I do not like to waste a lot of time to find out, which of several dozen man pages actually contains a description of the feature I wanna use (like zsh), and than skim over several pages again to finally find it. ${builtin} --man is such a useful thing, that I do not really care about the size of the ksh93 binary.

Anyway, new stuff like native json support, possibly ksh thread safety, ... would be nice to have, yes. But I think, even changing the build system only would requires a lot of "review" time at least on distro maintainers' side to make sure, that they still ship a stable/usable version. Debian has shown, that this is probably too challenging, decided to just trust the most active members of the repo and bundled something that at least I do not want to see on any of the 100+ zones/containers I need to manage (or others, which use scripts I wrote in the last 20+ years). Therefore, encouraging people to fork and give it a distinguishable name seems to be the best option. If it is good, distros will pick it up pretty fast and do not need to care about compatibility wrt. ksh93* ...

kdudka commented 4 years ago

But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

siteshwar commented 4 years ago

I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

I give up and move on with my life. I made all my efforts with the goodwill to help the community and it turned out that I achieved quite the opposite. I hope that someone else would pick up this project and would make progress in the direction that's more acceptable to the community.

DavidMorano commented 4 years ago

@siteshwar

I give up and move on with my life. ...

I am very sorry to hear your decision on this matter. I think that you were the best thing to happen to KSH in years! Having RedHat (as represented by yourself) in the middle of decisions regarding KSH was likely the best thing to happen to KSH since the old haughty days of AT&T (pretty much long gone now). I feel badly that the general community did not seem to show you more respect (deference). Thanks very much for the service that you did give to this project over these past years.

All the best with whatever you do going forward!

jelmd commented 4 years ago

...

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

@kdudka, if no-one creates an repo issue, it is very questionable, that it get fixed "in time" by the members of the related repo at all - so thanx for bringing it finally (after 4+ month) to our attention. And BTW, silently inserting a very questionable patch, which just swings the big hammer, is ok? Or can you help me? I could not find a related issue in this repo using the keyword security or CVE . Even the commit contains neither the word "security" nor "CVE". Is this really your understanding of open source development?

marcastel commented 4 years ago

I would like to express an "external" (or consumer) point of view here on KSH for today and tomorrow.

Though I have not contributed any code to this GitHub repository, I have (since my #42 comment) maintained my own private build of ksh and a have some insight on its intrinsics -- and for one, I like its build system which, optimised, is better than the gmake/cmakes of the world, or their more recent python or javascript alternatives)

But back to the consumer view of ksh. In the comments I have seen so far we speak about the "community"... and I understand that to be the "community of developers" as opposed to the "community of users".

As a user, I am obviously very concerned with backwards compatibility, performance, and much less so about the build system. But more importantly I am concerned about the POSIX status of AT&T ksh which has landed this shell on all UNIX distributions.

This is what is important to me, as a consumer. I want my ksh to remain in my AIX, macOS or Linux distro "as-is". I don't want it to be called pksh, krsh, ... I want it to remain POSIX certified and to be the time-proofed shell I use for my production environments and secured managed operations.

As probably many "consumers" I am a system administrator who uses Korn shell over other shells because of its portability, its performance, and of its superior programming capabilities -- e.g. object-oriented REST interfaces for managed services using KAML (https://github.com/ISLEcode/KAML)

And as one of these "consumers" I wish we could upgrade modern operating systems from POSIX "sh" to POSIX "ksh". Imagine such a replacement in the GNU build system... ksh in lieu of sh and m4... lean and clean. Obviously a little provocative, only to emphasise the value of ksh by its functionality, its programability and its heritage.

All that to say this: Korn shell should not be just another open source project abandoned to a community of developers, however skilled and passionate these may be. If it is to survive as a POSIX certified shell, it also needs representation (and defence) in USENIX and the like associations. It needs governance and a roadmap aligned to modern operating system roadmaps.

I want ksh to survive as a POSIX shell... even if we are probably a decade late. I fully support (and ready to invest time) in any initiative to make this happen. The consumer view is the bird's eye view of where we want to go. This is the first step. The developer's view, which is how we get this done, is the next step.

I appreciate the initiative to bring governance to this repository. I hope the extended scope proposed here shall be considered.

kdudka commented 4 years ago

@jelmd Embargoed security issues cannot really be reported to a public bug tracker. There needs to be some confidential channel where security issues can be reported and fixes prepared before they are disclosed publicly. When the mentioned security issue was discovered, the confidential channel was @siteshwar + @krader1961. They took care of it code-wise and process-wise. If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

jghub commented 4 years ago

@marcastel

As a user, I am obviously very concerned with backwards compatibility, performance, and much less so about the build system. But more importantly I am concerned about the POSIX status of AT&T ksh which has landed this shell on all UNIX distributions.

This is what is important to me, as a consumer. I want my ksh to remain in my AIX, macOS or Linux distro "as-is". I don't want it to be called pksh, krsh, ... I want it to remain POSIX certified and to be the time-proofed shell I use for my production environments and secured managed operations.

its seems you are more or less paraphrasing what people have said here: ksh2020 is not a ksh93u+ drop-in replacement, so keep (or make again) ksh93u+ available, clarify ksh93u+ being "the" ksh for now and then proceed carefully (bug fix patches etc).

As probably many "consumers" I am a system administrator who uses Korn shell over other shells because of its portability, its performance, and of its superior programming capabilities -- e.g. object-oriented REST interfaces for managed services using KAML (https://github.com/ISLEcode/KAML)

yes. all of which was harmed to varying extend in ksh2020 already. I was just as you at the "receiving" end of it (as a user) ...

All that to say this: Korn shell should not be just another open source project abandoned to a community of developers, however skilled and passionate these may be. If it is to survive as a POSIX certified shell, it also needs representation (and defence) in USENIX and the like associations. It needs governance and a roadmap aligned to modern operating system roadmaps.

if the ATT people were able/willing to do that and provide a project lead in the spirit of dgk, that would be great. but it seems that this is completely out of the question if I get their comments right. so the envisaged strategy as outlined by @gordonwoodhull seems to be the next best thing. and then it depends on 'the community' -- in absence of any other option. and it will hopefully then be able to pool sufficiently many reasonable guys in a single fork of the archived ATT repo to make it "the" ksh93 repo which distros could honour and use. something like that...

I want ksh to survive as a POSIX shell... even if we are probably a decade late. I fully support (and ready to invest time) in any initiative to make this happen. The consumer view is the bird's eye view of where we want to go. This is the first step. The developer's view, which is how we get this done, is the next step.

I appreciate the initiative to bring governance to this repository. I hope the extended scope proposed here shall be considered.

it seems everybody who expressed his opinion here (or earlier in the cause of the project) agrees with your "conservative" approach. I for one, do .

jelmd commented 4 years ago

@jelmd Embargoed security issues cannot really be reported to a public bug tracker. There needs to be some confidential channel where security issues can be reported and fixes prepared before they are disclosed publicly.

@kdudka, hmmm, not sure like other projects handle it, e.g. like apache httpd, where one is able to find, which CVEs have been already fixed by just browsing the commit logs, or reading the file which documents all important changes in a more human readable style - e.g. CHANGES.

When the mentioned security issue was discovered, the confidential channel was @siteshwar + @krader1961. They took care of it code-wise and process-wise.

Well, and this approach has proved to fail. In a healthy environment I would expect a github issue, which has at least a security tag, contains some kind of a thread analysis, perhaps some suggestions how to fix it and sooner or later one or more proposed fixes, which can be discussed and properly adjusted. Under such conditions, I guess the mentioned patch (which is, to be polite, far far away from being "brilliant") would have never made it into a main or stable branch. But it is still there and not even a "serious performance breakdown" issue (#1449) was able to bring it back to a review.

So it shows, that what happened in this repo, was basically a one-man-show: the self-appointed "god" writes a patch and the dazzled believer just gave its 'LGTM' or 1+, and often not even that - the patch got simply committed w/o giving the opportunity to review it before. So IMHO we can't be sure, what other Easter eggs got already implanted by krader and a new repo is required to start from scratch, i.e. ksh93u+.

If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel. On one hand it makes somehow sense, on the other hand it seems to be in contradiction to open source development. And last but not least I do not know, whether I would be part of such a channel. At least wrt. C I do not consider myself a C programmer (not even sure, when I wrote last time a single line of C code - probably some years ago). Wrt. ksh93 I would say, I'm 99% on its user side, and only 1% on its developer side ...

So, IMHO distro maintainers should make their own choice, which ones are trustable, open a corresponding issue on demand and wait for whatever happens.

jghub commented 4 years ago

@jelmd

Well, and this approach has proved to fail. In a healthy environment I would expect a github issue, which has at least a security tag, contains some kind of a thread analysis, perhaps some suggestions how to fix it and sooner or later one or more proposed fixes, which can be discussed and properly adjusted. Under such conditions, I guess the mentioned patch (which is, to be polite, far far away from being "brilliant") would have never made it into a main or stable branch. But it is still there and not even a "serious performance breakdown" issue (#1449) was able to bring it back to a review.

I looked at that patch, too, and would agree that is is a "symptomatic treatment" (disable the respective ksh capability of passing such data at that point altogether). but I can see that security holes might better first be patched "in secret" to exclude exploitation by so far unaware actors. but otherwise you are fully right: no information afterwards, no discussion and no attempt of a better solution, not documenting that ksh2020 now simply behaved otherwise by rejecting certain input -- all that is bad.

So it shows, that what happened in this repo, was basically a one-man-show: the self-appointed "god" writes a patch and the dazzled believer just gave its 'LGTM' or 1+, and often not even that - the patch got simply committed w/o giving the opportunity to review it before. So IMHO we can't be sure, what other Easter eggs got already implanted by krader and a new repo is required to start from scratch, i.e. ksh93u+.

I wrote "booby traps" rather than "easter eggs" elsewere but my concern is exactly yours: the erratic way "development" was handled (and yes, very quickly it became a one man show) including, e.g., massive monolithic checkins, the danger is eminent of a multitude of regressions and completely new problems buried in all the unreliably or not correctly documented tinkering.

the obvious ones that immediately hit you are at least reported (I recall "segfault after typing typeset -f as an example, affecting also the "stable" release AFAICR) and thus "known" (if one memorizes the whole issue list...) and partly they were fixed. it is the hidden layer below that that concerns me even more than the already documented misbehaviour of ksh2020.

If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel. On one hand it makes somehow sense, on the other hand it seems to be in contradiction to open source development. And last but not least I do not know, whether I would be part of such a channel. At least wrt. C I do not consider myself a C programmer (not even sure, when I wrote last time a single line of C code - probably some years ago). Wrt. ksh93 I would say, I'm 99% on its user side, and only 1% on its developer side ...

here I disagree regarding what would be desirable. as said, if the present repo/project would reset and restart under official curation/supervision of one of the ATT people (specifically @lkoutsofios would seem a viable "candidate" as far as I understand) and include further people (but this time not letting them loose on the code unsupervised like the last time...) with modest goals like "maintain availability of 93u+by making it build everywhere" (even if only with old build system for a starter) and than start to compile a list of good patches from solaris or elsewhere and a list of operational bugs that really would need fixing that would help in my view.

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

I believe I understand the initial incentive of @siteshwar (and the position of @kdudka) as employees of RedHat regarding the wish to once and for all overhaul/refactor ksh so that it is "ordinary" maintainable software afterwards. but if RedHat really wants that they would have to do it by assigning their own people to it dedicatedly (and restrict it to refactoring of ksh93u+ in order to not run into the mess of trying to refactor while needing/trying to bug-fix ksh93v- bugs accompanied by misguided intentional breaking changes (including feature elimination) all at the same time as has happened in ksh2020...). but the idea that krader would do the "dirty work" for RedHat for free, competently, definitely did not work out. quite to the contrary.

So, IMHO distro maintainers should make their own choice, which ones are trustable, open a corresponding issue on demand and wait for whatever happens.

despite everything that has been said regarding unmaintainable code, opaque build system and known limitations/bugs of ksh93u+ and the hope expressed by @DavidMorano that ksh2020 still can be the future (I really do not believe that...) I am sure that the "make ksh93u+ build for everybody w/o hassle" will suffice to satisfy the needs of 99.9% of all ksh users out there simply because ksh93u+ is "close enough". the ideas tried in 93v- might be revisited at a later time

aweeraman commented 4 years ago

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

A question that I have is what is the upstream of 93u+ moving forward, that will host these curated defect fixes and enhancements. @jghub are you volunteering to be the curator and maintainer of the pristine 93u+ official fork which distros can rely upon and not consider this project or the community as dead or abandoned?

gordonwoodhull commented 4 years ago

I've changed the branches as discussed, and added messages to the ksh2020 and master branches. See 5058182 and 0be8255 respectively.

If anyone wants to fork ksh2020, please start from ksh2020^ to remove the messages. And again, I'm happy to transfer issues to any relevant fork.

Rewinding master removed the issue template and CONTRIBUTING.md. I didn't see the point of adding them, but if anyone has requests, I'm game.

I think it's best to leave the repo open for comments so that people can discuss forks and future plans.

I will monitor the repo but I also encourage the rest of the community to answer bug reports with advice about what version to use, as @jghub did here: https://github.com/att/ast/issues/1467#issuecomment-583654157

jghub commented 4 years ago

@gordonwoodhull thanks a lot for your help so far!

it also is good to leave the repo open for the reasons you state. thanks for this decision, too. :)

jghub commented 4 years ago

@aweeraman

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

A question that I have is what is the upstream of 93u+ moving forward, that will host these curated defect fixes and enhancements. @jghub are you volunteering to be the curator and maintainer of the pristine 93u+ official fork which distros can rely upon and not consider this project or the community as dead or abandoned?

maybe you can open a new issue "future plans" or something to that extent (this one is closed now...)? other people should have opinions, too :).

regarding your specific question: I have not considered to volunteer for that role so far since I am sure not the best qualified person to do so. if everything else fails: maybe... but I definitely would prefer to do it as a group -- and having at least one package maintainer (the one from debian ;)) included would be a good start I feel...

kdudka commented 4 years ago

@kdudka, hmmm, not sure like other projects handle it, e.g. like apache httpd

@jelmd and you did not try hard to find it out, did you?

http://www.apache.org/security/

Well, and this approach has proved to fail.

So what prevents you from showing us how to handle it better?

As far as I know, the security issue is still not fixed in ksh93u+. It is your turn now.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel.

Earlier you wrote:

But what we need right now is to get back the trust from vendors ...

I am not sure how you are going to get back the trust of vendors when you have no reliable non-public channel where security issues can be reported.

jghub commented 4 years ago

@kdudka

So what prevents you from showing us how to handle it better?

As far as I know, the security issue is still not fixed in ksh93u+. It is your turn now.

basically, it is now everybody's turn (including you), right? you should, e.g., be able to directly speak to @siteshwar and ask him for a list of bug fixes in ksh2020 that also fix/patch bugs in ksh93u+. that would be a good start in my view. I recall that whence -a bug (however marginal) that would be on that list, e.g., it seems (I have not checked but the claim was the bug is "20 years" old, so...)

also, you only provided that single example of a "security fix" (of, as @jelmd pointed out correctly in my view, questionable quality -- but maybe that's the best one can do in the short run, who knows?) in ksh2020.

question 1: is it also in ksh93u+ for sure? could you check? if yes, that code section probably is identical in 93u+ and 93v- and you do thus have krader's patch you can readily apply (and it might be considered to apply it in a hopefully emerging accepted upstream repo of 93u+ if no better solution emerges).
question 2: are there any further examples of relevant security fixes applied to ksh2020 that are also relevant for ksh93u+, too, you care about? I am not aware of any.

Earlier you wrote:

But what we need right now is to get back the trust from vendors ...

I am not sure how you are going to get back the trust of vendors when you have no reliable non-public channel where security issues can be reported.

honestly: how urgent is that right now (and how many issues have been addressed in the last 2-3 years)? in the long run that can sure be sorted out.

in the short run it is more important that the ksh93 vs. ksh2020 confusion now seems to be over and distros start to take note (first debian and ubuntu, and now FreeBSD it seems) and restore availability of ksh93u+.

kdudka commented 4 years ago

basically, it is now everybody's turn (including you), right? you should, e.g., be able to directly speak to @siteshwar and ask him

Why do you ask me to ask @siteshwar ? You should have talked to him directly and ask him under which conditions he would help you with the maintenance of ksh93u+.

* question 1: _is_ it also in ksh93u+ for sure?

Yes, for sure.

if yes, that code section probably is identical in 93u+ and 93v-.

Nope, the code is not identical any more. It required non-trivial effort to backport the patch to the old releases of ksh93. I do not understand why you keep speculating what should or should not be done without looking at the code first.

* question 2: are there any further examples of relevant security fixes  applied to ksh2020 that are also relevant for ksh93u+, too, you care about? I am not aware of any.

I am not aware of other security issues with CVE already assigned. On the other hand, @siteshwar and @krader1961 have fixed a lot of buffer overflow issues in ksh2020 (inherited from ksh93u+). It has not yet been analyzed which of them are exploitable...

jghub commented 4 years ago

@kdudka

Why do you ask me to ask @siteshwar ? You should have talked to him directly and ask him under which conditions he would help you with the maintenance of ksh93u+.

because Brno seems the home base for both of you and both of you view the situation as RedHat employees? because you might be interested in ksh93?

regarding asking him directly: really, go back to all the discussions/issues.... you will see that he was repeatedly asked to reconsider the situation and to reset to 93u+ and to stop krader from damaging ksh further. he made it very very clear repeatedly that this was out of the question for him and he categorically (as I read his last statements) abandoned the whole project the moment ATT decided to retake control over the ATT/AST repo.

so it seems that siteshwar wanted to pursue a special agenda, relevant for RedHat, under the "umbrella" of the ATT/AST repo, nothing else? he or you are welcome to correct me if I get that wrong.

* question 1: _is_ it also in ksh93u+ for sure?
Yes, for sure.

if yes, that code section probably is identical in 93u+ and 93v-.

Nope, the code is not identical any more. It required non-trivial effort to backport the patch to the old releases of ksh93. I do not understand why you keep speculating what should or should not be done without looking at the code first.

I asked a question, OK?

* question 2: are there any further examples of relevant security fixes  applied to ksh2020 that are also relevant for ksh93u+, too, you care about? I am not aware of any.
I am not aware of other security issues with CVE already assigned. On the other hand, @siteshwar and @krader1961 have fixed a lot of buffer overflow issues in ksh2020 (inherited from ksh93u+). It has not yet been analyzed which of them are exploitable...

looking forward to that analysis. what is "a lot"? and what prevents backporting the fixes if they are sound?

I get it that you are not happy with the new situation. @DavidMorano is the other guy that has expressed his wish for a future of ksh2020 (only in less harsh tone than you do ;)) and his concerns with the ksh93u+/v- code base.

what I do not quite get is whether you (or he) really would have preferred (or still prefer: it is not impossible to do that for siteshwar and krader in a different repo!) to simply continue the project along the ksh2020 line with siteshwar not defining a clear strategy and krader tinkering any which way (and in absurd ways, partly!) with the code and behaviour of ksh?

it seems to me that both of you place a disproportionate emphasis on the value of buffer overflow issues and the like (even if never proven to cause a real problem so far) in the C code while quite severe regressions (or completely originial bugs) made it in a so-called stable release of ksh2020 which was in each and every respect pitifully inferior to ksh93u+. to re-use my previous car example: you seem to think that ksh2020 is internally now a more "solid" car since all screws are now tightened with a torque key according to GMP requirements where they previously were tightened manually and don't care whether the car now really does not work any more decently since crucial parts whose purpose was not understood were thrown out or modfied.

I am of course not saying that fixing buffer overflows is irrelevant but I am saying that ksh2020 is a mess from the user side. and that there is no chance in the short run to repair that since krader was at it for two years. I am not going to list here again the 10-20 crudest ideas and actions previously mentioned by @jelmd or me, e.g.

I am firmly convinced that what has happened here over 2-3 years is a demonstration of what happens if one tries to refactor really complicated/intricate software without any idea what the thing is really supposed to do and without the working hypothesis that the original author might have been smarter than myself so that I need to proceed slowly and carefully (a thought which never ever crossed krader's mind once). at the very least krader and siteshwar were not up to it (and that is not disparaging since it might be impossible: I have heard from different people that really looked at the ksh93 code and found it incomprehensible).

it is also very clear that the work done at ksh2020 was not creative but destructive: all the new things in ksh93v- were simple deleted since they did not yet work perfectly. other things/features were killed or contemplated for killing later. the code at large is now not better understandable than previously. but now it contains modifications by krader that are undesirable or prone to blow up later.

so you now have a ksh2020 code base with makes lint and coverity happy and a ksh2020 binary resulting from it that makes the users very unhappy. I prefer it the other way round, really, if I need to choose.

kdudka commented 4 years ago

so it seems that siteshwar wanted to pursue a special agenda, relevant for RedHat, under the "umbrella" of the ATT/AST repo, nothing else? he or you are welcome to correct me if I get that wrong.

I personally do not care whether the project is hosted at att/ast or somewhere else. For me it is important who works on the project and what the results are. If someone has the manpower to maintain ksh93u+ in a backward-compatible way, it is certainly welcome. If not, we need to figure out what to do next.

looking forward to that analysis. what is "a lot"?

I would suggest you to look at the code and commit messages before asking further questions.

and what prevents backporting the fixes if they are sound?

I do not know what prevents you from backporting the fixes. Perhaps you should talk less to save some time for the real (code-related) work?

so you now have a ksh2020 code base with makes lint and coverity happy and a ksh2020 binary resulting from it that makes the users very unhappy.

Please speak for yourself. ksh2020 reached stable Fedora releases in April 2019. There has been no breakage reported by Fedora users since then. This means that ksh2020 works for them well enough, assuming any users of ksh in Fedora still exist. The only report we had was the mentioned security issue, which affects ksh93u+ as well.

jghub commented 4 years ago

so you now have a ksh2020 code base with makes lint and coverity happy and a ksh2020 binary resulting from it that makes the users very unhappy.

Please speak for yourself. ksh2020 reached stable Fedora releases in April 2019.

of course I do speak for myself. as were the other ≈ 10-30 people (unscientific estimate from looking at the issue history, probably there were more over the course of 2-3 years) speaking for themselves that expressed exactly that same sentiment at different point in times: ksh2020 is heading in the wrong direction and has become a failure. nobody except krader and siteshwar (not even you, although you were partly involved in the whole thing as far as I know) ever claimed the opposite as far as I can tell, namely that ksh2020 were superior to ksh93u+ for the user.

There has been no breakage reported by Fedora users since then. This means that ksh2020 works for them well enough, assuming any users of ksh in Fedora still exist. The only report we had was the mentioned security issue, which affects ksh93u+ as well.

the breakage has been reported here in extenso. conclusion: fedora users avoided ksh2020 or are not seriously interested in ksh at all. I presume the former since fedora users are sure not that much differerent from other linux/unix users.

ksh2020 was a danger to the future of ksh93 since the intention was to displace the latter by sailing under the ATT/AST flag and trying to install the wrong impression in the distros that ksh2020 was the officially sanctioned new release of ksh93. several distros succumbed to that misconception/story and are now paddling back, fortunately, as far as I can tell (debian+ubuntu for sure, FreeBSD is on its way it seems). probably fedora should follow suite and keep ksh93u+ available.

I maintain: the deficits of ksh2020 are grave and indisputable from a user's or KornShell language programmer's perspective. they are testable and verifiable (breaking changes, features harmed, performance, crazy "future plans" (eliminate math functions? eliminate embedded documenation via --man?), etc). the alleged/advertised advantages of ksh2020 over ksh93u+ are unsubstantiated claims like "less segfaults" (the opposite is closer to the truth: I have seen segfaults and malfunctions with the ksh2020 "stable" release that were laughable obivious to trigger).

and whatever good things are burried in the ksh2020 mess, they might be salvaged and backported in due time. and if, e.g., siteshwar were able and willing to change the buildsystem for 93u+ like he did for 93v- that would do more good than the whole rest of the ksh2020 endeavour, probably.

jelmd commented 4 years ago

@kdudka, hmmm, not sure like other projects handle it, e.g. like apache httpd

@jelmd and you did not try hard to find it out, did you? http://www.apache.org/security/

@kdudka, no - I didn't try at all ;-). I've learned, that if I devote my leisure time to something, I should relax, have a beer (or two), and try to make one step after another ;-)

Anyway, if you wanna know, whether I could live with it: yes. If it works for the ASF, it should work for others as well.

Well, and this approach has proved to fail.

So what prevents you from showing us how to handle it better?

Nothing special, just time.
The security issue report[s].
That you said, it should not happen in the public? ;-)

As far as I know, the security issue is still not fixed in ksh93u+. It is your turw now.

Well I already wondered, why RH has chosen to publish the CVE for ksh2020, but no word about ksh93. Isn't this a little bit irresponsible towards your user base ?

Somehow I have the feeling, that you want to promote your 2020 baby and disparage 93u+. From a professional entity I would expect at least a "control your env" and/or a "if in doubt, use env -i ..." warning ... Anyway, I doubt, that RH has no-one, who is able to fix it - and well, 4 month? I guess, you didn't try hard enough, did you? ...

Earlier you wrote:

But what we need right now is to get back the trust from vendors ...

I am not sure how you are going to get back the trust of vendors when you have no reliable non-public channel where security issues can be reported.

OK, we need to discuss and find a way, how to do it. Unfortunately private branches or repos are not allowed in "Open source Teams" ...

jelmd commented 4 years ago

@gordonwoodhull is it possible to remove the 2020 and 2017 releases on https://github.com/att/ast/releases ? It is pretty confusing and most people would think, that someone just forgot to update the master README.md ...

jelmd commented 4 years ago

...

I am not aware of other security issues with CVE already assigned. On the other hand, @siteshwar and @krader1961 have fixed a lot of buffer overflow issues in ksh2020 (inherited from ksh93u+). It has not yet been analyzed which of them are exploitable...

@kdudka , buffer overflows wrt. to GNU? malloc. But do they exist, if AST malloc is in use? So switching the malloc system might have caused all such overflows (and other problems) is also an option we should not oversee ...

kdudka commented 4 years ago

Well I already wondered, why RH has chosen to publish the CVE for ksh2020, but no word about ksh93.

This is not true. We are releasing updates for Red Hat Enterprise Linux, where we have ksh93. Other Linux distributions release security updates, too:

https://bugs.gentoo.org/708618 (they have ksh2020 only) https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-14868.html https://bugzilla.suse.com/show_bug.cgi?id=1160796 https://security-tracker.debian.org/tracker/CVE-2019-14868

The only problem is that there is no responsible upstream for ksh93 where the backported patches could be shared and reviewed.

OK, we need to discuss and find a way, how to do it. Unfortunately private branches or repos are not allowed in "Open source Teams" ...

Sending patches in (possibly encrypted) e-mails to individuals from upstream is also fine but there is currently no such contact for ksh93, as far as I know.

@kdudka , buffer overflows wrt. to GNU? malloc. But do they exist, if AST malloc is in use?

Yes, we were debugging crashes and incorrect behavior caused by buffer overflows in ksh93 using AST malloc and we have some downstream patches for them. They were handled as regular bugs, not security issues, but it does not mean that security impact cannot be revealed later on.

jelmd commented 4 years ago

...

regarding your specific question: I have not considered to volunteer for that role so far since I am sure not the best qualified person to do so. if everything else fails: maybe... but I definitely would prefer to do it as a group -- and having at least one package maintainer (the one from debian ;)) included would be a good start I feel...

Just to let you know: same thing for me (and I guess for @marcastel and @jens-maus, too?). But I guess, together we can handle it. And for most known bugs, I think we'll find workarounds we can put in an FAQ, etc. which takes the pressure away from needing to fix it "yesterday".

So I suggest to create a github org now. ksh is already taken, not sure whether one has already setup ksh93, but if we cannot use this one, than I suggest ksh-ast (what I reserved just case ...). There we could setup a repo e.g. named organization, where we can discuss next steps. If someone feels we need to discuss non-public stuff, I suggest to register on gitlab.com send me a hint, so that I can invite you to the ksh or ksh93 or ksh93-ast group (the gitlab term for github's organization) or create one by your own and invite me. Gitlab stuff is new to me, however, it allows private projects/groups with more than 5 members, i.e. unlimited members (github and bitbucket do not) ...

jelmd commented 4 years ago

Well I already wondered, why RH has chosen to publish the CVE for ksh2020, but no word about ksh93.

...

The only problem is that there is no responsible upstream for ksh93 where the backported patches could be shared and reviewed.

@kdudka, I do not buy this. IMHO this is a very lame excuse. You have had enough time 'til krader finally screwed up this repo! On the other hand, is there something which prevents you from adding patches to your issue tracker and add an appropriate link to CVEs? Wouldn't it be easy for you to create a github/redhat/ organization, clone the ast repo, and add your patches to its 93u+ master branch? Or doing the same as Oracle does, i.e. putting the makefiles etc. as well as patches into their repo and let users build the stuff by themselves?

...

Sending patches in (possibly encrypted) e-mails to individuals from upstream is also fine but there is currently no such contact for ksh93, as far as I know.

I hope, gitlab is able to fill this gap ...

@kdudka , buffer overflows wrt. to GNU? malloc. But do they exist, if AST malloc is in use?

Yes, we were debugging crashes and incorrect behavior caused by buffer overflows in ksh93 using AST malloc and we have some downstream patches for them. They were handled as regular bugs, not security issues, but it does not mean that security impact cannot be revealed later on.

OK. Than we should probably look over it, when the new repo is in place.

jens-maus commented 4 years ago

regarding your specific question: I have not considered to volunteer for that role so far since I am sure not the best qualified person to do so. if everything else fails: maybe... but I definitely would prefer to do it as a group -- and having at least one package maintainer (the one from debian ;)) included would be a good start I feel...

Just to let you know: same thing for me (and I guess for @marcastel and @jens-maus, too?). But I guess, together we can handle it. And for most known bugs, I think we'll find workarounds we can put in an FAQ, etc. which takes the pressure away from needing to fix it "yesterday".

So I suggest to create a github org now. ksh is already taken, not sure whether one has already setup ksh93, but if we cannot use this one, than I suggest ksh-ast (what I reserved just case ...). There we could setup a repo e.g. named organization, where we can discuss next steps. If someone feels we need to discuss non-public stuff, I suggest to register on gitlab.com send me a hint, so that I can invite you to the ksh or ksh93 or ksh93-ast group (the gitlab term for github's organization) or create one by your own and invite me. Gitlab stuff is new to me, however, it allows private projects/groups with more than 5 members, i.e. unlimited members (github and bitbucket do not) ...

Sorry, but I would like to raise my concerns regarding moving ksh to gitlab and especially using private repositories or such. All development, including discussions on security relevant issues, should he help in public especially since we are not dealing here with a highly security sensitive OS component IMHO. Only if we and others can discuss on such security relevant things in public others can step up and raise further issues and concerns or bring up other ideas, etc.. As such, I would like to ask to keep ksh on a public github repository and simply use the issue tracker functionality to discuss on bugs, security issues, etc.

Regarding creating a dedicated github ksh organization, I would indeed suggest to go that way and create either a "ksh93" ot "ksh-community" organization and to create a "ksh93" repository in it which is essentially a direct fork of the att/ast repository but stripped down to ksh relevant sources only.

jghub commented 4 years ago

@jens-maus: I believe the proposal by @jelmd was to possibly move confidential discussion to gitlab since they offer that functionality. he was not talking about moving the repo to gitlab altogether AFAICS. but even that (move non-public discussion there) seems not really desirable, I agree. that could be sorted out differently. and how exactly to deal with the (not so frequent) security relevant reports might need further discussion.

jens-maus commented 4 years ago

@jghub Actually, I cannot really think of any information/discussions that require a confidental channel here. Wasn't this one piece of all this krader repo mess that he probably also used confidential channels with some third-parties?!? IMHO all future development and discussions should he fully performed in public and through official GitHub channels to prevent the same from happening again...

marcastel commented 4 years ago

A lot of exchanges in the last couple of days. Perhaps time for concrete actions. IMHO the way forward summarised below. A lot of points on either side expressed in this thread yet to be addressed. This is a starting point.

AT&T no longer supports its incarnation of the KornShell (i.e. the original KornShell) and -- whether officially or not, has placed its source code on GitHub. Nobody from AT&T has stepped up to maintain this repository, and it can be safely assumed here that no one will.

Consequently, if the KornShell is to survive, it needs a community. But more importantly it needs an organisation and a roadmap. This is, at present -- and in my opinion, more important than deciding on the build system, on the source code versioning environment, or even on the channel for classified confidential discussions.

The community probably needs to exist under the umbrella of robust open source organisations -- I have made today a request to the GNU Project and to the Apache foundation to see how this would be possible. Speak up if you have other alternatives.

Whether or not we can exist within the framework of any of the aforementioned organisations, we must, as a new and emerging community decide on our motus operandi -- that is, how will we organise ourselves to make this a successfull and lasting endeavour. When the time comes or on request, I will pour in more documentation and howtos, but for now I would like to hightlight some of the -- non-technical, questions which should be resolved prior to any new action.

Essentially this is the traditional who-what-when-how-...

We need project guidelines under which the project should operate, in particular we should establish roles and responsibilities.

We need a project management committee that coordinates the community including enacting policies, approving releases, adding new committers and members, etc. The committe should hold veterans who known where the KornShell comes from, and explorers who prepare KornShell for tomorrow.

We also need a chair that ensures board reports are submitted and that the project's roster is up to date.

Ideally -- and before he turns 80, we should have support or at least some vested interest in our initiative from David Korn himself or other major contributors such as Glen Fowler and al.

We need to clearly identify roles and responsibilities both at the technical level but also at the marketing level to promote KornShell to all major UNIX and derivative distributions.

We need a decision making model that allows for easy community decisions even in difficult situations. Apache's model with consensus approval, majority approval and lazy consensus could be a way forward for the diverging opinions which have been expressed in this repository's issues.
We must establish clear release and distribution policies and guidelines which account for vendor specificities -- e.g. including KornShell and its patches in the AIX and macOS builds.
As the KornShell user community is aging -- or at least I am, it is important to highlight the historical interest of the KornShell explaining why this shell is and should remain a POSIX compliant shell and a foundational component of any UNIX or assimilated distribution.
We must establish how knowledge management will be handled. The KornShell source code learning curve is abrupt. Letting any one individual take control of it means, as has been demonstrated here, that the core values of the project's purpose may be, if not corrupted, at least diverted. It is for instance my opinion that documentation is of higher priority today than effective bug resolution.

All of this should allow us to establish a business plan for our new KornShell community. If my comments make way here, I will take time to initiate an outline for this document. Further, I have reserved domain kornshell.plus -- where the plus extension is an eye wink to the AT&T stable release naming convention). I am happy to setup tools and environments on that domain to further support the technical and non technical activities of the yet to become KornShell community.

First steps first however... we need a named list of members. And a process on how members can publicly or anonymously participate.

kdudka commented 4 years ago

@jelmd No worries. The backported patches will be available for everybody to be able to (re)build ksh themselves. Red Hat Enterprise Linux is open source. I was talking about ksh upstream that should coordinate the effort across distributions, like upstreams of other projects do.

@jens-maus Embargoed issues cannot be reported to a public bug tracker by definition. If you insist on everything being reported publicly, it basically means that embargoed issues need to be handled by someone else.

gordonwoodhull commented 4 years ago

Glad to see the discussion about a future organization for ksh93!

@jelmd, I am wary about deleting the ksh2020 releases just yet. What I've done for now is add a brief note in each of the ksh2020 releases saying that ksh2020 is not supported or maintained & this repo only hosts ksh93, pointing back to this issue and #1464.

GitHub creates a "release" for every git tag; the other tags are variants of ksh93 so I left them alone.

Once @lkoutsofios has checked in his patches we can do another release which will show at the top.

jghub commented 4 years ago

@marcastel:

here are my spontaneous first thoughts to your proposals:

AT&T no longer supports its incarnation of the KornShell (i.e. the original KornShell) and -- whether officially or not, has placed its source code on GitHub. Nobody from AT&T has stepped up to maintain this repository, and it can be safely assumed here that no one will.

I guess you are right but it would be good if @gordonwoodhull and/or @lkoutsofios would consider to actively take part in the planned 'ksh93 organisation'. any chance?

Consequently, if the KornShell is to survive, it needs a community. But more importantly it needs an organisation and a roadmap. This is, at present -- and in my opinion, more important than deciding on the build system, on the source code versioning environment, or even on the channel for classified confidential discussions.

I do not completely agree regarding priorities. or rather, I'd like to see initially only a very modest short-term roadmap agreed upon and a general consensus that the priority in the short run should be to conserve ksh93u+ and ensure that it builds "everywhere" (the latter to be defined ...) so that its availability can easily be guaranteed.

The community probably needs to exist under the umbrella of robust open source organisations -- I have made today a request to the GNU Project and to the Apache foundation to see how this would be possible. Speak up if you have other alternatives.

I have no experience in these things. license-wise AST/ksh probably is incompatible with GNU, no? I would think this question and how to proceed in this regard could be safely postponed for the time being. we first need a new agreed-upon home for the repo and define the members of the planned organisation. and then to make distros aware etc. I trust nearly all users will get ksh via their respective distro (or not at all...).

Whether or not we can exist within the framework of any of the aforementioned organisations, we must, as a new and emerging community decide on our motus operandi -- that is, how will we organise ourselves to make this a successfull and lasting endeavour. When the time comes or on request, I will pour in more documentation and howtos, but for now I would like to hightlight some of the -- non-technical, questions which should be resolved prior to any new action.

Essentially this is the traditional who-what-when-how-...

We need project guidelines under which the project should operate, in particular we should establish roles and responsibilities. We need a project management committee that coordinates the community including enacting policies, approving releases, adding new committers and members, etc. The committe should hold veterans who known where the KornShell comes from, and explorers who prepare KornShell for tomorrow. We also need a chair that ensures board reports are submitted and that the project's roster is up to date.

to the above I would say: I estimate currently we might be able to motivate ≈3-5 people to spent time on the short-term goal of "ksh curation" and these should just decide on a "yes, we do it" and create the github organisation and the fork of this repo from which to proceed. afterwards these people and the community at large can discuss and decide to what extent formal and administrative meta-structures are appropriate and required. I for one look immediately for a hiding place if someone shouts "committee" and "board reports" ;). I understand that big projects might require all this but right now ksh: no.

Ideally -- and before he turns 80, we should have support or at least some vested interest in our initiative from David Korn himself or other major contributors such as Glen Fowler and al.

as @jelmd said elsewhere: it might be better to first demonstrate that we really do care for ksh by taking good initial actions. this might possibly than convince the original authors to lend support (and be it only by "approving" of what is happening).

We need to clearly identify roles and responsibilities both at the technical level but also at the marketing level to promote KornShell to all major UNIX and derivative distributions.

not now.

We need a decision making model that allows for easy community decisions even in difficult situations. Apache's model with consensus approval, majority approval and lazy consensus could be a way forward for the diverging opinions which have been expressed in this repository's issues.

no opinion except the naive one that for the time being, 3-5 people who start from a common shared assessment of the situation and agreed-upon short-term goals should be able to find unanimous agreement. if that is no longer the case, formal instruments might be setup.

We must establish clear release and distribution policies and guidelines which account for vendor specificities -- e.g. including KornShell and its patches in the AIX and macOS builds.

yes. I would find it desirable if package maintainers were involved and if the build process could be made to work for "everybody" out of the box.

As the KornShell user community is aging -- or at least I am, it is important to highlight the historical interest of the KornShell explaining why this shell is and should remain a POSIX compliant shell and a foundational component of any UNIX or assimilated distribution.

one could try that somewhere down the road in the not too distant future. of course it would be good if ksh93 would have a decent new website pooling historical sources scattered currently over the internet (and partly vanishing already) and up-to-date information like a good FAQ and documentation/help beyond the manpage. but this sounds like a lot of work and would have to wait for now in my view.

We must establish how knowledge management will be handled. The KornShell source code learning curve is abrupt. Letting any one individual take control of it means, as has been demonstrated here, that the core values of the project's purpose may be, if not corrupted, at least diverted. It is for instance my opinion that documentation is of higher priority today than effective bug resolution.

ksh2020 sure was not only a diversion... but anyway: yes that is obviously important. I currently don't see anyone involved who would want to mess with the code, luckily. so if all agree that the goal -- for now! -- should be to "reanimate" ksh93u+ and to collect, possibly review all available patches from RedHat, Solaris, ATT and to integrate them into a "ksh93u+ legacy" release that might be the way to go. addressing other bugs indeed should wait. documenting the known important ones would be good. ksh93u+ is overall very stable and providing workarounds for the known problems might suffice for most use cases.

All of this should allow us to establish a business plan for our new KornShell community. If my

personally, I would be averse to a major formalisation effort right now. it might suffice to draft a document declaring what we want to achieve in finite time and what long-term goals there might be pursued but postpone a detailed discussion of the latter, really.

comments make way here, I will take time to initiate an outline for this document. Further, I have reserved domain kornshell.plus -- where the plus extension is an eye wink to the AT&T stable release naming convention). I am happy to setup tools and environments on that domain to further support the technical and non technical activities of the yet to become KornShell community.

as said, a website would be good, if you do have this in mind. but it can/has to wait.

First steps first however... we need a named list of members. And a process on how members can publicly or anonymously participate.

I will try to actively contribute if it is of any help/interest to the rest of you (but I am not going to modify the ksh C-code with own bug fix attempts, very probably. I don't feel qualified for that).

gordonwoodhull commented 4 years ago

@jghub, I think you are correct that the license, EPLv1, is not GPL compatible. It looks like we need to add the license back to the master branch, too...

I don't think you can count on active involvement from anyone at AT&T. That's kind of the point of starting a new organization. @dgkorn is retired now, and neither he nor the other authors have been involved for quite some time.

Yes it does mean the new organization has to prove itself on its own merits rather than having the imprimatur of AT&T. But this is very common in open source and I'm confident that the community can handle it with some work, and realistic goals.

jghub commented 4 years ago

@jghub, I think you are correct that the license, EPLv1, is not GPL compatible. It looks like we need to add the license back to the master branch, too...

probably :).

I don't think you can count on active involvement from anyone at AT&T. That's kind of the point of starting a new organization. @dgkorn is retired now, and neither he nor the other authors have been involved for quite some time.

understood.

Yes it does mean the new organization has to prove itself on its own merits rather than having the imprimatur of AT&T. But this is very common in open source and I'm confident that the community can

that I understand, of course. the question was whether there are people like @lkoutsofios on your side that could contribute on their own (not as ATT representative) since they are knowledgeable and care for ksh etc. if that is unrealistic -- too bad.

handle it with some work, and realistic goals.

yes. realistic (modest) goals I would like very much...

lkoutsofios commented 4 years ago

@jghub, I think you are correct that the license, EPLv1, is not GPL compatible. It looks like we need to add the license back to the master branch, too...

probably :).

I don't think you can count on active involvement from anyone at AT&T. That's kind of the point of starting a new organization. @dgkorn is retired now, and neither he nor the other authors have been involved for quite some time.

understood.

Yes it does mean the new organization has to prove itself on its own merits rather than having the imprimatur of AT&T. But this is very common in open source and I'm confident that the community can

that I understand, of course. the question was whether there are people like @lkoutsofios on your side that could contribute on their own (not as ATT representative) since they are knowledgeable and care for ksh etc. if that is unrealistic -- too bad.

handle it with some work, and realistic goals.

yes. realistic (modest) goals I would like very much...

I can't play a very active role but I can try to contribute from time to time.

marcastel commented 4 years ago

First draft at a roadmap document

Abstract

The KornShell Project is a community initiative to manage, maintain, and enhance David Korn’s heritage by the same name.

What is KornShell?

KornShell is an interactive command language for the UNIX system that was designed and developed by David G. Korn at AT\&T Bell Laboratories. KornShell is also a complete, powerful, high-level programming language for writing applications and utilities, often more easily and quickly than with other high-level languages such as Perl or Python. This makes it especially suitable for prototyping.

KornShell conforms to the Shell Language Standard developed by the IEEE POSIX 1003.2 Shell and Utilities Language Committee. KornShell has the best features of other shells, such as the the Bourne shell developed by Steven Bourne at AT\&T Bell Laboratories, the C shell developed by Bill Joy at the University of California, or the Bourne Again Shell (bash) developed by the GNU Project. KornShell has the best features of these shells, plus many other features of its own.

KornShell has been and remains the preferred shell in UNIX production environments because of its ability to enhance system administration productivity and the quality of the work of system administrators, both in interacting with the system, and in programming. KornShell programs are easier to write, and are more concise and readable than programs written in a lower level language such as C.

KornShell has the functionality of other scripting languages such as awk, icon, perl, pythonņ rexx, and tcl, making it, in many circumstances, the only interpreted language required to operate and manage your UNIX infrastruture services on premise or on the cloud.

KornShell is and extensible language that uses the same syntax for built-in commands as for foreign commands. System developers can add new commands transparently; that is, with minimum effort and with no differences visible to users other than faster execution. On systems with dynamic linking, it is possible to add new built-in commands at run time.

KornShell has a record track of performance and reliability which has made it a foundational component of both proprietary and open source UNIX distributions since 1982.

KornShell is and remains AT\&T intellectual property. Initially sold (source and binary) by AT\&T, KornShell is now open source managed by a dedicated and active community of users and developers.

Background

KornShell is AT\&T proprietary code which has been open sourced. The KornShell is no longer maintained by any of its original authors. The GitHub AT\&T repository is where the source code for the last stable and official release of the KornShell, as developed at AT\&T, can be found. Updates and further commits in this repository have be done under personal initiatives with no clear roadmap nor governance.

KornShell’s source code is complex, developed over several decades, with dependencies on other AT\&T open source components. Taking over the management of a such source code repository requires a lot of investment in various technical compartments – actual source code, dependent libraries, build system, built-in commands, … Currently too few people actually have good insight on those various components except perhaps for some self-proclaimed champions.

KornShell has lived by its reputation with no mentors or sponsoring entities for more than a decade. This has significantly impacted its penetration in new open source communities and platforms (as opposed to old style infrastructures and proprietary environments). Its capabilities, robustness, performance, and extensibility are rarely known or understood.

KornShell lacks a visible and substantial community of users to demonstrate its use and purpose in UNIX distributions and to position it as a modern and superior shell programming language. It lacks newsgroups, forums and mailing lists to demonstrate its vitality and pertinence.

Roadmap

Short term objectives:

Build a core team
Build a community of users
Establish contributor agreements
Elect for a licensing model
Elect for a versioning scheme
Make binaries available

The rationale behind the current roadmap is to get this project up and running and to grow its footprint in the open source world.

Build a core team Maintaining the KornShell is a challenging project given its current status. For that we need a core team of at least eight or ten highly commited persons. These persons will be requested to significantly invest their time to launch the project.
1. Champion
  
  We need at least one champion who leads the project, not necessarily from a technical standpoint, but as the one who sets (or endorses) the framework and the mid to long term objectives.
  
  The champion is not expected to invest significant time in the project itself, but rather in presenting and promoting the project in the open source world and in coordinating with UNIX distribution maintainers.
  
  The champion is also expected to handle the human aspects of the project and to ensure that the community operates smoothly.
2. Mentors
  
  We need mentors that defend various aspects of the project. Three mentors is probably a good count (and gives a quorum):
  
  1) A mentor to defend the legacy KornShell and its original philosophy; 2) A mentor to represent the user communities and the usability aspects of the shell, including backward compatibility; 3) A mentor to promote the transition and transformation of the code base and of its functionality.
  
  At least one, but preferably two, of the mentors should have good insight on the source code base and have hands on experience either in correcting/extending the code. The champion can also be a mentor while the community is small.
  
  Mentors are expected to invest significantly during the kick off period of this project; one can expect an investment of 10-16 hours per week. As the community grows and habits are taken, the investment can be reduced to 15-20 hours per month.
  
  Mentors should have good English writing skills in that they will be requested to frequently publish blogs and articles.
3. User representatives
  
  User representatives are needed to animate and monitor user communities through newsgroups, forums, chats or whatever platform of interest. I have in mind for instance the BigAdmin repository of system administrator resources for sun servers which was, at one time, my daily reference.
  
  Such representatives should be enthusiastic and have lots of time to spare as this is a very time consuming activity. The required number of representatives is unclear. We should probably at least have two. The more the better. Perhaps this could be a rolling assignment amongst all members of the project.
4. Developers
  
  Developers here are the guardians of the gold master; they are required to acquire a deep understanding of the source code base and to document it so that community members can get a quicker grasp of the code and are up and running faster to contribute bug fixes and new additions.
  
  The minimal requirement is for two developers, but most probably we will need a core devops team of four or six to keep the ball rolling. Developers are expected to regularly do sprints of at least four hours per day to address existing bugs and issues and to build the documentation base.
  
  Developers are also expected to do some mentorship to onboard new developers and encourage them to participate in the code maintenance and expansion.
5. Builders
  
  Builders are here to ensure that the latest release of the KornShell is always available in all supported binary and tarball forms. Builders are also expected to invest time in automating the build workflow and the use of third party platforms.
  
  Though developers could also handle the builder role it is preferable to have a dedicated contact point and person for all build aspects. This role is probably the one with the least workload. It is however of critical importance as on time availability of our deliverables is of paramount importance to build our credibility.
6. Contributors
  
  Contributors are any person willing to make contributions to the project. Contributors must have beforehand agreed upon a contributor’s license.
  
  Contributors are the life blood of the project. We must develop and sustain the potential to grow and renew our community by attracting new contributors.
Other roles will be required – e.g. a web master. Such roles will be addressed as we grow. In the meantime they will be handled on a best effort base by the core team.

All members of the core team have commit access to the source code repository – for security reasons, but, by agreement, only developers enact this privilege.
Build a community of users To establish the KornShell as a mainstream shell it needs visibility. A community of users is probably the best way of achieving this. We need newsgroups, forums, mailing lists, chats or whatever is needed to bring to existence this community or aggregate existing communities. This requires daily implication of at least one maintainer in the project team.
Establish contributor agreements So as to avoid some of the discrepencies noted on the GitHub AT\&T repository we should establish contributor license agreements that each committer is requested to sign, either in his own name, or in the name of the organisation he represents.

See the ASF individual contributor and corporate contributor license agreements as a starting point.
Elect for a licensing model The original AT\&T code is under a copyleft license (EPL). A more permissive licensing model (MIT, Apache 2) might be more appropriate to gain traction amongst developers, at least for the KornShell binary proper – and perhaps not for the other AT\&T libraries and utilities. This needs to be cleared up front with some sort of acknowledgement from AT\&T’s legal department.
Elect for a versioning scheme. We need to break away from the AT\&T versioning model to emphasise the segregation of origins. Semantic versioning is probably the way to go with version 1.0.0 being the exact replicate of the latest patched ksh93u+. Since we want the KornShell to be a standard component of UNIX distributions our versioning scheme should reflect life span with clearly established end-of-life deadlines.
Make binaries available Make the latest versions available as binaries and tarballs for all major package managers. This to build trust amongst operating system maintainers, including proprietary systems. For instance macOS currently includes ksh through a custom build of Darwin with macOS specific patches. The project should take over this build and deliver pkg and dmg images for macOS.

Current status

None – we havn’t yet reached D day.

Risks and rewards

TODO (e.g. orphaned code base)

Project name

TODO: KornShell appears to be the logical name for this project. We need however to check if that name is suitable and if the project has legal permission to continuing using it. Should also be considered, on the long run, if this could induce confustion about who owns the project or possible branding issues in the future.

Required resources

TODO:

Custom email addresses for the core team
Global email addresses for the project (e.g. info@kornshell.plus)
Mailing list(s)
GitHub repository with issues/projects/wiki
Chat channel

Note: The domain kornshell.plus has been reserved for this project (and donated to it) for short or long term purposes.

marcastel commented 4 years ago

@jghub

Thanks for your repeated feedback. I believe we are inline on short term priorities and this should reflect if the previously submitted draft roadmap document.

Consequently, if the KornShell is to survive, it needs a community. But more importantly it needs an organisation and a roadmap. This is, at present -- and in my opinion, more important than deciding on the build system, on the source code versioning environment, or even on the channel for classified confidential discussions.

I do not completely agree regarding priorities. or rather, I'd like to see initially only a very modest short-term roadmap agreed upon and a general consensus that the priority in the short run should be to conserve ksh93u+ and ensure that it builds "everywhere" (the latter to be defined ...) so that its availability can easily be guaranteed.

marcastel commented 4 years ago

@jghub

Clear roles and responsibilities are essential to give a consistent and coherent view to the outside world. They also allow to establish commitment levels within the team. Most, if not all, participants will have a day job. By clearly identifying roles and responsibilities, everybody has a more precise idea of the time that has to be invested by each participant. We move away from the best effort model for better planning of our activities and deliverables.

We dramatically need this visibility. Recent discussions here -- which we should relocate in a proper discussion thread, demonstrate the urge to clarify the situation for the KornShell. Its very existence as a primary shell on UNIX distributions now being at stake.

We need to clearly identify roles and responsibilities both at the technical level but also at the marketing level to promote KornShell to all major UNIX and derivative distributions.

not now.

marcastel commented 4 years ago

@jghub: of course it would be good if ksh93 would have a decent new website pooling historical sources scattered currently over the internet (and partly vanishing already) and up-to-date information like a good FAQ and documentation/help beyond the manpage. but this sounds like a lot of work and would have to wait for now in my view.

Yes documentation is hard work. But I, for one, am ready to invest time and effort to get this, if not done, at least started. Investment being on content not on layout -- the latter can wait. As an example, I very much appreciate the one-man job done by John MacFarlane to maintain Pandoc's documentation and mailing lists. Good documentation means visibility. Visibility means on boarding more people.... hence better chances to keep the KornShell alive :smile: