search

att / ast

AST - AT&T Software Technology
Eclipse Public License 1.0
560 stars 152 forks source link

`command` has unexpected output when part of arrays. #963

Closed orbea closed 4 years ago

orbea commented 6 years ago

Description of problem:

command has unexpected output when part of arrays ("$@").

Ksh version:

bd45ceb6

How reproducible:

Always.

Steps to reproduce:

rm -f /tmp/foo
touch /tmp/foo

set -- command file /tmp/foo
"$@"

or

foo () { "$@"; }

rm -f /tmp/foo
touch /tmp/foo

foo command file /tmp/foo

Actual results: ksh will print the path of file(1).

/usr/bin/file

Expected results: With other shells including ksh variants will print the output of file(1).

/tmp/foo: empty

Additional info:

This issue is also present in the ksh93-2012_08_01-x86_64-2 package from Slackware 14.2.

siteshwar commented 6 years ago

This bug lies in how b_command function is invoked. If it's invoked directly through command like command file /tmp/foo, argc is set to 0. Related comment from b_command() function:

57 // Command is called with argc==0 when checking for -V or -v option. In this case return 0 when -v                             
58 // or -V or unknown option, otherwise the shift count to the command is returned.

src/cmd/ksh93/sh/xec.c:

824                 while (np == SYSCOMMAND) {
825                     int n = b_command(0, com, &shp->bltindata);
826                     if (n == 0) break;
827                     command += n;

But when it's invoked through a variable like $@, argc is set to actual number of arguments and that changes output of the command (See line 1104).

src/cmd/ksh93/sh/xec.c:

1102                                        sh_close(fd);
1103                            }
1104                            if (argn) shp->exitval = (*shp->bltinfun)(argn, com, (void *)bp);
1105                            if (error_info.flags & ERROR_INTERACTIVE) tty_check(ERRIO);
1106                            ((Shnode_t *)t)->com.comstate = shp->bltindata.data;

This code around executing commands needs some careful review.

krader1961 commented 6 years ago

As the O.P. notes the last stable release, ksh93u+, exhibits this behavior. I think this is a bug but given the importance the ksh community places on backward compatibility any change in the behavior will require careful consideration. The key question is whether the original authors of this code had a reason for the difference in behavior. Is there a situation where the current behavior makes sense? Or is it a simple bug introduced due to the size and complexity of the code? Note the sh_exec() function is an obscene 1691 lines in length so a simple bug due to not understanding how all the pieces interact would not be surprising.

orbea commented 6 years ago

Given that most (All?) other modern shells behave in a more expected manner I don't see how this behavior is at all desirable. Additionally there are better ways to print the path of any command such as command -v, which(1) or even with a handmade function.

Other shells I have tested are ash (Slackware and busybox), bash, dash, loksh, mksh, pdksh, posh, yash and zsh.

krader1961 commented 6 years ago

@orbea Please note that both I and @siteshwar agree that the current behavior is surprising and wrong. The question is whether, and how best, to change it given that this behavior has existed for at least six years. The answer to whether it should be changed is almost certainly "yes." The risk is that the straightforward, obvious, fix may break something else. Also, we're currently focused on resolving bugs such as use-after-free and related problems found by tools like ASAN and Coverity Scan. Thus it may be a while before this gets resolved.

orbea commented 6 years ago

Yes I understand and thank you for the elaboration. I was trying to offer my two cents towards backwards compatibility in this case and that I do not think anyone should rely on the current behavior even if I generally agree that striving towards backwards compatibility is a good goal.

orbea commented 4 years ago

@siteshwar and @krader1961 Any news on this bug? Its been over a year and this bug makes ksh hard to support when using command.

siteshwar commented 4 years ago

@orbea We have stopped working on this codebase. See #1466.

orbea commented 4 years ago

@siteshwar So what is the relevant repo now?

hyenias commented 4 years ago

@orbea: As I am new to ksh, I cannot speak towards the possible nuances of command and its related alias. However, if you utilize eval in front of "$@" then it should work as expected.

eval "$@"

Additionally, one can use whence to print the path of any command. Overall, I do not feel that if all the other shells do it then ksh93 must without investigating as to why ksh93 was created in such a fashion. To me the eval --man string seems to be created for such a desired action.

eval is a shell special built-in command that constructs a command by concatenating the args together...

krader1961 commented 4 years ago

@orbea and @hyenias: As @siteshwar said the two of us are no longer working to improve ksh. The new owners seem to believe that ksh93u is perfect and have yet to fix a single bug. Even when I provided a detailed diagnosis of another bug present in ksh93u (see #1467), which has a straightforward fix, they essentially said I introduced the bug and there is no indication they will ever fix it. My recommendation is to switch to a shell being maintained by people who care about fixing bugs. I recommend bash (and don't recommend zsh). Waiting for the people now in charge of this code base to fix bugs is a fools errand.

krader1961 commented 4 years ago

LOL! The new owners of this project have complained many times that I don't care about POSIX compliance. There is a kernel of truth to that in as much as I think long obsolete features like the MAILCHECK var could be eliminated. However, note that ksh93u is not 100% POSIX compliant. This came to my attention when I read the bash POSIX mode documentation. Where Item 38 says this about bash in POSIX compliance mode:

The kill builtin does not accept signal names with a ‘SIG’ prefix.

Sure enough, ksh93u allows the SIG prefix whereas bash does not:

$ bash --posix
bash-5.0$ sleep 33 &
[1] 64330
bash-5.0$ kill -SIGTERM %1
bash: kill: SIGTERM: invalid signal specification

$ /bin/ksh
KSH PROMPT:1: sleep 33 &
[1] 64451
KSH PROMPT:2: kill -SIGTERM %1
[1] + Terminated               sleep 33 &

$ bash
bash-5.0$ sleep 33 &
[1] 64471
bash-5.0$ kill -SIGTERM %1
bash-5.0$
[1]+  Terminated: 15          sleep 33

The relevant POSIX spec is here. So, if you want 100% POSIX fidelity, you might be better off using bash -posix.

orbea commented 4 years ago

Given that ksh is essentially dead with the current direction I think mksh is the next best alternative for a ksh derivative shell. Personally I just use ksh for compatibility testing, but it seems it might be time to retire it entirely.

I just want to be clear that this is all very disappointing...

krader1961 commented 4 years ago

I just want to be clear that this is all very disappointing...

I agree. However, given the numerous quirks of the "official" ksh it seems unwise to use a clone like mksh which attempts to track the real ksh behavior. Those ksh clones will not have the same core bugs, such as blithely doing array[-1] style accesses that can result in a SIGSEGV, but they will be incompatible with the real ksh in many other ways. Honestly, given the current state of the project, and what we know about the state of the code in the ksh93u+ release, the only sensible thing to do is deprecate the use of ksh.

orbea commented 4 years ago

Yes, I realized that mksh won't be fully compatible with the real ksh and I agree about deprecating the use of this repo with the current state. Nevertheless I think mksh is the best working and maintained ksh derivative left, although the only other ones I know and would consider are oksh and loksh which are ports of openbsd's ksh.

Personally I'm partial to yash, but I don't need any ksh specific features.

Edit: Btw this repo doesn't build now, but this old build system is so appalling I have 0 interest in investigating further.

krader1961 commented 4 years ago

Edit: Btw this repo doesn't build now, but this old build system is so appalling I have 0 interest in investigating further.

The AST nmake build system made sense two decades ago when they had to support a lot of platforms that were not even C99 compliant. But that is no longer true and the reversion to that idiosyncratic build system pretty much guarantees the death of this project.

krader1961 commented 4 years ago

@jghub and @jelmd: Care to comment? Or, better yet, fix this bug? Or commit a change that documents this behavior as expected?

cschuber commented 4 years ago

I managed to get it to build under FreeBSD, resurrecting old makefiles and patches. It's currently in a stash. I'm waiting to see what comes of ksh-community before proceeding in one direction or another. The build is simplified compared to the FreeBSD ksh93v- port requiring fewer patches and environmental tweaks.

jghub commented 4 years ago

@orbea: since krader mentioned me, I got a notification, so had a look at this thread with its 15 months history. my remarks concerning your questions/statements:

  1. regarding the issue, I would agree with @hyenias. a) it definitely is the right way to use eval in situations like your example to avoid surprises/failures. I have long stopped (except in very easy cases) to simply do $cmd (where cmd is the constructed command string -- in your case you used $@ for that) and have made it a custom to use eval $cmd which indeed is there for this purpose. the point is, that with eval interpolation of $cmd is done twice: once to construct the string, once when executing (so that eval $cmd does always the same thing as if you type in the "content" of $cmd yourself and that is what you want...).

the reasons when and how $cmd reacts differently to eval $cmd can be subtle. I don't have an answer right now to why ksh does it different to mksh (say) in this special situation and whether it is "intentional" or a bug (but @siteshwar pointed to what is happening in the C code). so that might be worthwhile to investigate given enough time but personally my primary stance is "use eval to execute command strings" which, if obeyed, makes the whole point of whether ksh is wrong or right here mute.

  1. you asked @siteshwar after the "relevant" repo and he pointed you to #1466. I would point you to #1464 if you are really interested what exactly was the tipping point making the repo owners (the ATT team around @gordonwoodhull) decide to revoke commit rights to krader (and also other external committers).

  2. you said "Given that ksh is essentially dead with the current direction". I'd say that depends on whether you mean "ksh2020" is essentially dead" (i.e. the line of development done mainly/nearly exclusively by krader) or "ksh93 is essentially dead".

for the former you might be right if krader does not fork to continue with whatever he wants to do in his ksh2020 effort. for the latter I believe that you are mistaken:

a new github organisation "ksh-community" is in its nascent state (born today: so nothing to see there, yet ;)) and we want to proceed from the last official stable release (i..e. ksh93u+) to which this repo was rewound. so a fork will happen for this. if you are interested you might follow that endeavour.

the initial goal will be modest: to get ksh93u+ (plus patches) build easily for everybody. you are welcome to do a head-to-head comparison of that hopefully-soon-to-be "new" ksh93u+ vs. ksh2020 featurewise, bugwise, performancewise. I would be interested in your final assessment. my own experience is simple: stock ksh93u+ is a more descent offering than ksh2020 in all (well let's say: nearly all) ways. krader's claims to the opposite are wrong. featurewise. bugwise. segfaultwise. performancewise.

so it definitely is not correct in my view that "ksh is dead and no longer maintained". quite to the contrary. but the ongoing readjustments will take a bit of time, naturally. so depending on what you want: maybe stay tuned or switch to mksh or continue to hope for ksh2020 going forward with krader.

  1. mksh: mksh is a very descent shell. I have learned it is the system shell on android so it has a quite good installation base :). it is not a ksh93 clone but a ksh88 clone plus several features from ksh93. so my experience is you can make many things work easily when porting a ksh93 script (if it does not use ksh93-exclusive features like compound variables, say). it has very good POSIX compliance, too. so I recommend mksh over bash, definitely, since it is much closer to ksh93 than bash, much smaller than bash, etc.. and btw: mksh is a fork of oksh which derived from pdksh and much better than the latter two.

here are some benchmarks:

-----------------------------------------
name             ksh   mksh      bash
-----------------------------------------
braces.ksh       1.00  52.45     **fail**
charsplit.ksh    1.00  0.42      0.90
extglob.ksh      1.00  146.33    203.00
fibonacci.ksh    1.00  70.04     77.09
gsub.ksh         1.00  3213.00   **fail**
numfor.ksh       1.00  **skip**  15.77
numloop.ksh      1.00  19.53     21.74
parens.ksh       1.00  40.68     41.11
piln2.ksh        1.00  **skip**  **fail**
plus-equals.ksh  1.00  125.00    5.32
printf.ksh       1.00  72.83     1.33
qsort.ksh        1.00  22.95     6.00
rand.ksh         1.00  **skip**  5.78
rand2.ksh        1.00  **skip**  7.92
sample.ksh       1.00  **skip**  **fail**
shuffle.ksh      1.00  **skip**  **fail**
-----------------------------------------

the benchmarks were primarily written for ksh93 (so the skips/fails are OK :)). these are run time ratios with ksh93u+ as baseline. as you can see mksh is about on par with bash most of the time performance wise. (NB: ksh2020rel would be about 2-3x slower than ksh93u+, ksh2020current would be about 1.5-2. x slower). (NB2: the plusequals.ksh problem of mksh has already been solved and it is now again as fast as bash here.). regarding yash: yes, interesting but the fact that ${#x[@]} does something different from the other shells did it for me... that is a serious incompatibility in my view.

what also should be obivous: ksh93 is in a different league performance wise. this aspect will remain relevant and already a sufficient reason for many people to use ksh rather than bash.

last not least: "old build system/zero interest". up to you, obviously. there are different opinions around regarding this question. maybe ksh-community will provide an alternative build system (Cmake, probably) at some point, rather than only a replacement for the old system.

  1. the assorted statements of krader regarding mksh, bash, and ksh are either reflecting his cluelessness (by no means does mksh try to "track" ksh93 and if it did it would not be a bad thing) or his drive to now harm ksh since he is no longer "in charge". or both.

I would prefer if he would go on with ksh2020 and let the user's decide whether they want it or whether they want ksh93 proper.

jghub commented 4 years ago

and now to the tedious part of setting the record straight and not letting krader get away with spreading intentional misinformation and intentional or clueless falsehoods.

let's start with https://github.com/att/ast/issues/963#issuecomment-588570491

claim: "The new owners seem to believe that ksh93u is perfect and have yet to fix a single bug. " : fact: there are no new owners. this repo is owned by an ATT team. nobody believes ksh93u+ is perfect.

claim: " Even when I provided a detailed diagnosis of another bug present in ksh93u (see #1467), which has a straightforward fix, they essentially said I introduced the bug and there is no indication they will ever fix it."

fact: read through #1467, notably here https://github.com/att/ast/issues/1467#issuecomment-586997694: this is a 100% reproducible ksh2020 bug with zero occurrence in ksh93u+.

claim: "My recommendation is to switch to a shell being maintained by people who care about fixing bugs. I recommend bash (and don't recommend zsh). Waiting for the people now in charge of this code base to fix bugs is a fools errand."

my opinion (but mostly "fact"): the "people" of course do care for bugs, but see above regarding #1467: we do no longer care for ksh2020 bugs. krader is free to recommend whatever shell he likes (previously fish, now bash). unfortunately he does not know (Korn)Shell language well (quite to the contrary) so his newly discovered preference for bash is of no relevance whatsoever to me. I believe he is driven by other motives now anyway, obviously. his final "fools errand" statement is a baseless prognosis to spread "fear and uncertainty" among users. we will see what really is happening in the future. I also reiterate ("fact") that stock ksh93u+ is superior to ksh2020 to the user in each and every relevant aspect that I have tested (stability, compatibility, performance)-- even without any future bug fixes.

jghub commented 4 years ago

setting the record straight, part 2: https://github.com/att/ast/issues/963#issuecomment-588579793

claim: "However, note that ksh93u is not 100% POSIX compliant. ... The relevant POSIX spec is https://pubs.opengroup.org/onlinepubs/007908799/xcu/kill.html. So, if you want 100% POSIX fidelity, you might be better off using bash -posix."

fact: ksh supports the POSIX-required behaviour (and only documents it in this way in its manpage regarding kill). POSIX is a minimal requirement and does not forbid to do "more". ksh93 is a superset of a strict POSIX shell. the `--posix' flag of bash is just an attempt (don't know how successful) to make bash behaviour more predictable (it's usually quite messy and prone to surprise you ...). ksh93 seems not to provide an option to restrict it to a bare-bones POSIX shell (but if I want that I can use other smaller shells...). ksh is fully POSIX compliant and does run all POSIX scripts correctly AFAIK.

jghub commented 4 years ago

setting the record straight, part 3: https://github.com/att/ast/issues/963#issuecomment-589018928

krader's claims:

However, given the numerous quirks of the "official" ksh it seems unwise to use a clone like mksh which attempts to track the real ksh behavior. Those ksh clones will not have the same core bugs, such as blithely doing array[-1] style accesses that can result in a SIGSEGV, but they will be incompatible with the real ksh in many other ways. Honestly, given the current state of the project, and what we know about the state of the code in the ksh93u+ release, the only sensible thing to do is deprecate the use of ksh.

facts:

I disapprove of this behaviour.

jghub commented 4 years ago

setting the record straight, part 4 :https://github.com/att/ast/issues/963#issuecomment-588600177

krader's claim:

The AST nmake build system made sense two decades ago when they had to support a lot of platforms that were not even C99 compliant. But that is no longer true and the reversion to that idiosyncratic build system pretty much guarantees the death of this project.

facts: top priority of the ksh-community effort will be to provide an easy-to-build ksh93u+ again for all current/major platforms (on which ksh2020 is building, say). whether and when an alternative build system will be provided remains to be seen. it is totally possible.

opinion: the "death of the project" (that is, in the present context, only ksh93 although there is much more to the ATT repo here...) would have been to let krader go on. that has luckily been prevented by the ATT team.

from now on this utterance is only krader's wishful thinking.

jghub commented 4 years ago

@cschuber

I managed to get it to build under FreeBSD, resurrecting old makefiles and patches. It's currently in a stash. I'm waiting to see what comes of ksh-community before proceeding in one direction or another. The build is simplified compared to the FreeBSD ksh93v- port requiring fewer patches and environmental tweaks.

ksh93u+ you mean? good! are you the official maintainer of the ksh port(s) for freebsd (I am not on FreeBSD...)? the discussion so far in the just starting ksh-community org considers to try and provide a way for the major "distros" to just download+compile w/o necessity of local patching. maybe your experience with freebsd than can come in handy?

orbea commented 4 years ago

@jghub That is most certainly not the way to use eval, that is a terrible suggestion and you should feel bad for making it.

Additionally you wrote a lot of stuff without really saying anything relevant or otherwise. My honest advice is to remove yourself from this repo and don't come back. Bluntly its so amazing how bad you are this that there are no words. The ksh you so idealize has been long dead for at least 8 years and will not come back no matter how badly you kill this project.

jghub commented 4 years ago

@orbea:

orbea commented 4 years ago

Using eval on arbitrary input is generally not safe and does not at all accomplish what command does. Please read the posix spec and reflect upon it until you understand why you are so wrong. As a hint, in my actual script I use command -p which results in the same issue.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html

You ever meet that person that says a lot of things without conveying any real information, then proceeds to hammer at it until everyone else leaves out of frustration ultimately killing whatever activity that was occurring before they appeared? That is you. Words can't express how much disappointment I have in what you accomplished here.

jghub commented 4 years ago

Using eval on arbitrary input is generally not safe and does not at all accomplish what command

of course not. nobody said so. eval and command have nothing to do with each other. you hit a general problem in that $cmd (where cmd holds whatever constructed expression/command) does not in general achieve what you naively might expect: execution of the string value as if you would have typed the same text at the command line. eval $cmd does that. that's what @hyenias was suggesting and I agree that he is right.

does. Please read the posix spec and reflect upon it until you understand why you are so wrong. As a hint, in my actual script I use command -p which results in the same issue.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html

I know what the command builtin is for and what it does, thank you very much (that question was mostly at the heart of the incredible #1445, by the way).

what you see (difference between $cmd and eval $cmd) is not specific to the occurrence/using of the command builtin in your "command string" $@. what is specific to command being part of the $cmd string is that it seems(...) that in your special example $cmd and eval $cmd should yield the same (and this being the naively expected) result and that this might be a ksh bug. but what you specifically want to achieve in your script/use case is not relevant to the actual problem/question.

You ever meet that person that says a lot of things without conveying any real information, then proceeds to hammer at it until everyone else leaves out of frustration ultimately killing whatever activity that was occurring before they appeared? That is you. Words can't express how much disappointment I have in what you accomplished here.

I repeat: stop ranting. please spare me that. stay to the facts. read #1464 and #1466 if you care what actually has happened (or read through a ton of older issues documenting what has gone on here for a couple of years).

and also realise what has not happened: nobody took krader's "baby" (ksh2020) away from him. he just is no longer allowed to foster it officially in the ATT/AST repository. this is a good thing in my view. if this development is bad in your view, you might tell that to krader and ask to cooperate with him on ksh2020 in a fork.

several people, me included, believe that another road should be taken for ksh93. in another fork. we will try that.

the problem never was the sole existence of ksh2020 or krader's questionable approach or insufficient suitability to modify the ksh93v- code base. the latter only explains, why ksh2020 is, what it is today: inferior to ksh93u+. if ksh2020 would have been "just another project on github" no harm would have been done.

but the problem was that there was a clear strategy to establish ksh2020 as "the" ksh everywhere no matter what and (wrongly) sailing under the ATT/AST flag was instrumental to that effort. that did harm (by displacing ksh93u+ completely in some distros, replacing it by a ridiculously buggy allegedly "stable" release in the fall of 2019) and that harm has now been stopped it seems. I am happy about this development. if a ksh2020 package appears in distros beside the ATT sanctioned ksh93u+ release (or a new ksh93u+ upstream incorporating incremental bug fixes etc) I am very comfortable with the situation. the more so, since users then could very easily compare the two and find out what they want rather than being force-fed ksh2020.

there is now a clear statement by the ATT/AST team that the official last stable release of ksh is ksh93u+. this will ensure long-term availability of that release (and subsequent bug-fix releases, hopefully, if we succeed).

and for the second time: I propose to leave it at that since you won't convince me and I won't convince you. other readers will make up their own minds...

gordonwoodhull commented 4 years ago

I don't know the history of this repo but IMO when there is this much animosity between two camps in an open source community then forking is the civil thing to do.

I stand by our decision to revert this repo to ksh93u+ (2012) and I wish the best of luck to both ksh-community and ksh2020.

AT&T is not able to actively maintain this project, so we need both new communities to fork this repo and continue development elsewhere.

orbea commented 4 years ago

AT&T is not able to actively maintain this project, so we need both new communities to fork this repo and continue development elsewhere.

@gordonwoodhull This repo had two capable, willing and active maintainers. Now it has 0, if anyone wants the old broken 2012 repo they should do that elsewhere. This result is not only disruptive, it is highly destructive to the project and the commit history. Most of the community was not involved in this decision process.

gordonwoodhull commented 4 years ago

There was about a week of discussion in #1464 and #1466. It is true that not too many people on the ksh2020 side participated.

I still hope that @krader1961 and @siteshwar will consider taking the ksh2020 fork (on the ksh2020 branch) to another repo. Forks are very common on GitHub; in fact they are a deep part of the design of git. There is nothing any different about github.com/att from any other location on GitHub.

In any case, calling ksh2020 or ksh-community an AT&T project does not make sense, since we have no one from the AT&T side who is able to actively participate.

Nor are we able to moderate the increasingly bitter debates that are going on here.

We ask both sides respectfully to find their own spaces to continue the work.

jghub commented 4 years ago

@orbea I naturally accept your preference for the ksh2020 line of development, I just don't share it. however, you wrote to @gordonwoodhull:

This repo had two capable, willing and active maintainers. Now it has 0, if anyone wants the old broken 2012 repo they should do that elsewhere. This result is not only disruptive, it is highly destructive to the project and the commit history. Most of the community was not involved in this decision process.

and this necessitates a few comments:

otherwise it seems a bit bizarre, that @gordonwoodhull has to repeatedly explain to "the ksh2020 side" the idea behind github and the option to fork the present repo. ksh2020 could go on without any hiccup after the fork and undo the recent changes to repo layout.

what the "ksh2020 side" cannot accept, seemingly, is that after forking ksh93u+ and ksh2020 would be at eyelevel with each other so that the "best might win". before the rollback, ksh2020 was perceived as representing the "official ksh development" (which it never was!). this had the previously described adverse consequences of marginalising (ultimate goal of the ksh2020 project: eliminating) ksh93u+.

orbea commented 4 years ago

Nor are we able to moderate the increasingly bitter debates that are going on here.

There is only one person here contributing to that, ban @jghub for being a toxic spammer and the issue will vanish.

We ask both sides respectfully to find their own spaces to continue the work.

We had a perfectible working space here already, please fork the old bitrotten code to another repo.

jghub commented 4 years ago

@gordonwoodhull: I am very sorry for the noise and misguided anger you receive (even if only from 1-2 persons altogether) for your team's decision and your personal volunteering and committing time to make the transition as smooth as possible. almost everybody following this "affair" sure appreciates your approach and contribution a lot.

FYI: ksh-community will/does happen. maybe one of the other guys involved will give an update soon.

matvore commented 4 years ago

There is only one person here contributing to that, ban @jghub for being a toxic spammer and the issue will vanish.

I don't know @jghub personally, but considering they have a big investment in ksh93 (a large codebase of productivity scripts and a long .kshrc, based on https://github.com/att/ast/issues/1445 and in particular https://github.com/att/ast/issues/1445#issuecomment-559687232), it is hard to blame them for getting emotionally engaged while criticizing the state of the repo.

Distro and repo maintainers don't have the time or attention needed to determine which commit in this repo is the real stable, backwards-compatible, performant one. They will just pick the commit that is tagged as stable or tagged with the highest version number. My employer doesn't currently let me chsh to ksh93u+ since (apparently) it uses the Debian repos and ksh2020 is considered the modern replacement, so ksh93u+ is not on the list of blessed shells. Granted, I can probably hack something together by building ksh93u+ myself and putting an exec in .kshrc, but this kind of roadblock is unsettling to have to deal with. Right now I'm using ksh93u+ on everything except my main machine.

Speaking of which, I intend to raise a ticket with our support to swap out ksh2020 for ksh93u+.

We had a perfectible working space here already, please fork the old bitrotten code to another repo.

I like the version you call "bitrotten" better than the ksh2020 fork. I ran into a glaring stability issue with ksh2020 after using it for about a day (https://github.com/att/ast/issues/1467) and it has performance degradation which seems to have no particular cause but is apparently gradual and something that was not noticed until it was too late (https://github.com/att/ast/issues/1449). Also, the most active ksh2020 maintainer seems to have strong opinions on what ksh should be used for, and doesn't mind backwards incompatible changes that are justified by that mission. This kind of mindset is what an unofficial fork is for, and I think @krader1961 has the drive to actually make such a fork work.

It seems like over the decades ksh93u+ found a comfortable local maximum of quality, and ksh2020 has ventured away from it to look for a better state (using a more compatible build system and swapping out a quirky memory allocator are good ideas in this light, though the latter has caused issues that haven't been addressed).

But ksh2020 hasn't found a new local maximum yet. Unfortunately, a fork requires dropping the well-recognized ksh name and competing with more trendy options (like zsh, fish, or bash) on merit alone, and I don't suspect ksh2020 is ready for that yet.

orbea commented 4 years ago

I don't know @jghub personally, but considering they have a big investment in ksh93 (a large codebase of productivity scripts and a long .kshrc, based on #1445 and in particular #1445 (comment)), it is hard to blame them for getting emotionally engaged while criticizing the state of the repo.

Being emotionally engaged is understandable, long off-topic and irrelevant diatribes that amount to mere spam, harassment and bad advice is not acceptable behavior.

Users that depend on bugs and / or undefined behavior should not expect any backwards compatibility and their code should be broken, preferably the sooner the better so that it gets fixed instead of having decades old bugs that no one can or will fix like the current state of this repo. Its unavoidable that with a code base like this its impossible to fix everything without breaking some things in the process. In my experience upon finding such regressions and reporting them on irc they were immediately fixed (https://github.com/att/ast/issues/959). The old 2012 code is still in the current Slackware stable release (14.2) and its not practical to use at all for anything other than ancient and unmaintained scripts which most people have long since abandoned.

Also your stability issue is apparently present in the old code, but I can't reproduce it with either ksh2012 or ksh2020 in Slackware stable and Slackware current respectively. And the relatively minor performance regressions was a result of fixing much more serious issues with valgrind/asan. I agree that ksh2020 is still rough around the edges, but that is the result of being based on a very old and broken code base that had no future until a few people spent the time keeping it alive. However their reward for years of hard effort is to throw the baby out with the bathwater.

Personally unless the ancient bug documented in the op of this issue is fixed, I can not support ksh for my own scripts regardless of what happens to this repo.

cschuber commented 4 years ago

@cschuber

I managed to get it to build under FreeBSD, resurrecting old makefiles and patches. It's currently in a stash. I'm waiting to see what comes of ksh-community before proceeding in one direction or another. The build is simplified compared to the FreeBSD ksh93v- port requiring fewer patches and environmental tweaks.

ksh93u+ you mean? good! are you the official maintainer of the ksh port(s) for freebsd (I am not on FreeBSD...)? the discussion so far in the just starting ksh-community org considers to try and provide a way for the major "distros" to just download+compile w/o necessity of local patching. maybe your experience with freebsd than can come in handy?

I haven't decided yet what our ksh ports/packages will look like. We currently have ast-ksh (maintained by @saper), ksh93 (currently at 2020.0.0), ksh93-devel (just prior to the big revert), and ksh2020 (based on the new 2020 branch). I maintain the ksh* ports. @saper and I will discuss our plan going forward, probably in March as I have a high priority issue I'm working on now. As to contributing to the ksh-community effort, I can help out in a minor way, with a number of projects underway right now. You might want to ask @saper, the other ksh maintainer at FreeBSD.

jghub commented 4 years ago

Being emotionally engaged is understandable, long off-topic and irrelevant diatribes that amount to mere spam, harassment and bad advice is not acceptable behavior.

let the readers decide whether they agree with you regarding 'off-topic', 'irrelevant', 'spam', 'harrassement' 'bad advice', 'not accetable behavior'. but Chapeau!: 6 invectives in a single sentence is really impressive (if one cares for this communication style). and actually I am rather calm, not prone to emotional outbursts. which cannot be said of everybody around here.

Users that depend on bugs and / or undefined behavior should not expect any backwards

I have no idea what you might mean by that in the context of ksh93u+ vs ksh2020. do you?

compatibility and their code should be broken, preferably the sooner the better so that it gets fixed instead of having decades old bugs that no one can or will fix like the current state of this repo. Its unavoidable that with a code base like this its impossible to fix everything without breaking some things

you also are continuing to ignore that ksh2020 has not "fixed" the ksh93u+ code base but has heavily modified the ksh93v- code base.

if ksh2020 is even more buggy than ksh93v- I cannot reliably say but I would suspect it is. in any case, 93v- is buggy (and so is ksh2020, definitely!) and 93u+ is not anywhere near that fragile state. if you would care you could test it yourself rather than ranting. I am a dedicated believer in hard facts and hard data.

in the process. In my experience upon finding such regressions and reporting them on irc they were immediately fixed (#959). The old 2012 code is still in the current Slackware stable release (14.2) and its not practical to use at all for anything other than ancient and unmaintained scripts which most people have long since abandoned.

"not practical to use etc": this is utter nonsense. you obviously are not a serious ksh user at all, I presume. and ksh2020 has not a single new feature. it is just more buggy than ksh93u+. so why is it "more practical to use"?

Also your stability issue is apparently present in the old code, but I can't reproduce it with either

in OSX it is a 100% reproducible segfault in ksh2020 (not a "stability issue"). nobody was able to see that behaviour with ksh93u+ so far. so: no. it is only reported with ksh2020.

ksh2012 or ksh2020 in Slackware stable and Slackware current respectively. And the relatively minor performance regressions was a result of fixing much more serious issues with valgrind/asan. I agree that

a factor of 2-3 performance degradation is "relatively minor"? ok ... interesting attitude. would you accept if the next clang/gcc release would compile to binaries 2-3 times slower than before?

ksh2020 is still rough around the edges, but that is the result of being based on a very old and broken

where does your knowledge regarding "broken" come from? from krader telling you so? gvie me the data, please... broken means buggy. please provide the bug-list comparison for user-facing bugs. out of the back of my mind I estimate I could quickly collect 5-10 issues from the history here with bugs present in ksh2020 and absent from 93u+. the present issue and the one I reported in #1464 are the only two which also apply to ksh93u+ that I am immediately aware of. as said, krader (or you?) are welcome to provide hard data with a head-to-head comparison which bug applies to what shell. I am talking user-facing bugs (segfaults, wrong behaivour, breaking changes, etc)

code base that had no future until a few people spent the time keeping it alive. However their reward for years of hard effort is to throw the baby out with the bathwater.

nonsense. accept what you have been told by the ATT team: everybody wanting to further work on the code base can just fork. at least ksh93u+ (and what might, slowly and carefully, evolve from it) will have a future, I am rather sure.

Personally unless the ancient bug documented in the op of this issue is fixed, I can not support ksh for my own scripts regardless of what happens to this repo.

up to you. you have been told to use eval. your reply regarding the dangers of eval is completely beside the mark if you are willing/wanting to use $cmd expansion for code execution. so the issue is practically a non-issue (but still might be a bug including ksh93u+).

jghub commented 4 years ago

@cschuber

I haven't decided yet what our ksh ports/packages will look like. We currently have ast-ksh (maintained by @saper), ksh93 (currently at 2020.0.0), ksh93-devel (just prior to the big revert), and ksh2020 (based on the new 2020 branch). I maintain the ksh* ports. @saper and I will discuss our plan going forward, probably in March as I have a high priority issue I'm working on now. As to contributing to the ksh-community effort, I can help out in a minor way, with a number of projects underway right now. You might want to ask @saper, the other ksh maintainer at FreeBSD.

thanks for this clarification of the situation. I seem to recall @saper arguing for maintaining the full AST code rather than only ksh during the discussion in #1466 (I believe?) but was not aware that he maintains ast-ksh at FreeBSD (I remember not so long ago, FreeBSD users telling me that they were not happy since they could not "avoid" ksh2020 because it had replaced 93u+ in the FreeBSD ports? did I get that wrong?)

regarding ksh-community: I believe everybody involved (so far 6 people) will welcome input/feedback from you two.

jghub commented 4 years ago

@matvore:

It seems like over the decades ksh93u+ found a comfortable local maximum of quality, and ksh2020 has ventured away from it to look for a better state (using a more compatible build system and swapping out a quirky memory allocator are good ideas in this light, though the latter has caused issues that haven't been addressed).

But ksh2020 hasn't found a new local maximum yet. Unfortunately, a fork requires dropping the well-recognized ksh name and competing with more trendy options (like zsh, fish, or bash) on merit alone, and I don't suspect ksh2020 is ready for that yet.

I like that picture of "local maxima". it describes the situation rather nicely. and it is correct that ksh93u+ is the result of a decade long search for the optimum. to stay in that picture: ksh93v- also strayed away from the ksh93u+ maximum in search of a "higher peak". which search korn et al. were not able to complete. and ksh2020 than strayed further away from the already "less optimal" last position of ksh93v-.

the build system question is a valid one. the emerging ksh-community org is considering to also look into this (but not immediately, I believe, since there are more pressing priorities).

the malloc subsystem question is also valid but here (switching away from ast vmalloc) I believe is possibly the root course of many "original" ksh2020 bugs/regressions as well as the performance degradation. but I am not an expert in these things.

regarding the naming/branding: I don't believe that is a disadvantage for the future of ksh2020: if it really becomes at some point in the future superior (or an attractive alternative) to ksh93u+ and its subsequent bugfix releases (to state only the most modest goal for the future of 93u+), than it will be an advantage to distinguish it as "ksh2020" (or whatever name the ksh2020 supporters might come up with) from the "old-fashioned and conservative" ksh93.

my expectation (and personal preference) is, that most current ksh users will be content and happy with a stable and incrementally improving (by bug fixing) standard ksh93u+ since it leaves not much to be desired featurewise (and performance is about 1-2 orders of magnitude higher than bash and zsh). so far, in my view, ksh2020 is just the not really successful attempt of a code-overhaul of ksh93v-, culling all new features in the process (thus making ksh2020 feature-identical with ksh93u+), which did not create anything "original". a "ksh93v+" release would be great, but ksh2020 not even attempted to go there.

and if ksh is going to increase its user base, than this will -- in my view -- only happen by emphasizing the high standards compliance and performance. yes, ksh93 does have additional features distinguishing it from bash (zsh is a mess anyway) but that will not draw people away from bash. performance might.

marcastel commented 4 years ago

Forking is the civil thing to do

@gordonwoodhull thank you for your conciliating and logical approach. May I ask you one more favour: return to a clean state. The state at which AT&T support for AST developments were ended. 2012 was the end year. No teams are needed, no wikis, no projects, no releases or anything else. And despite losing knowledge and information, even issues should be zapped given the appalling image they promote. Anything done since is more than welcome to continue its existence, as rightly mentioned, in its own fork.

it is my opinion that we owe respect to the work done by all those at AT&T that made AST exist and have allowed for decades, and still today, system administrators around the world to rely on an efficient, secure, performant and feature rich shell and programming language. Despite not having been updated since the 2012 release, ksh93u+ remains the Kornshell of UNIX and POSIX certified (as opposed to mostly POSIX compliant) operating systems such as AIX or macOS and as delivered respectively by IBM and Apple.

It is a gift to, and consequently a responsibility for, the entire Open Source community to make this donation of proprietary intellectual property continue to exist here, alone, unmodified, in its final state as provided to us by AT&T (or more importantly by those at AT&T who took responsibility to bring this to us). AST software, and the KornShell in particular, are part of the UNIX history.

All are encouraged to fork, transform, adapt, evolve, praise or blame as they desire. But not here. This is the AT&T repository. The software is end of life and no updates, by AT&T, have been or will be provided here. Consequently we don't need issues. We don't need teams. We don't need projects. We don't need releases. We need a legacy vault with all reference material made available by the AST team in its final state.

Thank you David Korn, Glen Fowler, Kiem-Phong Vo and all those at AT&T who have worked hard to make this exist. May your code RIP here as-is and may it take on a new life of its own elsewhere, triggering innovation and evolution as was the case at the very inception of the KornShell.

I highly suggest that it is now time to turn this page and restore, in this repository, some dignity and savoir-vivre owed to those who have made this software available in the first place.

cschuber commented 4 years ago

@cschuber

I haven't decided yet what our ksh ports/packages will look like. We currently have ast-ksh (maintained by @saper), ksh93 (currently at 2020.0.0), ksh93-devel (just prior to the big revert), and ksh2020 (based on the new 2020 branch). I maintain the ksh* ports. @saper and I will discuss our plan going forward, probably in March as I have a high priority issue I'm working on now. As to contributing to the ksh-community effort, I can help out in a minor way, with a number of projects underway right now. You might want to ask @saper, the other ksh maintainer at FreeBSD.

thanks for this clarification of the situation. I seem to recall @saper arguing for maintaining the full AST code rather than only ksh during the discussion in #1466 (I believe?) but was not aware that he maintains ast-ksh at FreeBSD (I remember not so long ago, FreeBSD users telling me that they were not happy since they could not "avoid" ksh2020 because it had replaced 93u+ in the FreeBSD ports? did I get that wrong?)

Prior to my involvement FreeBSD never had 93u. The latest was based on a couple of files it fetched from www2.research.att.com. I don't recall what version they were. They may have been 93v.

To my knowledge the only person to raise this issue was @saper. OTOH updating ksh93 to the development here addressed a couple of segfaults.

I'm working out a plan true up the existing ksh93* ports. The ksh2020 port will remain and probably follow ksh-community.

regarding ksh-community: I believe everybody involved (so far 6 people) will welcome input/feedback from you two.

gordonwoodhull commented 4 years ago

@marcastel, sure, but in good time. The wiki has already been moved. Some day down the line, once discussion has died down and new communities formed, we will officially archive this repo and close it for comments.

I am waiting to see if ksh2020 starts a new fork and wants to transfer issues. Ditto for releases.

I take it ksh-community does not want to transfer any issues, even if they pertain to ksh93u+?

jghub commented 4 years ago

@gordonwoodhull

I take it ksh-community does not want to transfer any issues, even if they pertain to ksh93u+?

as far as keeping the information is concerned, quite to the contrary, I'd say. there is important information buried in the issues history that is relevant to ksh93u+ one way or another. e.g., to compile and address a list of bugs reported (closed or open) for ksh2020 that also apply to ksh93u+ because they reflect ksh93v- bugs inherited from ksh93u+.

so the information should in my view be kept easily available to ksh-community. but it need not (actually: should not I'd say) become somehow part of the issues history of the new active repo(s). that would be pointless.

so what is the best way, technically, to achieve the above?

saper commented 4 years ago

I am not sure it is terribly relevant here, but here it is:

Prior to my involvement FreeBSD never had 93u. The latest was based on a couple of files it fetched from www2.research.att.com. I don't recall what version they were. They may have been 93v.

https://svnweb.freebsd.org/ports/head/shells/ast-ksh/Makefile?r1=495520&r2=499547 switched from ksh93 2012-08-01 (which is tip of the ksh93u branch here) to ksh93 2014-12-24 which is ksh93v-). 2016-01-10 version is one commit ahead but somewhat the version number has not been changed everywhere in the code.

To my knowledge the only person to raise this issue was @saper. OTOH updating ksh93 to the development here addressed a couple of segfaults.

This fixed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208098 crash, which caused ksh93u to crash when executing the file containing this:

a() { b }

ksh93v- had also multiple patches already in that made ksh building possible on BSD.

Apart from pretty broken filename completion/editing (something I struggle during my daily interactive work with ksh93v+) there is also a pretty interesting bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238248 - it seems that AST builtins started to override the system-shipped commands for some reason. (Initially I thought this was by design, now I believe it is a bug in ksh93v, relatively easy to fix).

jghub commented 4 years ago

@saper thanks for bothering. I believe getting all the pieces from everywhere together is valuable...

questions:

  1. "ksh93v+" is a typo and you still mean 93v-, right?

  2. that example should read a() { b; }, correct? if yes this runs fine here on OSX (unspectacularly, though ;))

  3. the "multiple patches" would be good to know in more detail. I presume they then should be backported (where applicable) to 93u+. can you provide more details?

  4. builtins overriding system commands: without reading up on it again, I believe that, indeed, builtins do have preference over external commands by design. always... I recall that I did switch off the new ls builtin when testing 93v- (there was a setting or envir var for that I believe (too long ago...))

  5. have you tested 93u+ against 93v- in FreeBSD? what is your assessment? as said, my own experience (and that seems to reflect experience out there in general) was, that I went back to 93u+ and was happy again.

saper commented 4 years ago

Since this bug got hijacked to general comments about ksh93 future here is what I would like to do:

I hope I can work on this in this or another repository. Somebody needs to bless releases, though. I think we should start with ksh93v-, otherwise lots of interesting features that were in the works will get lost.

saper commented 4 years ago
  1. "ksh93v+" is a typo and you still mean 93v-, right?

Yes, fixed.

  1. that example should read a() { b; }, correct? if yes this runs fine here on OSX (unspectacularly, though ;))

No, see the original bug report. It should be a() { b }

  1. the "multiple patches" would be good to know in more detail. I presume they then should be backported (where applicable) to 93u+. can you provide more details?

Linked in the next comment: https://svnweb.freebsd.org/ports/head/shells/ast-ksh/files/

  1. builtins overriding system commands: without reading up on it again, I believe that, indeed, builtins do have preference over external commands by design. always... I recall that I did switch off the new ls builtin when testing 93v- (there was a setting or envir var for that I believe (too long ago...))

Reading the specification those builtins should be active only if /opt/ast/something is on the PATH. As I have started to analyze the code I started to believe this is a simple macro bug so that should not be treated as the desired behavior.

  1. have you tested 93u+ against 93v- in FreeBSD? what is your assessment? as said, my own experience (and that seems to reflect experience out there in general) was, that I went back to 93u+ and was happy again.

93v- wins for me (but maybe only for me). I haven't looked back after upgrading really. I have only one bug report filed with FreeBSD so far, but I am neither ksh expert not a very heavy user.

jghub commented 4 years ago

thanks for your thoughts and the provided links. it seems that you'll resonate well in several respects with @marcastel and @dannyweldon if I am not mistaken.

personally, I believe ksh93v- has to wait and one should not stumble into a situation similar to ksh2020. in the longer run: maybe. I also agree that 93v- has interesting new features (but essentially all of them did not quite work etc.).

so ksh-community now really first wants to "play it save" and see to it that 93u+ (plus collected patches/fixes) becomes available (== hassle-free buildable) to everybody again ASAP. the reasons for this have been discussed at too much length in the present repo's issues all over the place. :|

if we have reached a stable state (hopefully within a couple of months or end of year???) one can discuss further options. e.g. : what about starting to incrementally backport features from 93v- to the present u+ so that one can get them to work one by one?

saper commented 4 years ago

Is there any list of issues with ksh93v vis-a-vis ksh93u?

jghub commented 4 years ago

No, see the original bug report. It should be a() { b }

really? don't get it: that just throws an (to be expected, I'd say) syntax error with 93u+ at least