search

oilshell / oil

Oils is our upgrade path from bash to a better language and runtime. It's also for Python and JavaScript users who avoid shell!
http://www.oilshell.org/
Other
2.79k stars 150 forks source link

Test OSH on real shell scripts #161

Open andychu opened 5 years ago

andychu commented 5 years ago

This is one of the best ways to help the project! Find some important / interesting shell scripts, and run them with OSH.

There are a bunch of ideas in test/wild.sh, and published with each release:

http://www.oilshell.org/release/0.5.0/test/wild.wwz/

Off the top of my head, these capture the flag scripts are interesting:

https://github.com/janosgyerik/capture-the-flag

gaycomputers commented 5 years ago

How do we contribute after testing the scripts?

andychu commented 5 years ago

If the script doesn't behave the same way that it does under bash, then definitely file a bug here.

If it works, you can simply reply to this bug (#161) or you can post a message on #oil-discuss at https://oilshell.zulipchat.com/ . I'd like to hear success stories!

Also, I added an option for "deep testing", which I should document. After installing OSH, you can now do this:

OSH_HIJACK_SHEBANG=$(which osh) osh
$ ./foo.sh

If foo.sh has a shebang of #!/bin/sh, #!/bin/bash, etc., then OSH will hijack it and run it as itself.

It's also possible to view the debug stream like this:

mkfifo _tmp/debug
OSH_HIJACK_SHEBANG=$(which osh) osh --debug-file _tmp/debug  # in one tmux pane
cat _tmp/debug  # in a separate tmux pane

I've mentioned this in a few places, but I should write a blog post about it:

https://github.com/oilshell/oil/wiki/Debugging-Completion-Scripts

Thanks for asking and let me know if you have problems testing!

Someone already reported a Ctrl-C bug that I didn't know about, so I'm sure there are more issues. Everybody tickles a different part of the shell :)

andychu commented 5 years ago

This wiki page will have some ideas of things to test:

https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run-Under-OSH

Crestwave commented 5 years ago

Pure Bash programs tend to use more obscure Bash features and thus are good for testing. @dylanaraps has quite a few of them, e.g., https://github.com/dylanaraps/pure-bash-bible, https://github.com/dylanaraps/birch, etc.

wertercatt commented 5 years ago

The popular shell script neofetch is non-functional when ran through osh. It simply outputs four errors.

powerkitten@blaze-pc:~$ osh /usr/bin/neofetch
  shopt -s nocasematch
  ^~~~~
/usr/bin/neofetch:38: 'shopt' Invalid option 'nocasematch'
  read -rd '' config <<'EOF'
       ^~~
/usr/bin/neofetch:48: 'read' doesn't accept flag '-d'
  %b\e[3%sm
   ^
(source.ArgvWord word_spid:30042):1: osh printf doesn't support backslash escaping (try $'\n')
      ((simple)) && exit
        ^~~~~~
/usr/bin/neofetch:4769: warning: Coercing undefined value to 0 in arithmetic context
dylanaraps commented 5 years ago

I'm willing to help in any way I can. I've been following this project for quite a while. :+1:

It might be a better first step to test some POSIX sh scripts with oil.

These would also make a good bash test-case: https://github.com/dylanaraps/codegolf

wertercatt commented 5 years ago

Of note, osh is 0/6 on running your codegolf snippets @dylanaraps.

andychu commented 5 years ago

Thanks for trying it! I filed #356 and #357 based on this. I've seen those 2 constructs elsewhere so it definitely seems like we need them.

The last one is a warning that's probably legitimate.

I think nocasematch is just a flag on fnmatch(), so that one isn't hard either.

Any help is appreciated!

andychu commented 5 years ago

@dylanaraps I recognize your project because @Crestwave mentioned it! If you are interested in contributing to Oil I'm available to help, either here or on https://oilshell.zulipchat.com

@wertercatt A list of all the failures would be useful!

wertercatt commented 5 years ago 
powerkitten@blaze-pc:~/Desktop/osh-testing$ cat divisors.sh
for((;i++<100;)){(set $[i%++j?0:{1..100}];echo ${@#0})}
powerkitten@blaze-pc:~/Desktop/osh-testing$ osh divisors.sh
  for((;i++<100;)){(set $[i%++j?0:{1..100}];echo ${@#0})}
                  ^
divisors.sh:1: Invalid word after for expression
powerkitten@blaze-pc:~/Desktop/osh-testing$ cat fibonacci.sh
1
printf %d"
" $[i+=_,_=i-_]{a..C}
powerkitten@blaze-pc:~/Desktop/osh-testing$ osh fibonacci.sh
  1
  ^
fibonacci.sh:1: Can't execute '1': No such file or directory
  " $[i+=_,_=i-_]{a..C}
                  ^~~~
fibonacci.sh:3: Mismatched cases in character range
powerkitten@blaze-pc:~/Desktop/osh-testing$ cat fizzbuzz.sh
for((;i++<100;)){
FizzBuzz$i
echo ${_:i%3?i%5?8:4:0:i%15?4:8}
}
powerkitten@blaze-pc:~/Desktop/osh-testing$ osh fizzbuzz.sh
  for((;i++<100;)){
                  ^
fizzbuzz.sh:1: Invalid word after for expression
powerkitten@blaze-pc:~/Desktop/osh-testing$ cat leap.sh
set {1804..2400..4}
history -p ${@%?[^04]00}
powerkitten@blaze-pc:~/Desktop/osh-testing$ osh leap.sh
  history -p ${@%?[^04]00}
          ^~
leap.sh:2: 'history' doesn't accept flag '-p'
powerkitten@blaze-pc:~/Desktop/osh-testing$ cat niven.sh
for((;i++<100;)){((i%(i%10+i/10)))||echo $i;}
powerkitten@blaze-pc:~/Desktop/osh-testing$ osh niven.sh
  for((;i++<100;)){((i%(i%10+i/10)))||echo $i;}
                  ^
niven.sh:1: Invalid word after for expression
powerkitten@blaze-pc:~/Desktop/osh-testing$ cat prime.sh
for((;j=i++<97;)){
let j+=i%{1..97}?0:1,j^3||echo $i
}
powerkitten@blaze-pc:~/Desktop/osh-testing$ osh prime.sh
  for((;j=i++<97;)){
                   ^
prime.sh:1: Invalid word after for expression
andychu commented 5 years ago

Yes the for { is #304 . That shouldn't be too hard to implement.

history -p should also be implemented

andychu commented 5 years ago

Hm and the fibonacci script tickles a strictness errors I recognize... I'm not sure yet if it justifies relaxing them.

dylanaraps commented 5 years ago

If you'd like to avoid the "Command not found" error you can just add the : builtin to the first line.

: 1
history -p $[{1..31},i+=_,_=i-_]

As it's a "golf" to save 2 characters I drop the builtin (stderr is ignored in golfs).

wertercatt commented 5 years ago

The popular interactive shell enhancement thefuck also breaks when osh usage is attempted. (/bin/bash demonstration included for clarity.)

powerkitten@blaze-pc:~$ osh
osh$ apt install example
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
osh$ fuck
                  export TF_HISTORY=$(fc -ln -10);
                                      ^~
[ eval at line 1 of '/home/powerkitten/.config/oil/oshrc' ]:1: Can't execute 'fc': No such file or directory
                  history -s $TF_CMD;
                          ^~
[ eval at line 1 of '/home/powerkitten/.config/oil/oshrc' ]:1: 'history' doesn't accept flag '-s'
osh$ exit
powerkitten@blaze-pc:~$ apt install example
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
powerkitten@blaze-pc:~$ fuck
sudo apt install example [enter/↑/↓/ctrl+c]
wertercatt commented 5 years ago

https://gist.github.com/wertercatt/5820fe28f9e107a55d6f28929c919c8f hl.sh is used when booting GoldSrc engine games (ie: Half-Life 1) on Steam.

wertercatt commented 5 years ago

Decided to test random shell scripts in my /usr/bin/ as well, and found one from a popular program that completely crashes osh! https://gist.github.com/wertercatt/a84e728ef0d87e6478f2ce072f36d715 /usr/bin/VBox, used by Oracle VM VirtualBox gave me a pretty nasty looking crash.

powerkitten@blaze-pc:/usr/bin$ osh VBox
Traceback (most recent call last):
  File "/home/andy/git/oilshell/oil/bin/oil.py", line 846, in _cpython_main_hook
  File "/home/andy/git/oilshell/oil/bin/oil.py", line 827, in main
  File "/home/andy/git/oilshell/oil/bin/oil.py", line 797, in AppBundleMain
  File "/home/andy/git/oilshell/oil/bin/oil.py", line 610, in ShellMain
  File "/home/andy/git/oilshell/oil/core/main_loop.py", line 168, in Batch
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1188, in ExecuteAndCatch
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1136, in _Execute
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1021, in _Dispatch
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1157, in _ExecuteList
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1136, in _Execute
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1077, in _Dispatch
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1136, in _Execute
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 1077, in _Dispatch
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 538, in RunSimpleCommand
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 324, in _RunBuiltin
  File "/home/andy/git/oilshell/oil/osh/cmd_exec.py", line 276, in _RunBuiltinAndRaise
  File "/home/andy/git/oilshell/oil/osh/builtin_bracket.py", line 213, in __call__
  File "/home/andy/git/oilshell/oil/osh/expr_eval.py", line 566, in Eval
  File "/home/andy/git/oilshell/oil/osh/expr_eval.py", line 628, in Eval
NotImplementedError: <Id_t BoolUnary_c 275>
FATAL: couldn't import from app bundle '/usr/local/bin/oil.ovm' (1)
Stripping the oil.ovm binary may cause this error.
See https://github.com/oilshell/oil/issues/47
andychu commented 5 years ago

@wertercatt Thanks, I just filed that as #361 and fixed it.

These bug reports are great! Keep them coming. They all seem valid, so please open new issues rather than adding to this one. They will be less likely to get lost that way!

(And yes help is appreciated :) I will help anyone get up to speed with the code, which is plain Python with a bunch of build steps that are automated.)

andychu commented 4 years ago

FWIW I went back and tried some of the codegolf ones:

The others don't work because:

FizzBuzz doesn't work for me in bash 4.3 or 4.4?

$ _deps/spec-bin/bash -c 'for((;i++<100;)){
FizzBuzz$i
echo ${_:i%3?i%5?8:4:0:i%15?4:8}
}'

...
_deps/spec-bin/bash: line 1: FizzBuzz99: command not found
Fizz
_deps/spec-bin/bash: line 1: FizzBuzz100: command not found
Buzz

The last two releases have fixed a ton of stuff due to ble.sh and some reports from Crestwave! I think #679 to run neofetch is getting close.

Crestwave commented 4 years ago

FizzBuzz doesn't work for me in bash 4.3 or 4.4?

What's wrong with the output? If you're referring to the error messages, only stdout is counted in the golf and the stderr junk is intended. You can prepend it with : to stop it.

The prime numbers program works with let as let(){ IFS=,;return $((!($*)));} and eval_unsafe_arith.

The divisors program doesn't work even with workarounds for the lack of $_ and $[ as OSH doesn't support mixed case in brace expansion ({a..C}).

The FizzBuzz program doesn't work even with workarounds for the lack of $_; it seems to be because OSH doesn't support nested ternary:

osh$ echo $((1?2?3:4:5))
  echo $((1?2?3:4:5))
             ^
[ interactive ]:1: Parser expected Id.Arith_Colon, got Id.Arith_QMark
$ echo $((1?2?3:4:5))
3

bash, dash, ksh, mrsh, yash, and zsh seem to support it, but ash doesn't seem to either

andychu commented 4 years ago

Ah OK thanks for the diagnosis. Even after 4 years of writing a shell those programs give me headaches :-) But they are useful for finding bugs -- I filed an issue for what I suspect is a parsing error.

Keep the reports coming -- I think Oil is converging, given that

And C++ translation is working out too, so Oil will eventually be fast.

andychu commented 4 years ago

FWIW there are more than 17 issues fixed for the next release, so it should be a significant jump in compatibility. Thanks for all the reports!

https://github.com/oilshell/oil/labels/pending-release

andychu commented 4 years ago

@dylanaraps Thanks for accepting the patch to neofetch! I mentioned this on the latest release:

http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html (if you haven't seen it already)

I tried neofetch from head under OSH 0.8.pre4 and it works great. It's slow as expected, but we've been making good progress on that !

As mentioned in the release notes I'm curious how much of neofetch runs on these invocations, e.g. is it closer to 1K lines or 9K lines? So #687 for coverage has been on my mind. And it would also be useful for ble.sh.

I'm also looking for shell experts to help test and design Oil. For example I want to fix -e "once and for all" #709, so if you have any feedback there, I'm interested. (Although it looks like neofetch doesn't use -e, so maybe it's not one of your concerns?)

original neofetch issue: #679 

$ time osh neofetch
            .-/+oossssoo+/-.               andy@lisa 
        `:+ssssssssssssssssss+:`           --------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 16.04.2 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Host: XPS 8700 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 4.15.0-96-generic 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 8 days, 13 hours, 23 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 2883 (dpkg), 33 (nix-user) 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 4.3.48 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1920x1200 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   DE: Unity 7.4.0 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM: Compiz 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   WM Theme: Ambiance 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Theme: Ambiance [GTK2/3] 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    Icons: ubuntu-mono-dark [GTK2/3] 
  +sssssssssdmydMMMMMMMMddddyssssssss+     CPU: Intel i7-4790 (8) @ 4.000GHz 
   /ssssssssssshdmNNNNmyNMMMMhssssss/      GPU: NVIDIA GeForce GT 730 
    .ossssssssssssssssssdMMMNysssso.       Memory: 12230MiB / 24055MiB 
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`                                   
            .-/+oossssoo+/-.                                       

real    0m6.977s
user    0m5.819s
sys     0m1.161s

compared with time bash nofetch

...
real    0m0.323s
user    0m0.244s
sys     0m0.068s
danyspin97 commented 4 years ago

Another big program to test is Paludis, Exherbo package manager. It is written in C++ but rely heavily on Bash scripts for the package format. According to tokei, its codebase contains ~16000 lines of shell code.