search

holzschu / a-shell

A terminal for iOS, with multiple windows
BSD 3-Clause "New" or "Revised" License
2.6k stars 116 forks source link

gawk #571

Open Wobbegong3 opened 1 year ago

Wobbegong3 commented 1 year ago

I have written dozens of gawk scripts that run on Windows, Ubuntu, macOS, and iSH, but they won't run on a-Shell (on iPadOS 16.3.1) because the awk interpreter that is available there does not support functions like asorti(), systime(), and strftime(). Has anyone compiled Gnu awk for use on a-Shell? iSH is awesome (if a bit glitchy), but I would love to have a POSIX-compliant gawk because a-Shell is so much faster.

holzschu commented 1 year ago

For the time being, the position of the FSF is that the GPL is incompatible with AppStore distribution. Which is why we have the original awk rather than gawk.

I'll try to compile gawk to WebAssembly.

Wobbegong3 commented 1 year ago

Great! Thank you.

On Mon, Feb 27, 2023 at 2:39 PM Nicolas Holzschuch @.***> wrote:

For the time being, the position of the FSF is that the GPL is incompatible with AppStore distribution. Which is why we have the original awk rather than gawk.

I'll try to compile gawk to WebAssembly.

— Reply to this email directly, view it on GitHub https://github.com/holzschu/a-shell/issues/571#issuecomment-1446952112, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUZGAQZEH2F6CI2OCWXTZJLWZT7FLANCNFSM6AAAAAAVJZ646A . You are receiving this because you authored the thread.Message ID: @.***>

Wobbegong3 commented 1 year ago

Perhaps I could somehow install it myself if you (or someone else) were to build it. There were many tools missing from iSH (Alpine) and I was able to install them with commands like these: apk add bash apk add gawk apk add unzip apk add zip

These were not part of the AppStore distribution, so maybe this is a way to get around this restriction.

holzschu commented 1 year ago

To answer the second question: iSH uses x86 code and an interpreter. They can install anything that has been compiled for Linux on x86. We use a combination of Arm64 binaries (distributed in the AppStore distribution) and WebAssembly binaries (that can be installed with pkg). It's more difficult to compile for WebAssembly, so the list of packages is a lot smaller. But I'll give it a try for gawk.

Wobbegong3 commented 1 year ago

Is the gawk source code in C or C++? If so, is there any reason why I couldn't compile gawk myself within a-Shell? If not, I could use some help with the commands to run (I only just discovered a-Shell yesterday, when I did an app store search of iSH to see when it was last updated, so I'm still learning my way around).

In any case, it looks like I may be able to get by with standard awk, Instead of using strftime(). I can determine date and time with code like this:

cmd = "date \"+%Y/%m/%d %T\"" while (cmd | getline > 0) datetime = $0 close(cmd)

And I can define my own asorti() function like this:

function asorti(arg1, arg2) { n = 0 for (indx in arg1) arg2[++n] = indx for (j = 1; j < n; ++j) { for (i = j + 1; i <= n; ++i) { if (arg2[i] >= arg2[j]) continue temp = arg2[j] arg2[j] = arg2[i] arg2[i] = temp } } return n }

However, I may run into other issues that I can't work around (I haven't looked at some of my gawk scripts in over a year), so I would still like to know if compiling gawk from source is a possibility.

On Tue, Feb 28, 2023 at 1:46 AM Nicolas Holzschuch @.***> wrote:

To answer the second question: iSH uses x86 code and an interpreter. They can install anything that has been compiled for Linux on x86. We use a combination of Arm64 binaries (distributed in the AppStore distribution) and WebAssembly binaries (that can be installed with pkg). It's more difficult to compile for WebAssembly, so the list of packages is a lot smaller. But I'll give it a try for gawk.

— Reply to this email directly, view it on GitHub https://github.com/holzschu/a-shell/issues/571#issuecomment-1447668350, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUZGAQ2FGM2VG6YCVY3AVLLWZWNK7ANCNFSM6AAAAAAVJZ646A . You are receiving this because you authored the thread.Message ID: @.***>

holzschu commented 1 year ago

You can compile gawk yourself (I'm trying to do it as well). From experience, automatic configuration on the iPad (sh ./configure) might not work, so you would have to edit the config.h file, then run make.

update: I've run into a very common problem: gawk uses setjmp(), which is not available in WebAssembly. I'll see if I can work around that.

Wobbegong3 commented 1 year ago

I just discovered that output redirection within awk appears to not be supported (it's not mentioned as supported in the awk man page, anyway). And this code:

output = "test.txt" print "this is some text" >output

generates an "awk: can't open file test.txt" error. (I think that it's peculiar that awk knows that the code wants it to open a file, yet it can't do it.)

That code works fine with gawk on Windows, macOS, Ubuntu (under WSL), and iSH (Alpine).

I don't believe that there's a way I can work around this, so the original awk won't work for me.

On Tue, Feb 28, 2023 at 2:41 AM Nicolas Holzschuch @.***> wrote:

You can compile gawk yourself (I'm trying to do it as well). From experience, automatic configuration on the iPad (sh ./configure) might not work, so you would have to edit the config.h file, then run make.

— Reply to this email directly, view it on GitHub https://github.com/holzschu/a-shell/issues/571#issuecomment-1447715661, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUZGAQZFHYRAUWT7MBYMWN3WZWTZXANCNFSM6AAAAAAVJZ646A . You are receiving this because you authored the thread.Message ID: @.***>

holzschu commented 1 year ago

I'll have a look. Your example gives me a syntax error, so can you provide a full script? It could be that the "cannot open" error is related to file permissions.

update: the first time you run it with redirection, awk crashes (segmentation fault). When it crashes, it doesn't go through the file closing. The following times, it says "cannot open file" because the file is already open.

holzschu commented 1 year ago

Here is gawk, compiled to WebAssembly. It has the limitations of WebAssembly: no forks, no pipes (and definitely no pipes to commands inside awk). But it could fit your needs. To install, unzip and move gawk to ~/Documents/bin.

gawk.zip

Wobbegong3 commented 1 year ago

a-Shell doesn't appear to have a "script" command for capturing terminal output, which is inconvenient (it's pretty standard for Linux, and very useful for situations like this). With it I could have captured an "ls -l" command, two "cat" commands, and the execution and the output. Then I would have sftp-ed that text file to this Windows 10 machine so I could email it to you. So I'm going to have to type this in manually:

[~/Documents]$ cat test.awk END { output = "test.txt" print "this is some text" >output } [~/Documents]$ cat test

!/bin/sh

awk -f test.awk /ve/null [~/Documents]$ chmod 755 test [~/Documents]$ ./test segmentation fault

I created test and test.awk in this directory (using vim), so there shouldn't be any problem with the script creating a file there as well.

On Tue, Feb 28, 2023 at 12:07 PM Nicolas Holzschuch < @.***> wrote:

I'll have a look. Your example gives me a syntax error, so can you provide a full script? It could be that the "cannot open" error is related to file permissions.

— Reply to this email directly, view it on GitHub https://github.com/holzschu/a-shell/issues/571#issuecomment-1448541782, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUZGAQ7ZJ4T5J5SX5MVUZGLWZYWFHANCNFSM6AAAAAAVJZ646A . You are receiving this because you authored the thread.Message ID: @.***>

Wobbegong3 commented 1 year ago

I can't figure out how to get that gawk.zip file to a place where I can unzip it.

I'm reading this in gmail in a Chrome browser on Windows 10, but I also see it at https://github.com/holzschu/a-shell/issues/571 (but I'm not very experience with git -- I worked in IT for 50 years but retired 5 years ago, which was before git was much of a thing).

On Tue, Feb 28, 2023 at 3:06 PM Nicolas Holzschuch @.***> wrote:

Here is gawk, compiled to WebAssembly. It has the limitations of WebAssembly: no forks, no pipes (and definitely no pipes to commands inside awk). But it could fit your needs. To install, unzip and move gawk to ~/Documents/bin.

gawk.zip https://github.com/holzschu/a-shell/files/10854332/gawk.zip

— Reply to this email directly, view it on GitHub https://github.com/holzschu/a-shell/issues/571#issuecomment-1448809383, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUZGAQYOFBWW72FPPB4MROLWZZLERANCNFSM6AAAAAAVJZ646A . You are receiving this because you authored the thread.Message ID: @.***>

holzschu commented 1 year ago

That is a good question. I see several process:

Or (depending on how you like to do things):

gawk is a direct compilation of the source code to WebAssembly, minus the parts that WebAssembly cannot do. I expect it to have a few issues, so I’m looking forward to your tests. Thanks to your tests, I can confirm that awk has an issue with redirection. I’m going to see if I can fix it.

Wobbegong3 commented 1 year ago

Nicolas,

OK, I got this far, but there's no unzip or pig command in my a-Shell -- they both generate a "command not found" message, and neither command appears in the output of "help -l". No problem -- I used sftp to copy gawk.zip to a macOS machine, did the unzip there, and then copied the gawk file back and put it in my local bin directory. That worked fine and "gawk --version" tells that it is version 5.2.1 (the latest).

You did say no pipes, and I realize that I have several scripts that have code like this:

cmd = "env" while (cmd | getline > 0) {

} close(cmd) That causes an error, but I can work around that limitation by instead doing this: tempfile = "temp.txt" system("env " tempfile) while (getline 0) { } close(tempfile) And I there's some other problem that's causing a script (or a-Shell itself) to completely lock up, doesn't get me back to the shell prompt -- the only thing I can do to get a shell prompt back is stop and restart the a-Shell app, which is annoying. Is there an equivalent to ? And is there any documentation for the shell? For instance, I figured out what most of the icons at the bottom of the screen do, but what are this [image: image.png] and these [image: image.png] for? I really think that the best solution for me would be to compile gawk from source myself. I have the source for 5.2.1 that I downloaded from https://www.gnu.org/software/gawk (which is probably where you got your copy), and it does appear to have been written in C (there are lots of .c files anyway). But now what? I suspect that the make files use gcc and a-Shell has clang. Are the command-line options identical? What other issues am I likely to encounter? It's been a very long time since I've compiled anything. Peter On Wed, Mar 1, 2023 at 1:58 AM Nicolas Holzschuch ***@***.***> wrote: > That is a good question. I see several process: > > - open the #571 page > on your iPad. > - long press on the “gawk.zip” link > - scroll down do “Copy link” > - open a-Shell > - curl -OL [paste link here] > - unzip gawk.zip (maybe you’ll need pig instal zip first) > - mv gawk ~/Documents/bin/ > > Or (depending on how you like to do things): > > - open the #571 page > on your iPad > - click on the “gawk.zip” link > - it will ask if you want to download it, say yes. > - open the Files app > - navigate to the Downloads folder > - click on the “gawk.zip” file. It will unzip. > - long press on the new “gawk” file > - scroll down to “Move” > - navigate to the a-Shell folder > - click on “Copy”. It will copy (not move) gawk to the a-Shell folder. > - open a-Shell. > - mv gawk ~/Documents/bin/ (create bin first if it doesn’t exist yet). > > gawk is a direct compilation of the source code to WebAssembly, minus the > parts that WebAssembly cannot do. I expect it to have a few issues, so I’m > looking forward to your tests. Thanks to your tests, I can confirm that > awk has an issue with redirection. I’m going to see if I can fix it. > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
holzschu commented 1 year ago
holzschu commented 1 year ago

The TestFlight version of a-Shell mini: https://testflight.apple.com/join/REdHww5C has the updated awk, that doesn't crash with redirection. So now you have two different awks that you can use.

Wobbegong3 commented 1 year ago

Nicolas,

I'm on the verge of giving up entirely on trying to use a-Shell instead of iSH -- there's just way too many glitches.

I write all of my scripts so the exact same file can be run on every platform -- maintaining different versions for each platform is just too much trouble, especially when I make an enhancement (or bug fix) on one platform and have to propagate those changes to all other platforms). I do this with a determine_OS_version() function (attached, with a bit of code to invoke it) that figures out (by interpreting the output of the "set" command) what OS it is running on and then (elsewhere in the script) I can have the code do whatever is appropriate (e.g., on Windows I delete a file with "del" rather than "rm", and file path names use "\" rather than "/" for directory delimiters). This works fine for Windows, Linux, WSL, macOS, and iSH.

a-Shell doesn't have a "set" command but it does have "env" (which Windows doesn't have), so I modified my determine_OS_version() function to deal with that. But, for some unknown reason, awk on a-Shell does not run "env" reliably, so I had to make it run several times in a loop until it was successful. Here's a tiny wrapper script to execute the attached AWK script:

!/bin/sh

awk -f osver.awk /dev/null

and here's the sort of output I get maybe once every five times I run it on a-Shell:

set >.temp.txt reading .temp.txt set: command not found env >.temp.txt 0 records read from .temp.txt for try 1 env >.temp.txt 21 records read from .temp.txt for try 2 env >.temp.txt 0 records read from .temp.txt for try 3 env >.temp.txt 0 records read from .temp.txt for try 4 env >.temp.txt 21 records read from .temp.txt for try 5 env >.temp.txt 0 records read from .temp.txt for try 6 env >.temp.txt 21 records read from .temp.txt for try 7 env >.temp.txt 0 records read from .temp.txt for try 8 env >.temp.txt TERM_PROGRAM=a-Shell 65 records read from .temp.txt for try 9 rm -f .temp.txt

OS: a-Shell

Note that it took 9 attempts to be successful. That's just unacceptable.

If I try to run it with the gawk you built for me (just change "awk" to "gawk" in the wrapper script), all I get for output is this:

set >.temp.txt set: command not found

I have no idea why I don't get more, and I'm exhausted from trying to figure it out.

Note that quite a while back I had written an awk script that grew to over 10,000 lines that managed all of the email addresses and email forwarding for a company of over 2000 employees. It was eventually replaced with MS Exchange when the company realized that I was going to retire soon. And long before that I wrote code that analyzed the performance of the Apollo Guidance Computer (and mostly the performance of the navigator operating it) so that it could get the astronauts safely back from the moon. Back then an IBM mainframe computer was the size of a refrigerator and it had 4MB of main memory in a separate cabinet the size of a washing machine and it had 9 disk 20MB drives that each were the size of a microwave oven. How things have changed in just 55 years!

Peter

On Thu, Mar 2, 2023 at 9:33 AM Nicolas Holzschuch @.***> wrote:

The TestFlight version of a-Shell mini: https://testflight.apple.com/join/REdHww5C has the updated awk, that doesn't crash with redirection. So now you have two different awks that you can use.

— Reply to this email directly, view it on GitHub https://github.com/holzschu/a-shell/issues/571#issuecomment-1451963565, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUZGAQ5LYVMIOZALNUMOLTLW2CVVHANCNFSM6AAAAAAVJZ646A . You are receiving this because you authored the thread.Message ID: @.***>

holzschu commented 1 year ago

I don't see the osver.awk file that you said was attached?

Wobbegong3 commented 1 year ago

Regarding SSH and SFTP: How do I get it to use PKI so I don't have to keep entering my password when I want to transfer files to/from my Mac? I ran ssh-keygen on the Mac and then copied the public key (in id_rsa.pub) to ~/Documents/.ssh/id_rsa as "man ssh" states, and I still get asked for my password. I also tried putting it in ~/Documents/.ssh/authorized_keys (which works elsewhere) and that didn't work either.

Message ID: @.***>

holzschu commented 1 year ago

This is more of an ssh question than an a-Shell question, so you would have more help looking at a ssh tutorial.

Basically, authorizations are decided on the receiving host. Here, you are connecting from a-Shell to your Mac, so your Mac is the receiving host. So you need to have both private and public keys in ~/Documents/.ssh/ in a-Shell. Then you need to add the public key to the authorized keys on the Mac.

Wobbegong3 commented 1 year ago

Nicolas,

If "the position of the FSF is that the GPL is incompatible with AppStore distribution" is really true, then why is this allowed?: https://macappstore.org/gawk

I just remembered that that's how I installed gawk on macOS Ventura three months ago:

/Users/pcv ~# uname -a Darwin Baldwin.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103 arm64 /Users/pcv ~# which gawk /opt/homebrew/bin/gawk /Users/pcv ~# gawk --version GNU Awk 5.2.1, API 3.2, (GNU MPFR 4.1.0-p13, GNU MP 6.2.1) Copyright (C) 1989, 1991-2022 Free Software Foundation.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

If the FSF doesn't have a problem with that, then it shouldn't have a problem with you building an ARM64 version of gawk for a-Shell.

Does the MAC App Store have different rules than the iOS/iPadOS App Store?

Also, you built "original-awk". Could you build nawk or mawk (which do not belong to the FSF)?

Peter

On Mon, Feb 27, 2023 at 2:39 PM Nicolas Holzschuch @.***> wrote:

For the time being, the position of the FSF is that the GPL is incompatible with AppStore distribution. Which is why we have the original awk rather than gawk.

...

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free.www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

holzschu commented 1 year ago

Does the MAC App Store have different rules than the iOS/iPadOS App Store?

Yes, the FSF has issues specifically with the iOS App Store, see here for example: https://www.fsf.org/news/2010-05-app-store-compliance

Regarding nawk and mawk, there are several challenges. Like locating the source code, checking the license, then cross-compiling and then checking for bugs.