New behavior for existing standard-shell commands?

[bhakta0007]

I am exploring marcel and I see that existing commands have different behavior in marcel. Most of us have ingrained muscle memory with regular shell commands (like I always do "ls -ltr").

Is this by design or are these bugs that need to be fixed?

for e.g.

(with bash)

(venv) bhakta@blr:~/dev/marcel$ ls -fr
.  ..  .git  .gitignore  .idea  COPYING  LICENSE  README.md  bin  experiments  marcel  notes  setup.py  test  venv
(venv) bhakta@blr:~/dev/marcel$ ./bin/marcel

(with marcel)

/home/bhakta/dev/marcel> ls -fr | less
Missing filename ("less --help" for help)

/home/bhakta/dev/marcel>
/home/bhakta/dev/marcel> ls -fr > x
/home/bhakta/dev/marcel> vim x

<hangs - I had to ctrl-c>
^C

Ctrl-c did not come out, I had to kill the window

/home/bhakta/dev/marcel> ls -ltr
Operator ls: Unknown flag -ltr
/home/bhakta/dev/marcel>

[geophile]

My thought was that I want to keep using ls (and ps, head, tail, and a few others), but I don't mind changes in behavior. In Linux, flags mix behavior (what is reported) and formatting (how it is reported). Linux's ps flags, in particular, are a complete mess, inheriting flags and even incompatible flag syntax from different species of Unix. I thought a clean slate for ps and ls was a better way to go, and resulted in better compatibility with other marcel concepts: piping streams of objects; a separate operator (out) writing to the console instead of adding that logic to other operators; and the "rendering" of objects, e.g. marcel.object.file.File.render_full().

However, I do recognize the downside of this choice which is that ls (for example) doesn't work as it does in bash. Other possible designs include:

• Just use a different command, e.g. marcel would use list instead of ls, and ls would invoke the Linux command. This leads to confusion because ls and list would "both list files", but ls produces textual output while list would produce File objects, and this could lead to confusion.

• Provide an escape, e.g. ^ls to get the Linux escape.(If the goal is to preserve "muscle memory" this is not a good choice.) By the way, an escape is present right now. If you do "bash ls -rlt" that passes "ls -rlt" to bash. The bash operator simply passes all of its arguments to bash.

What are your thoughts? It is early enough that nobody is relying on marcel, so fundamental changes like this are still possible.

As for ctrl-C not killing your command. That is a regression. Working on a fix now.

Other items you raised:

• vi/vim: Marcel and vim both rely on stdin/stdout/stderr, so they require special handling. Marcel recognizes "vi" and handles it specially. For now, "bash -i vim" should work for you. In the next release, you will be able to just run "vim".

• "ls -fr > x" does not write to a file x, it writes to a variable x. You can examine the contents by running the command "x >". To write to a file, you want "ls -fr | out -f x".

I expect to have a new release fixing the ctrl-c and vim issues in a day or so.

[geophile]

1. • Ctrl-c is now working again.
2. • vim should now work

3. • was working as intended, but the documentation on > and >> needs to be improved.

4. • Invoking less from marcel still does not work.

Not closing this due to items 3 and 4.

[bhakta0007]

I do see your point on new behavior versus muscle memory and supporting existing ways of doing things. I do think that muscle-memory-friendly products would help in initial adoption and building a community of users.

I am not sure what your design goals are for marcel are but I have always felt the need of some of the below features in a modern-day programmer/power-user friendly shell:

• A shell that can be one step ahead of the user and help in most basic tasks that an advanced user does.
  • An Interactive programmable (or automatable/scriptable) back-end engine
  • AI-driven response framework - like can the shell think about what the user wants to do next and help in doing this?
  • Ability to hibernate (save context, close, and open it back again)
  • Ability to store multiple result buffers (aka output of previous commands).
  • Ability to access the stored buffers and run processing functions on them (e.g. I take a snapshot of counters and after a while another snapshot - can I see the diff?)
  • Ability to auto-complete commands (like how the google has it) - based on what the user has been doing.
  • Ability to auto-parse outputs and convert into structured output (text output to JSON)
  • Fully programmable and manageable from external API
  • Creation of an ecosystem of useful packages that someone can download/install. Like PIP or NPM does today. Imagine I can supply a template-parser for a popular command that can be made available and anyone can download and they will be able to use it.
  • Complex Output formatting - Say I want to run more than one command, get the outputs, and throw out a single combined parsed output for the combination command.

(of course, this list is not complete - there are a lot of interesting things that can make life easier)

[geophile]

Thank you for your thoughts. Some of what you ask about is already present, and I like many of the other ideas. (I really need to step up the documentation. A tutorial is probably a good thing to provide.)

• There is a need for a programmer-friendly (or power-user) shell. A shell that can be one step ahead of the user and help in most basic tasks that an advanced user does.

I'm open to suggestions.

• An Interactive programmable (or automatable/scriptable) back-end engine

Not sure what you mean here, can you expand on this?

• AI-driven response framework - like can the shell think about what the user wants to do next and help in doing this?

There are certainly repetitive patterns in shell usage, e.g. yesterday I ran make, cd .., runtests repeatedly. It should be feasible to design something to support this and reduce repetition. But AI? (I want to avoid a Clippy-like "You seem to be writing a business letter! Can I help?")

• Ability to hibernate (save context, close, and open it back again)

I've been thinking about this. There is a file string command history, of course, (~/.marcel_history), but things get messy when there are concurrent sessions. (Currently, one sessions's history will overwrite another's.) To avoid those difficulties you would need some concept of a named session. And then it would be easy to record the marcel namespace on exit. Perhaps there is one default session, but you can create named ones?

• Ability to store multiple result buffers (aka output of previous commands).

That exists, although the syntax is probably not what you expect. If you do "ls -fr > x", that saves the result of "ls -fr" in the environment variable x. "ls -fr >> x" appends to x. Marcel pipes streams of tuples, and what you are doing here is gathering up the stream and placing it in a variable, (it's just the stream stored as a list).

You can then read out from the saved stream, turning the contents of x back into a stream, e.g. "x > select (...) | map (...) | ..."

• Ability to access the stored buffers and run processing functions on them (e.g. I take a snapshot of counters and after a while another snapshot - can I see the diff?)

See above. You can process these stored streams using set operations (union, intersection, difference), and join (which is like a relational join). Note that difference is set difference, not a source management diff operation. But that certainly would be useful. I will start thinking about a design for a diff operator.

• Ability to auto-complete commands (like how the google has it) - based on what the user has been doing.

I'll have to investigate. Marcel has pretty traditional auto-complete for operators, arguments, and options.

• Ability to auto-parse outputs and convert into structured output (text output to JSON)

CSV support is there (see read --csv and --tsv). I've been thinking about JSON, but that's a bit tricky. Here's the problem: The "data model" of shell usage is typically tabular. You have a stream of files or processes, for example. These types have attributes, so you can turn a stream of files, for example, into a stream of (file name, mode, user, group, size). That's tabular. JSON is hierarchical. I don't really see how piping JSON from one command to another would work in practice. Can you provide an example that would help me see this?

• Fully programmable and manageable from external API

That already exists (for Python)! Take a look at this: https://www.marceltheshell.org/scripting. Or do you mean a remote API?

• Creation of an ecosystem of useful packages that someone can download/install. Like PIP or NPM does today. Imagine I can supply a template-parser for a popular command that can be made available and anyone can download and they will be able to use it.

I completely agree that this would be a useful thing to have. It's on my mind, and I'll start on it once there is some interest in marcel.

• Complex Output formatting - Say I want to run more than one command, get the outputs, and throw out a single combined parsed output for the combination command.

I think this may be present in part. As mentioned above, you can use > and >>, set operations, and join to build up collections of data in memory. You can then pass all of this to the out operator, which supports Python formatting, e.g. "ls -fr | map (f: (f.name, f.size)) | out '{}: {}'". You can also write arbitrary Python functions in ~/.marcel.py (or import them), and use those to format strings for output, e.g. "ls -fr | map (f: format_file(f))", where format_file is a functioh you provide.

About compatibility ("muscle memory"): Do you think, for example, that it would be useful to have a -l flag that was ignored? (Or maybe printing a warning?) Or more generally, providing a way to approximate the behavior of commonly used flags? (I have no idea how "commonly used" would be defined for ps which has a huge assortment of flags.)

[geophile]

One important additional point, on set ops: The set ops do not guarantee order. For the scenario you describe, order is important, and of course, the traditional diff does preserve order.

[bhakta0007]

<meaning to respond to this thread - been tied up with things, I will reply back on this>