Raw, but working, ChrysaLisp argument processor

[FrankC01]

From a crawl, walk, run perspective - I'm still looking for the kneepads to start crawling with.

Here is the heart and test driver - Gist link

I haven't hooked up the 'help' yet, been focused on class spikes and getting something working. I moved away from the raw 'class' for the moment but may return on that another time...

Narrow test, I've since fixed boundary conditions:

tstap cat this is just stdio input tstap cat -f cmd/echo.cmd

[FrankC01]

Latest with '-h' working can be found HERE

The test driver is below the main file. It is a sort of echo and/or cat command for demonstration only.

If it passes 'good enough' as a beginning I can do a pull request. But I've questions:

1. I've got the argparse functions in file with .lisp extension but are you thinking .inc? As .inc is a assembler de-facto for include files but not so much for lisp I'm wondering intent.
2. I truly am a bit confused over the difference between defun-bind func ...) and (defun func ...)

[vygr]

1. I generally use .lisp to mean a file that can be launched as a process, and .inc to mean a file that is normally included by (import) or (include), so the later sometimes contains assembler and sometimes Lisp source.

2. the only difference between defun-bind and defun is to do with the pre-binding or pre-evaluation of function symbols.

So the symbol '+ in (+ 1 2 3) would normal, at runtime, be evaluated and be found to be bound to the #class/num/lisp_add asembly function. And that assembler function is then called. That runtime eval of the '+ symbol will happen every time, even if this is in a loop, it's standard Lisp procedure 'late binding'.

Now late binding is a cool thing, lets you change the function that might get called even during run time, even during the running of the code calling whatever is bound to '+. But it does incur the search for the binding of '+ (and every other function symbol) and for a function binding that is never going to change this can be a waste of time. So defun-bind is a version of the defun macro that calls (bind-fun) on the function body at macro expansion time, this finds any function symbol and pre-binds them to there lambda or #vp functions, replacing the function symbol with the lambda or #vp reference. Eval still gets call on these at runtime, but these are no longer symbols and like "" self evaluate with no symbol search, so much faster.

eg.

(defun frank (a b c) (+ a ( b c))) (lambda (a b c) (+ a ( b c)))

whereas:

(defun-bind frank (a b c) (+ a (* b c))) (lambda (a b c) (#class/num/lisp_add a (#class/num/lisp_mul b c)))

Chris

[FrankC01]

So, for consistency, argparse is meant to be included (i.e. not a app/cmd/etc.). I consider argparse, options, etc. to be a kind of library module in a sense. I'll rename the file extension to .inc

For argparse all the functions would benefit from the pre-binding.

Let me know if you think (I have very thick skin so don't hold back) the latest code passes the sniff test and is worthy of adding to get going with a pull request.

[vygr]

Also worth knowing is that (defun-bind) can only pre-bind function symbols that are bound at the time it calls (bind-fun), so you can't pre-bind to a function you havn't declared yet, so try declaring functions in order so that later functions can take full advantage of the pre-binding if your using it.

AND this is a potential gotcha to watch out for with (bind-fun). It will bind ALL symbols that can be seen to be bound to a function ! And not just in the traditional first place in a list position ! So if you are using a variable that has the same name as a function beware.

I add the .Lisp_syntax highlight file for VSCode to the repo, and this really helps avoid this problem (plus gives you nice looking editing in VSCode). It a drop in replacement for Mattn's Lisp syntax highlighter. Install Mattn Lisp extention from the extentions menu in VSCode, then once it installed, copy the .Lisp_syntax file in the repo over into your ~/.vscode/extention/Mattn/Lisp.../syntax folder.

If you then associate the ChrysaLisp .lisp, .vp and .inc files with it, you will get MUCH nicer enviroment to edit in. Plus VSCode is a pretty spiffy editor as well.

[FrankC01]

You'll have to pry sublime text from my cold dead hands first. I could work on syntax highlighter if it really is bothersome but I'm fine.

I'll go ahead with the pull request if you are too busy to review the code now and you can then comment on the pull. I will provide some degree of documentation as well...

[vygr]

Just had a quick look through the code and it certainly does look clean ! No CamelCase so that's a bonus ;) only got to fight over using tabs till the first blackspace than spaces afterward and we're good :)

[vygr]

(defmacro last (seq)
  ; (last seq) -> el
  ; Retrieves last element in sequence
  ; Example: (last '(1 2 3)) -> 3
  `(elem (dec (length ,seq)) ,seq))

Can just do (elem -2 seq)

Element index as - means backwards from the end of the sequences. -1 is the element just off the end. The reason for this is so you can do the folowing:

(slice 1 -2 "<abc>") -> "abc"

(each!), (some!) etc also all let you use - indexes in this manner.

[vygr]

(defun reverse (inlist)
  ; (reverse inlist)
  ; Reverses contents of list
  (if (/= (length inlist) 0)
    (reduce-rev (lambda (acc el)
                (push acc el)) inlist (list))))

I do believe I have (map-rev) in boot.inc...

Maybe just:

(map-rev (lambda (_) _) '(1 2 3))

I'm trying to think if there is a built in VP asm function that will just return a single input, like a unity type of function...so the lambda here isn't required.

[FrankC01]

Yep, map-rev is available. I've removed mapcat btw, not used

Is there an id function or macro which just (defmacro id (_),_)`

Which then works with (tested in REPL): (map-rev id '(1 2 3))

[vygr]

(defun isarg? (arg)
  ; (isarg? value)
  ; tests if argument type which has
  ; '-x' or, by definition, '--xname' prefix
  (cond ((starts-with "-" arg) t) (t nil)))

Can just (starts-with "-" arg), maybe just as a macro ?

[vygr]

Looking very nice Frank ! Look forward to the pull request.

[FrankC01]

re: arg? of course it can just use `(starts-with "-" ,arg) ! I'm still going through a few "DOH!" moments and adding what isn't needed, now if I just study these calls in REPL more often...

[vygr]

Yep, map-rev is available. I've removed mapcat btw, not used

Is there an id function or macro which just (defmacro id (_),_)`

Which then works with (tested in REPL): (map-rev id '(1 2 3))

Careful with using a macro here ! Although you might be getting away with it by accident. It not the done thing to call a macro rather than a function. (It just so happen in ChrysaLisp under the hood they are exactly the same thing, but some purist would castigate me for that...)

But yeah, I was trying to think of a built in #vp level function that could be used as a 'id'... I did think 'quote' would work, but then realised I break a Lisp rule here that means it won't. Still we can allways add a built in VP level (id) function if we want.

[vygr]

OK, this is the one ;)

(map-rev progn '(1 2 3))

(defun-bind reverse (l) (map-rev progn l))

[FrankC01]

Great, I'll put it back to a function and use progn

Now for the documentation.

I'm thinking of starting a "TERMINAL.md" (not exhaustive but a starting point) something like:

Overview - Talks to it being the current 'shell' equivalent, available in both GUI and TUI. Call out this being the place to REPL from

Inventory

• Commands - Overview of commands (e.g. echo, etc.)
• Libraries? Supporting? Something befitting the ".inc" stuff with places for options argparse and asm

Examples of passing to commands and argument processing (from 'hello-world' using options to more complex argument processing using argparse: options.inc - etc argparse.inc - etc

[vygr]

Definitely !

[vygr]

I was going to try adding a phase to the 'make doc' command that scanned all the .lisp files in cmd directory and pulled out the 'help' info. Created a CMD.md file with that.

Fancy that as the next thing to have a go at ?

I'm happy if during that you restructure the current 'make.lisp' to use your new argparse code. I'm not pleased at the moment that you can't type:

'make all platforms boot doc syms'

[FrankC01]

Here is the roadmap I was thinking for argparse

1. Type validation and auto-conversion (e.g. nums, file existence, etc.)
2. Textual representation of argparse tree to 'read' to construct the argparse data model ready for parsing. Similar to what you have for options so command developers can get up and running quickly.

And then a revisit on the 'make doc' or similar?

[vygr]

That sounds like a plan.

For file existence you can just use (age xyz), if you get 0, it not there.

[vygr]

(defmacro last (seq)
  ; (last seq) -> el
  ; Retrieves last element in sequence
  ; Example: (last '(1 2 3)) -> 3
  `(elem (dec (length ,seq)) ,seq))

Can just do (elem -2 seq)

Element index as - means backwards from the end of the sequences. -1 is the element just off the end. The reason for this is so you can do the folowing:

(slice 1 -2 "<abc>") -> "abc"

(each!), (some!) etc also all let you use - indexes in this manner.

These macros are looking like a good addition to the boot,inc sequences section ?

first, second, last, rest.

And: Done.

[FrankC01]

Yeah, they are pretty 'standard' when it comes to functional sequence stuff IMHO.

I'll try another day to push list functions semantic alignment. My perspective is to stay reasonably true to Lisp to attract more contribution.

[vygr]

(defun walk (self arglist &optional res)
  ; (walk-arguments self arglist)
  (defq result (opt res (list)))
  (while (/= (length arglist) 0)
         (defq current (stack-peek arglist))
         (cond
           ((defq arg_object (container-for self current arguments argument))
              (if arg_object
                  (setq arglist (consume-argument self arg_object arglist result))
                  (throw "Unrecognized argument " current)))
           ((defq cmd_object (container-for self current commands command))
              (if cmd_object
                (setq arglist (consume-command self cmd_object arglist result))
                (throw "Unrecognized command " current)))
           (t (push result (stack-pop arglist)))))
  result)

into the nit picking now, but worth knowing.

Here you pass in an optional 'res' but then create a new variable result with the value of res if not nil otherwise (list).

Don't create that new variable, it makes all symbol lookups slower as it makes the hash table, for this function, bigger. Just use the same optional variable name but assign its default with (setd), set default.

(setd res (list))

(setd) like (setq) can take multiple pairs.

(setd a 5 b 7 c 8)

[vygr]

Also, and this is definitely worth knowing. In the cond statement cases you are defq'ing a variable and then testing with an if to see if it was nil ?

It can't be nil by the time you get to the if test ? The cond case would never happen if the defq resulted in a nil result ! Now if you had multiple defq vars the last one may not be nil and so you might need to test with the none last one with if, but in this specific example your ifs are redundant, and so are the throws !

So I suspect if your really meaning that part of the validation is to do those throws, this needs some thought ?

in this case if arg_object, for example was nil, you would end up doing the next case or maybe even:

(t (push result (stack-pop arglist)))))

[FrankC01]

For now I have this:

(defun walk (self arglist &optional res)
  ; (walk-arguments self arglist)
  (defq result (opt res (list)))
  (while (/= (length arglist) 0)
         (defq
          current (stack-peek arglist)
          arg_object (container-for self current arguments argument)
          cmd_object (container-for self current commands command))
         (cond
          (arg_object
            (setq arglist (consume-argument self arg_object arglist result)))
           (cmd_object
            (setq arglist (consume-command self cmd_object arglist result)))
           ((isarg? current)
            (throw "Unrecognized flag " current))
           (t (push result (stack-pop arglist)))))
  result)

I could have left the defq in the condition test but I'm thinking ahead to mutual exclusions, exceptions, etc... the if was redundant...

What is the difference between setd and setq is it all about the scope/env?

[vygr]

(setd) is just a macro that's all about doing some (opt) statements for you.

I've passed in some optional vars, and if they are nil I'd want them to default to these things.

Your still creating that extra var result ? Maybe just change the optional 'res to optional 'result.

[vygr]

Also be sure you want the existence of arg_object to override the existence of cmd_object, because that cond is going to prioritise arg_object.

[FrankC01]

I changed it as you described now. Convenient that...

[FrankC01]

I can convert the current cmd/*.lisp to use argparse or we keep options and argparse (lite weight and heavier lifting) letting developer of cmd decide. Thoughts?

Also, in terminal is it hard coded to search only the cmd directory when entering at terminal prompt?

[vygr]

I see no issue with providing both ? There could be arguments for a single technique though If that helps us with the auto gen of documentation?

The pipe.inc file in apps/terminal assumes the commands in the pipe are to be wrapped with “cmd/“ and “.lisp” but we could change that to only do so if there is no path “/“ char found ?

[vygr]

The GUI context aware tab completions currently assume they are looking for command position extensions in “cmd/“ with a “.lisp” file extension.

[FrankC01]

Yes, both makes sense as well. I wanted to start taking a look at terminal in more detail as there also seems like an opportunity for a few things.

[vygr]

In case you didn't notice on the pull request comment. I added self evaluation of any symbol beginning with : to the repl_eval function. No need to use (def:) anymore, just freely use :abc and things will just work.

[vygr]

I'm pretty committed to the : keyword symbol idea now, I'm going to be making some global source changes to reflect this new ability. Starting off with the VP class structures...

[FrankC01]

I did see the comment on keywords and will be verifying it with argparse tonight or in the morning.