Configuration Dialect

[andychu]

Summary: YAML alternative / COWS: configuration in Oil without Side effects

This is part of #631: Oil Blocks

Rough sketch of what we need:

• evalblock() function
  • it can do PushTemp() and PopTemp() ?
• blockstring() function? I noticed this need for source/hut travis inline shell
• the COWS evaluator uses shopt --set dynamic_scope ?
  • if evalblock() is creating new scope, I guess we need this?
  • I think we need to limit scope at the proc level, but not within blocks? hmm...
• shopt --set parse_equals -- this is NO LONGER a synonym for const
  • I think it's like a setvar without a corresponding var ??? Inside a block.
  • but is there a difference at the top level ???

Example of evalblock

proc p(name, &myblock) {
  var d = evalblock(myblock)
  setvar d->name = name

  json write :d
  setvar _cows = d     # does this make sense?
}

proc localized(@names) {
  for name in @names {
    p $name {
       age = 30
       ratio = 3.0
    }
  }
}

# top level configuration

p myname {
  age = 30
  ratio = 0.1
}
p other {
  age = 42
  ratio = 0.2
}

# macro-ized configuration

localized 

How does scope work?

[andychu]

Config Dialect Use cases -- MOVED TO https://github.com/oilshell/oil/wiki/Config-Dialect

[andychu]

Naming idea: evalcows()

Just to recap:

• push temp -- its own scope
• Null executor. How to do this? Not core/pure.py, but core/cows.py?
• shopt parse_equals
• shopt dynamic_scope
• ALSO, we need a way to define procs to evaluate with

idea:

# eval-config.oil

proc server(name, &b) {
  push-context route {  # define route
    var d = evalblock(b)
  }
}

dialect --name myproject -- server otherproc {
  source $1  # the filename ?  Does it give good errors?
}

# config.cows

use dialect myproject  # ensure it's running under this dialect

server www.example.com {
  root = '/'
  port = 80
}

Use cases

• generating Ninja. This may require string safety #938
• generating JSON for travis, sourcehut, etc. Because they accept YAML.

[andychu]

Idea for evalblock() semantics

• only consts are exported. So x = 42, but not var y = 43.
• location info is also exported somehow? So if you validate the value, you can point to an error

Does value_t need a span_id then? I think I would find that useful in some cases, like the dynamic const check? If it's already defined?

[andychu]

Instead of evalblock(), we have #1025

[andychu]

From Zulip, an idea to make blocks syntactic sugar for dicts, and then gracefully allow more validation with procs:

https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/Using.20oil.20to.20represent.20complex.20data

---

However I'd still like to define the valid first words, so you could have

eval ('myfile.oil', procs=%(package user))

And then 'package' and 'user' could be ways to define dicts, like

package cppunit

user bob

would turn into

{ "type": "package",
  "name": "cppunit"
}

{ "type": "user",
  "name": "bob"
}

[andychu]

New minimal idea (in NOTES.txt)

• existing shvar PATH='' to disable externals -- fix hash -d bug
• push-procs to bring in procs from old namespace to new
  • it creates a brand new namespace; no "looking up the stack"
• parse_file(path Str) Block
• eval_to_dict (block command_t) -- see evaluation rules below
  • top level variables are allowed
  • and then you have procedural abstraction with by invoking the procs you pushed
• do the procs need a reference to the root of the config? Yes I think so?
  • _ append(_config->users, 'bob') -- we'll probably need a variant to append or CREATE _config->users
  • but how does _config get passed to procs? Is it a magic variable? what about nested evaluation of configs?
• default actions: does {"type": "package", "name": "arg1"} transformations

Is that it?

Issue for later:

• what if one config file wants to "source" another?
  • I think this can be coordinated by the evaluator. It can declare it, and then the evaluator can be configured to do parse_file() and so forth.

[andychu]

Evaluation rules:

• Top level, and most blocks: EvalBlockToNamespace, which is command_t -> Dict[Str, Obj]
• UPPER blocks like TASK: just leave it alone -- type command_t
  • and then provide a block_to_string() function

[andychu]

More isolation issues:

• redirects must be disabled
• source builtin shouldn't be able to open random files!
• The parse_config() function too!

Doh so we need to restrict the builtins dictionary ...

Or we can rely on containers to do this?

It is an argument for always forking a process ... But then how do we get procs from one place to the other?

I think we don't want push-procs. I think we want to have an inherent source before the config file evaluation

bin/oven --source mydialect.oil -- myconfig.oil

It's a bit more complicated that way but probably more general and secure ... we don't want a blacklisting approach; whitelisting is better.

Although push-procs is whitelisting of procs -- it's not whitelisting of builtins like source though

[andychu]

Although Executor holds both self.builtins and self.procs, so we could have

push (builtins = %(echo printf json), procs=%(foo bar)) {
  shvar PATH='' {
    const d = eval_to_dict(myblock)
  }
}

[andychu]

First cut of hay works! After a huge commit

https://github.com/oilshell/oil/commit/47c972391969bf59d9d421abad81bfdae74019e2

More:

• [x] Implement printing of code
  • or maybe put it in JSON automatically ... child.block and child.code_str
  • shell_code_str or oil_code_str depending on string or block
• [x] add location info somehow, at least of the first word / type
• [x] make sure evaluation happens with oil:all -- important! Test it out
• [x] parse hierarchical names, and respect them -- hay define package/user package/resources
• [x] fix bug with incomplete hay
• [x] make NAME optional? why not. I guess make it null?
• [x] add ARGS if there are 2 or more
• [x] enforce either ARGS or block
• [x] hay eval :result { } -- for INLINE hay
  • maybe shopt --set hay_eval or something? The haynode builtin can set it
  • _hay() is only for the interactive shell -- although slightly redundant with hay pp result. Maybe undocumented?
  • [x] hay names only visible in hay eval
  • [x] clear result after hay eval

Related issues

• 1021 -- important for config file serialization

• 838

• 1082 -- because block_as_str() must do it

• 1024

  • check if a block is a valid arg

• 496 -- suffixes on numeric literals

• 1099

[andychu]

• Usability issue: can we do something about parse_equals inside .oil files? #1139

[andychu]

• [x] inline hay -- refine the 3 contexts, only 2 of which are sandboxed
• [x] add basic sandboxing against "reasonable" code, but don't make any guarantees within the same process
• [x] improve error messages? it assumes an external command if it's not defined
  • could be solved by "outside hay"
  • improved inside hay eval with its basic sandboxing, not sure about outside
• [ ] Create an end-to-end demo, especially of process-based config file evaluation
• [ ] both source and parse_hay() and need to close the file

More:

• [ ] more docs
  • [ ] put it in a tour of Oil. Ruby-like blocks are a major feature
  • [ ] Variable declaration mutation scope -- needs a section on hay evaluation. It does ctx_Temp which is like FOO=bar. Also hay eval does.
  • [ ] doc/hay.md should have a quick start
• [ ] make it statically typed and translatable. This is hard and related to #636

After some usage and feedback:

• [ ] hay bind Package/TASK myproc_TASK ?
  • so you can run validators? They receive @ARGV and a b Block ?
• [ ] Implement shopt --set sandbox_{redirect,extern,source} ? (under group sandbox:all ?)
  • internally this is self.fd_state, self.ext_prog, self.job_state, etc.
  • or is this just part of hay eval ?

[bar-g]

Had a look into the new doc/hay skeleton, and as you have mentioned DSLs there, and I still had your comment about the common push / pop issue in mind (https://github.com/oilshell/oil/issues/1059#issuecomment-1151734946), I felt maybe there is some common solution there to find.

Ruby/crystal seems to have two types of blocks for DSLs:

The difference between using do ... end and { ... } is that do ... end binds to the left-most call, while { ... } binds to the right-most call. The reason for this is to allow creating Domain Specific Languages (DSLs) using do ... end to have them be read as plain English: [...] You wouldn't want the above to be: [...] (https://crystal-lang.org/reference/1.4/syntax_and_semantics/blocks_and_procs.html)

Maybe some kind of general apply/with/to/do or and? -block might help for DSLs, too?

[andychu]

Yeah there is definitely possibility of things like that ... though I want people to use it first, and actually hit the problem, before we add complexity!

I'm pretty happy with the simple first pass, and hopefully we'll get a bunch of feedback on it

This feature is a bit more obscure than others, because I think platforms / maintainers will generally use it, not end users. i.e. it's for people building something like sourcehut, Github Actions, systemd, podman/Docker, etc.

[bar-g]

Sure, a universal/left/right block application idea/rules will likely need some time to ripe anyway.

I had thought hay could be nice even just for exposing simplest config files to users, thanks to the sandboxing, safe to "source", no need to write parsing code? But I didn't grasp the whole usage/practical picture just yet.

[bar-g]

Something in your latest "Sketches of YSH Features" blog and your intention for ysh to be well-extensible by users, made me remember my comments here. You wrote:

[proc] Arguments are bound left to right, with splats like ... picking up the rest, except for the block argument. It's always the last argument, so it's neither positional or named.

This sounds as if it wouldn't allow to pass multiple blocks.

I'm not sure if that would still allow for exposing/allowing for more generic block usage, i.e. having left and right binding blocks (for better readable DSLs https://github.com/oilshell/oil/issues/951#issuecomment-1152953400), as well as making use of hay-based config in ysh itself (e.g. no fuzz readable redirection syntax https://github.com/oilshell/oil/issues/1059#issuecomment-1137481928), and generic apply {...} to {...} block application or "operators".

With code+data in the blocks (i.e. hay) could that maybe be a useful basis to allow for implementing, e.g. said matrix algebra operators etc., as extensions as ysh libraries instead of compiled code?

[andychu]

I think you can only pass on block in the {} style, as the last arg

But there's nothing stopping you from passing multiple blocks the expression style, like

myproc (block1, block2)

It's mainly a syntactic limitation

But yeah when we implement that, we should have some test cases for it