metaeducation / ren-c

Library for embedding a Rebol interpreter into C codebases
GNU Lesser General Public License v3.0
128 stars 27 forks source link

Tune DELIMIT and PARSE handling of BLANK!, ISSUE! #1077

Closed hostilefork closed 4 years ago

hostilefork commented 4 years ago

This tweaks the handling of BLANK! to allow it to be used at "source level" in dialects, while prescribing it as a synonym for NULL if it is fetched from evaluation or a variable.

Being able to discern a "source" blank from a fetched one is thus important, and the previous commit's tweak is used to move in steps that allow blanks after comments to be discerned as source or not:

>> spaced ["a" _ comment <b> _ "c"]
== "a  c"

>> spaced ["a" blank comment <b> blank "c"]
== "a c"   ; acts as if you'd written `spaced ["a" "c"]`

DELIMIT is also modified to use tight spacing for ISSUE!, which is an aspect of the transition to a merged ISSUE!/CHAR! datatype. This integrates well with the source-level spacing feature, for clear results that don't require mixing SPACED and UNSPACED calls, and thus can work cleanly with PRINT:

project: 'Ren-C
bad-thing: "Software Complexity"
new?: does [project <> 'Rebol]

>> print [#<< project #>> _ {The} (if new? 'NEW) {War On} bad-thing]
<<Ren-C>> The NEW War On Software Complexity

This also changes PARSE to embrace the feature of source-level BLANK! representing a synonym for SKIP:

>> did parse [1 <two> "three"] [integer! _ text!]
== #[true]

While fetched blanks act as NULL-equivalents:

>> subrule: _

>> did parse [1 <two>] [integer! subrule tag!]
== #[true]

It was actually the case that PARSE was not allowing NULL parse rules when it should (due to not being sync'd with the void-as-misspelled change). So routines like REWORD were using BLANK! for nulling out rules made via reduce. Changing to NULL was the easy fix, but the usage of REDUCE made it fail in a tricky way...so this provided the critical momentum for REDUCE to mutate NULL into VOID! by default. Predicates will allow usage of reduce .try [...] to get blanks from nulls, or reduce . [...] to use the identity function and dissolve the nulls.

hostilefork commented 4 years ago

This moves to try and make a happy compromise about the need for a reified NULL. Please see the theory behind it:

Forum: "Treat BLANK!s from variables or evaluation like NULL"

This pushes the BLANK!-and-space equivalency into a fairly narrow band of source-level dialecting. That pretty much rules out things like:

>> to char! blank
== #" "

 >> append "abc" blank
 == "abc "

Realizing that is no longer something that's going to be "a thing" helped clear the air on the fact that PARSE wouldn't be obligated to treat blank as space when parsing strings (for example). It could choose to do so, but this commit takes the different direction of having it mean "any value can go here", as an abbreviation for SKIP.

All of this implies that we now know:

 >> append "abc" blank
 == "abc"

SO @rgchris, it's up to you to decide the fate of APPEND on arrays w.r.t. blank...which would also be the default behavior of KEEP w.r.t. blank. You're the person who was most vocal about blanks in DELIMIT being nothing, so the ball is in your court to say what you want for this scenario:

data: [<one> _ [] "four"]

collect [
    for-each item data [
        keep item
    ]
 ]

Do you get [<one> "four"] or do you get [<one> _ "four"]? e.g. do you need an /ONLY to actually append a BLANK! to an array?

NULL would seem to provide a nice disambiguation here. But if append "abc" blank is going to ignore the blank and be "abc", then the default behavior for APPEND may make sense to match. Really I think looking at usage of scenarios like the COLLECT above has to inform it...

If BLOCK! is the universal container and hence has unique splicing default ability, is BLANK! the universal reified NULL and thus also has a unique behavior? An answer to this is needed!