Closed ncannasse closed 5 years ago
~ seems rather perl-esque, and there is a risk of ~ being used in the markup. I guess the problem with less common symbols is more hassle to type ...
var myQuotedString = ¿<code $specific>${ if( flag ) ¿<div>something</div>¿ }</code>¿ ;
var myQuotedString = <¿code $specific>${ if( flag ) <¿div>something</div¿> }</code¿> ;
and there is a risk of ~ being used in the markup
You'd need to have at least ~<
or >~
for it to be an issue
Fairly easy to spot but in a large project if it was macro generated it might be very hard to debug as you may not even know that the embedded mark up contained them.
An alternative with the same concept, it feels more readable and with an additional character it's going to reduce hits?
<-< >->
var myQuotedString = <-<code $specific>${ if( flag ) <-<div>something</div>-> }</code>-> ;
I wouldn't mind to just keep <thing>
and sacrifice selfclosing tags if that makes everything complicated. I don't like the ~<thing/>~
syntax (nor <-<
and I don't know how to write ¿
on my keyboard)
As alternative I would love to propose (<node
and />)
so you can do (<thing/>)
Also to solve the newlines issue, we could reconsider https://github.com/HaxeFoundation/haxe-evolution/pull/49
As alternative I would love to propose
(<node
and/>)
so you can do(<thing/>)
That's also my preference because that looks more natural and closer to JSX (sorry for being specific here because I want to convert all JS users to haxe)
Though we need to communicate clearly that no space can exist between (
and <
so would
var empty: MyDSL = (<>);
be empty one? I presume you occasonally would need to define empty ones. Would sad face be a problem?
(<test>:(<test/>)
Do we really need tag names in these cases? Can we shorten required syntax to just (< whatever here >)
?
I think the concept is that if it's xml style then...
var xml: XML = (<xml>stuff<xml/>);
And for other cases like perhaps YAML then you might have
var yaml: YAML = (<
item:
- method: UPDATE
where: &FREE_ITEMS
- Portable Hole
- Light Feather
SellPrice: 0
BuyPrice: 0
npc:
- method: MERGE
merge-from: {name: General Goods Vendor}
items: *FREE_ITEMS
>);
Well that was my understanding. I like (< >) but not sure if this could clash with some use cases and domain languages, that would be my worry. By the way I like to have the ';' at the end I think it feels more consistant?
Three chars is likely to be safer if they are picked well but they will look uglier and maybe harder to remember.
var xml: XML = (_<xml>stuff<xml/>_);
var special: Special = (_< special >_);
I kind of like (<
but the tilde is disgusting.
(<
looks acceptable but i can see people being confused like hell why parens are required and why you cannot have spaces between them (that said maybe we could actually allow whitespace between paren and <
Well spaces should be allowed between (
and <
.
I presume that allowing spaces between is likely to make the risk of failure higher if there is no tag.
(<emoji> :( </emoji>)
I do think that '(<' is much easier to read than '~<' but long term they seem quite fragile usless there is an option to change the term used by the compiler which would allow '(<' but then allow user to swap for say '[[[<' if they wanted. I don't see how you can allow spaces without numerous parse failures. It's just too easy for real user text to have (< and >) within it and even more so with ( < and > ).
( <someUserText> keep value low ( <5 ) </someUserText> )
( < keep value low ( <5 ) > )
Also allowing spaces could increase compile time for complex project, although that may depend on implementation.
-D openTag '~<'
-D closingTag '>~'
So default could be '(<' but could be adjusted, this would allow alternatives, but then that would not work with different haxelib libraries, and never get used anyway.
What's the reason for not using inline metadata to hint to the compiler that a dsl expression/string will follow?
@skial Even if you mark the beginning of a DSL expression with metadata, the parser would need to know where the end is.
Which would be provided by the dsl's EOF "event".
(Note that I have no idea how haxe's parser could let a macro parser take care of parsing the source code from position X until it (the macro parser) determines it has finished, or if that would be possible at all)
@Aurel300 I'm assuming, because I lack knowledge of how the compiler works, so please correct me, but doesnt the compiler expect certain closing characters in different contexts, calls require a closing )
, arrays a closing ]
etc, so in these cases can't @:dsl(<node att></node>)
be expected. @:dsl(
marks the opening, )
marks the end.
For return expressions or variables var html = @:dsl <node att=""></node>;
relax the bracket rule, if possible 🤷 and allow the semicolon to be one of the dsl closing characters?
Basically doesnt metadata, @:dsl(...)
, already solve this largely? I know there will be examples that break this or make the compiler choke, but thats likely to the case with whatever start and end token is chosen, no?
@skial Your use of metadata is no different from other opening tags, because this has to be solved at the parser level. So you are essentially suggesting @:dsl(
as the opening tag and )
as the closing tag. As an example @:dsl(<node value="()">)
breaks this, because the parser can't tell the first )
is not meant to close the DSL expression.
So, my two cents here is that the opening and closing need not be entirely unique, but it does need to be escapeable. In essence, what we are building here (as far as the compiler is concerned) is a new, potentially multi-line, string type. Just like we do with other strings, we need to be able to embed the delimiter inside of the string. So we need an escaping mechanism. No need to invent one, we have one, if we just follow the current constant string rules. And we don't need to invent new embedded code rules, either. (e.g. 'Some string with ${ some_code("With and embedded " + 'string.') }'
). Right?
So, how about continuing to use the current string delimiters and adding a multi-line option to the initial quote. A leading character before a quote is currently a syntax error, but we could treat it like a constant modifier (a la 2454654378u
), just placing the modifier at the beginning of the constant, rather than at the end. Something like this:
doSomething( x'
<div>
\'Are you alive?\' <br/>
I am ${ live ? 'alive' : 'dead' }.
</div>
');
The only real difference from standard string syntax here is that the embedded newlines are included.
Escaping is standard (if ungainly) syntax. Engineers already know how to do it. The trouble comes when you want to "validate" it. But we weren't doing that anyway, just requiring an XML-ish syntax so that we could parse it. It would be much better if we didn't have to care about the markup format at all, other than as we already do for strings.
A string modifier is also easily added to both of our string types (single and double-quoted).
And, if you really, really, wanted to validate the string contents, there is no reason we can't extend the modifiers (h for HTML, x for XML, m for simple multi-line).
@EricBishton Regular Haxe strings already allow newlines.
Why again it's that such a problem detecting self-closing tags? Too XML-specific?
Because the parser is not aware of what is inside it, it only checks for opening and closing tags, so this <div</div>
is fine for it, as is <div *anything* except "/ + >" or "< + / + div + >" </div>
.
Things have to be balanced, so <comp head=<div></div>></comp>
is fine. However, self-closing tags share closing tokens, so <comp head=<div /> />
will break.
This would be fine with a proper parser (made compatible with both domkit, xml, jsx, angular, etc.), but Nicolas wants the syntax to be very permissive (which won't be able to do more than check opening/closing tokens)
My suggestion is to introduce semantics to the current markup syntax:
<node attr={/* haxe code in braces */}/>
(I am sure @back2dos will be more capable in coming up a more well-thought spec for that)
Then, introduce another generic syntax for DSL which uses matching number of backquotes as open/close delimiter. Backquote is chosen as it has a developed recognition thanks to markdown. Correct me if I am wrong, this "matching number" mechanism should solve any escaping problems (as I have already utilized it when writing the following code block, which is wrapped by a four-backquote).
var yaml =
yaml:
foo
}
Recap my reasoning from slack: The world have been using xml-ish syntax (HTML) to describe UI for decades and it is definitely going to live for some more decades. So supporting them with semantics in Haxe is nothing near "trendy" in my opinion, it is just pragmatic. In fact the "generic DSL" syntax is to solve another, different, problem, which is called "escaping strings"
Defining a flavor of jsx as @kevinresol suggests would make IDE completion more straight forward too I would think.
So like this, but normally with new lines?
var markDown = ```` ``` haxe //test ``` ````;
what would happen with
var markDown = ````` ``` haxe //test ``` ````;
and
var markDown = ```` ``` haxe //test ``` `````;
Well you can error but haxe may not realize where it errors, it does not always know where a { has been missed by the user, and an error has to be very clear as you may easily miss typing the number of backquotes.
Generally three ` are treated like
in markdown so if you copied pasted some mark down you can't be sure it only used three.
Using ```` seems pretty much the same arguments against that apply to """ in the orginal brief?
Since @Aurel300 pointed out that we already support multi-line strings, and our current single-quoted strings are already re-entrant (my example above works), why do we need this at all? Are single-quoted strings really that ugly?
The only thing that requires escaping in a single-quoted string is single-quotes, which are not all that common in xML and frameworks (that I've seen anyway -- I'm not a web front-end guy). If validation and completion (e.g. tooling) are the driving factor, then let the tooling figure out how to do that. That doesn't have to be part of the language, and the Haxe compiler should never be involved in completion of DSL contents.
If the user wants validation and you want to help the tooling out, adding a comment or a compiler-ignored metadata tag is harmless enough. But generalized data can't be validated in a generalized way anyway, so if this was part of the reasoning for the feature, then there are competing requirements.
As far as using <xml
and </xml>
are concerned, that flies in the face of generalized data. And it forces non-xml to look like this:
<somename
Here is some random documentation.
It uses multiple lines, but isn't XML.
</somename>
How is that any easier to read than the following? (bash style)
<endtoken
Here is some random documentation.
It uses multiple lines, but isn't XML.
endtoken
It isn't. And they're both ugly. ```...```
, [[[...]]]
, <<<...>>>
are all better looking. The thing that this latter version give us is that the user can define the end token to be something that doesn't occur in the text. So, there's less reason to escape. At least in the bash version, the begin and end tokens are removed, but in our not-XML, they are not.
The idea of using the content of the string (or part of the content, as in ~<
) itself as the delineator is specious. It forces the content to have a certain format. In the current incarnation, only things that start like XML can be used, so we've already limited the value of the new string type. We've also determined that it doesn't work for XML in practice (thus, this RFC). I understand the motivation to get rid of delimiters, but it's just not practical in the general case. If we want to be more specific and actually require the content to be XML-compliant, then we can actually enforce those rules and the issues of mismatched tags can all go away -- and the tools can provide help. (To be clear, I'm not advocating for this.)
If we're going to think about re-entrancy, and having to type some sort of delimiter anyway, what about overloading ${...}
, so that if it occurs in code sections (e.g. also inside of 'string data ${code ${string}...}
the compiler just changes state from code to string contexts? Still ugly:
var myQuotedString = ${<code $specific>${ if( flag ) ${<div>something</div>} }</code>}
or non-web uses:
var myQuotedString = ${Are you alive? I am${ if( flag ) ${ dead} }.}
var myQuotedString2 = ${"Are you alive?" she said. I replied, "I am${ if( flag ) ' dead' }."}
But, the non-web uses are better served by the current strings:
var myQuotedString = 'Are you alive? I am${ if( flag ) ' dead' }.'
var myQuotedString2 = '"Are you alive?" she said. I replied, "I am${ if( flag ) ' dead' }."'
If we really need a new opening and closing delimiter, then the ~<
and >~
look fine to me, but they should not be part of the content of the string.
var myQuotedString = ~<<code $specific>${ if( flag ) ~<<div>something</div>>~ }</code>>~
(Also '>~' is currently two operators, and it becomes hard to detect from 1000 >~ 0001
.)
(By the way, we could keep this out of the compiler -- and the language -- completely by allowing a pre-processing tool to run across the sources and encode [fix] them to current syntax. (Is that within the current capability of macros?) An issue with this approach is how it would work with compiler caching.)
@EricBishton because a lot of people would like literal XMLish. We already use multiline strings and it's a poor experience, and while I appreciate that generalising DSLs is a valid goal, adding complicated control characters essentially nullifies what we have in Haxe 4 now.
@elsassph (and @benmerckx, @back2dos), care to quantify that? A "poor experience" is entirely subjective. What problems, exactly, are trying to be solved? By reading the original proposal from Kevin, it's primarily escaping quotes.
So, if we need a leading character to address the issues (and @Simn should appreciate this), how about we make the leading/following character sequences '<
and >'
for embedded XML-ish with interpolated/embedded code, and "<
and >"
for embedded XML-ish that should not be interpolated. And the opening <
and closing >
remain part of the string, while the leading/trailing '
or "
do not.
(Yes! So far, that sounds exactly like what we had [pre-4.0] already! But read on, because things change.)
However, because the pain that people wish to eliminate is the need to escape things (e.g. use \'
and \\
) in their XML-ish, we add these extra rules to strings that use the XML-ish delineators:
1) Importantly, there can be no space between the quote mark and the <
or >
characters in either delineator sequence.
2) Between the two delineators, the backslash character \
will be itself, it will no longer be used to delineate Unicode characters or unprintable characters (e.g. newlines, tabs). XML-style escaping such as ሴ
, (which the Haxe parser can effectively ignore) will be used for those.
3) Between the two delineators, a quote character that would otherwise terminate a non-XML-ish string will neither terminate the string nor require an escape character of its own; only the matching terminating sequence will cause the end of the string to be found. (only >'
terminates interpolated XML-ish and only >"
terminates non-interpolated XML-ish.)
4) The Haxe compiler will not enforce any formatting of the XML-ish content itself. Tools, such as IDEs, can apply formatting rules if they wish.
5) Any embedded use of a terminating delineator within an XML-ish string will have to use XML's escaping syntax: >'
.
@EricBishton The big issue I see with that is that it could (silently!) break existing strings that happen to start + end with those delimiters. It also breaks the principle of least surprise, IMO - imagine somebody randomly stumbling over this "no-quoting" behavior with very specific strings when he's new to Haxe...
Maybe something like @
for C#'s verbatim strings would have to be prefixed to avoid that. But at that point, why not use something like backticks instead of single quotes.
@gama11 Well, backticks compared to single-quotes are hard to differentiate on screen, depending upon your font. But I have no other objection to them.
AFA breaking existing strings, you are correct. I don't know how prevalent the issue is, but I suspect it's not so bad. Primarily people will end up with extra characters in their output if the string is already 'balanced'. If the strings weren't balanced, the user would end up with compiler errors, most likely (Unterminated string). And, the compiler could scan the string (possibly controlled by a #define) for old-style backslash sequences and warn. The IDE can certainly do that. (I'll bet tokentree can too for vscode support.) As far as breaking changes are concerned, 4.0 is a major release where we are allowed some lattitude.
AFA the principle of least surprise, I think that new operators are more of a surprise than familiar syntax, but that's subjective. (If you want to talk about surprise, though, the fact that you currently don't have to escape single-quotes inside of an interpolated code section within a string threw me.)
The thing that I like about my latest proposal is that we introduce two types (interpolated and non-interpolated) of strings with one rule. Also, overloading the current operators allows other operators to be kept for other purposes.
@EricBishton for one it potentially breaks existing code as @Gama11 pointed out. And silent breaking change on a common construct (string) is bad, really bad.
Also, I believe that (please correct me if I am wrong) that the re-entrant problem just can't be solved without having some semantics in the markup. e.g.
Nicolas proposal:
~<div child={~<div attr="</div>~"/>~}></div>~
(note that the first occurrence of </div>~'
terminates the markup.)
Eric proposal:
'<div child={'<div/>'}/>'
(note that the first occurrence of >'
terminates the markup.)
I believe (again please correct me) this could only be solved if the parser is aware of re-entrancy (a.k.a. back into Haxe expressions mode). and my suggestion is:
2. With interpolation
<node attr={/* haxe code in braces */}/>
I don't know how prevalent the issue is, but I suspect it's not so bad.
IMO it doesn't matter so much how prevalent the issue is, since there is definitely code like that out there. The keyword here is silently breaking, which can lead to bugs that are incredibly hard to find. As far as I'm aware, silent breakage is quite rare with the changes in Haxe 4, but that's getting a bit off-topic. What matters is that this sort of breakage should be avoided if at all possible.
A warning for these sorts of strings doesn't seem feasible either, because that would only work during the Haxe 3 -> 4 transition. You wouldn't want to have them afterwards / for legitimate '<
strings.
AFA the principle of least surprise, I think that new operators are more of a surprise than familiar syntax.
I assume by operator you mean a new kind of literal? Anyway, I have to disagree strongly here, there's much less of a chance for anybody to use a different string kind accidentally, which would definitely happen to some people with regular single or double quotes.
@kevinresol
Also, I believe that (please correct me if I am wrong) that the re-entrant problem just can't be solved without having some semantics in the markup. e.g.
Re-entrancy within strings is solved when you enter an interpolated code section. With the double-quoted string case, you are correct: because there is no interpolation, the first >~
(or >"
) terminates a string. But with single-quotes, the interpolated section (everything between ${
and }
) is a separate parsing context and does not require escaping. (Note: Both Nicolas and myself still require the $
before brackets to introduce interpolation. Those rules do not change, unlike your proposal.) This is what the compiler already does, and I propose that this not change; in fact, I am depending on it.
var myQuotedString = '<code $specific>${ if( flag ) '<div>something</div>' }</code>'
@Gama11 and @kevinresol AFA silently breaking strings. I agree that is a bad thing. I think that a compiler warning is far from silent. And, there is no reason to remove such a warning after the 3.0-4.0 transition, because developers can always write new strings without realizing the semantics, and should be warned (at their option).
@Gama11
Using @
(or X
or any letter) as a string prefix to turn off escaping is just fine in my book. In that case, we don't need to break -- or reinterpret -- current strings. Only new strings would be affected. However, I wouldn't take C#'s semantics, which changes the escaping rules (there are fewer), and would look really odd in XML. Instead, I would propose that the prefix change the escaping rules to XML's escaping rules. The problem of an adequate terminator still exists, though.
There are a couple of other ways to solve it, though:
1) Require a prefix-plus-delineator option, like @'<
, >'
an @"<
, >"
.
2) Require an opt-in command-line argument, in which case a developer is selecting the new semantics (and the requirements and responsibilities thereof).
The second allows a (slightly) cleaner syntax in the case of re-entrancy.
(using -XmlString on the cl)
var myQuotedString = '<code $specific>${ if( flag ) '<div>something</div>' }</code>'
vs
var myQuotedString = @'<code $specific>${ if( flag ) @'<div>something</div>' }</code>'
Or, allow both. In the case where the prefix exists and the #define is set, it is the same as only one being set.
I think I'm missing part of the issue here; if the xml-ish syntax is the most desired one here, could the issue be solved by making the parser smarter so it can understand self-closing tags?
Considering some common DSL languages is perhaps useful, as it will refocus on specific use cases and perhaps provide a source of unit tests for any solution. https://tomassetti.me/domain-specific-languages/
Edited above to stay focused: Considering some common DSL languages is perhaps useful, as it will refocus on specific use cases and perhaps provide a source of unit tests for any solution. https://tomassetti.me/domain-specific-languages/
@utvc Yes, it could. The part you are missing is that there is a contingent of people who are convinced that there be as few constraints upon the contents as possible. Really, they don't want to be in the business of parsing and/or validating XML (or any DSL) within the string (Nicolas, myself, and Simon, if I'm not mistaken). There are others that have been advocating for heredocs (a la bash) where they just get free form text between two markers that are not included in the resulting string. Then there is the opposing contingent (including Kevin, Juraj, and Philippe) who want fully validated XML (JSX, really) and neither want to type a delineator or use Haxe's string escaping mechanism between opening and closing XML tags.
The current implementation is a compromise that doesn't work, and so we are going back to the tug-of-war to see who can pull the knot closer to their own direction.
For my part, as a tool developer (and Simon's, I believe, as the compiler's parser writer,), I don't want the Haxe construct to be XML-ish without delineators, because it introduces too many vagaries into the parsing. If I have a well-defined (constant) delineator, it is much easier to understand and change lexer/parser states. Without that, it is much more difficult to lex the tokens properly and understand that something like a<b>c
is a syntax error (because a variable cannot immediately prepend a string without and operator in between) and a<b>c
is just a comparison of variables without looking ahead to find the potential end of the string. Sure, I can figure that out using heuristics (by finding the end marker, or whether a variable name has been declared) of the string (or not), but then I'm way down the token stream and have to back up again. (Beside that, certain types of logic do not belong in the lexer where string state is maintained.)
Thanks so much for the explanation, Eric.
Would it make sense for there to be not only a syntax for the general case (no constraints on the content within it) :
// following Nicolas's original example and open/close marker style
var stuff = #OPEN
anything that can contain newlines
and
also ${ if( flag ) #OPEN can be reentrant #CLOSE }
#CLOSE;
but also an XML-ish -specific variant that could take more care to parse/validate/whatever the DSL embedded within it:
var stuff = #OPEN:XML
<foo this="that">Lorem ipsum <i>${ if (flag) ... }</i>
<img src="foo.png" />.</foo>
#CLOSE:XML;
?
If all it takes it to not accept a single self-closing tag then I'm taking it over any additional markers...
Personally not in favour of using XML based syntax especially if the goal is to allow any sort of DSL to be embedded within Haxe.
How about using a syntax similar to Rust?
xml! {
<something attr=${if (flag) !{ 5 }>Hello<self-closing/></something>
}
Where xml is the macro function that receives the DSL as a series of tokens/expressions. It can then provide completion, error checking, etc.
We could even embed other languages in Haxe code as well.
var code = css! {
.root {
width: 500px;
height: 120px;
flex-direction: row;
${ if (true) !{ padding: 20px; }}
}
}
Thanks for all comments, I'm closing this in favor of #60
As part of our syntax changes in Haxe 4.0, we wanted something allowing users to embed any kind of DSL within Haxe syntax. That could be JSX, HTML, or anything else not necessary XML based.
In order to do that we need two new syntactic elements (tags). One tag for opening a block string and one tag for closing it.
Some languages uses triple quotes as tags, such as:
But since such strings can be processed with macros, we want them to be reentrant (so they can contain haxe code which itself contain block strings), so we need to distinguish opening tag from closing tag, in order to detect end of parsing, such as the following example shows:
In that example, we use #OPEN/#CLOSE as block strings delimiters, and we make sure that when we enter the first #OPEN, we will detect #OPEN/#CLOSE inside it until the last #CLOSE.
Since several of the DSLs that we wanted to embed in Haxe 4 were XML based, we choose to have
<node-name
as opening tag and</node-name>
as closing tag.This works pretty well as the following example shows:
This work well with reentrency too:
However we found a problem with self-closing xml nodes, for two reasons:
var x = <code something/>
is not valid under the strict definition of opening and closing tags, and we can't make/>
a closing tag as well without validating the whole xml, which we definitely don't want to do as this would restrict the allowed syntax inside block stringsvar x = <div><div/></div>
because since our opening tag is<div
, it detects two opening tags but only a closing one.So far we have patched this doing the following rule: when we find
<div
, we look until we either find<
or>
, in which case the block is opened, or until we find/>
which means it's a self closing tag.IMHO this is not a good solution as this makes some assumptions about the content and syntax of the block string, which we should avoid (apart from opening/closing tags definition).
As the feature is still experimental, we need to either accept it the way it's done, or find better opening/closing tags. Candidates are ones that would not conflict with existing syntax, and allow for nice reentrency.
For example, while
[[[...]]]
are not that bad, reentrency looks very ugly, especially when you want to embed some actual XML/JSX/etc. :My proposal would be use
~<node-name
and</node-name>~
as this does not impact much readability but still allows for correct detection of such tags:Please note that we could also add
/>~
as a closing tag, so self-closing nodes works as well.Of course, any other proposal is welcome before we change this.
Three possible choices for final design:
<node
and</node>
as tags, forbid usage of self-closing (only of the "top level" node), work on error detection to give good error messages<node
and</node>
but makes some very arbitrary assumption that the inner syntax is XML based (potential issue example:<code>if( x<code.length ) echo $T</code>
here<code
is wrongly interpreted as an opening node)~<node
...</node>~