Does moonxml provide special functions?

Lukc commented 5 years ago

I’m thinking mostly about raw and text from Lapis, which are very useful in practice. Other particular tags could also benefit from this, though, like the <?xml?> or <?xml-stylesheet?> tags, or DOCTYPEs.

Even better, is there a way to define new tags (preferably without doing it in the global environment)?

DarkWiiPlayer commented 5 years ago

It is actually possible to add custom tags with ease, though that feature is still undocumented because I'm not sure if this is how I want it to work in the end. If you look at xml.moon, in the last few lines you'll find

environment.html.html5 = ->
    environment.html.print '<!doctype html5>'

environment.html is the environment used by all the HTML generating functions, so you can just append custom functions to that, or set ENV to that table and just define your functions "globally"

As for raw and text, MoonXML uses print as a generic output function, and provides the escape function to escape HTML characters. I don't even know why that's not in the readme.

Lukc commented 5 years ago

So, if I understand correctly, print is the equivalent of raw in Lapis? That is, print adds to the output/buffer its arguments unprocessed?

DarkWiiPlayer commented 5 years ago

Exactly. And if you want them escaped, you just call escape on them first, like this print escape "<b> is the tag that makes text bold..."

Lukc commented 5 years ago

Alright. Going back to defining new tags, is there a way to have and handle parameters?

For example, something that would allow transforming the following:

render ->
    xml version: 1.0

… to this:

<?xml version="1.0"?>

DarkWiiPlayer commented 5 years ago

I don't have time for a longer explanation until this afternoon, but basically:

local moonxml = require "moonxml"
do
  _ENV = moonxml.environment.xml
  xml = (opts={}) -> -- "global" in the xml environment
    print "<?xml version=\"#{opts.version or '1.0'}\"?>

should do what you want

EDIT -- Okay, in a bit more detail:

Since you can access the environment that the builder functions are run in, you can easily define new methods inside them. If done as above, resetting the local environment before defining the function, you can also use the auto-generated functions within your custom ones as well as print, escape and all the other special stuff.

Adding arguments to your custom functions works just as one wold expect, since there's not really any meta-programmed magic behind that. Keep in mind though, that your custom defined functions will work differently from the automatic tag-functions, as they lack all the argument serializing magic unless you specifically build them like that. Lua itself allows for so much magic with functions that there's not really a need to add anything more than that, at least for now I don't see one.

The big problem with this though is that you only get one XML and one HTML environment each time the library is loaded, and monkey-patching it in one place affects the entire problem, which is something I really hate. An option that I'm considering right now is to add a mechanism for duplicating the environment before modifying it, but that comes with its own downsides, most importantly, added complexity for the user.

Another important consideration is to keep the DSL simple (After all, that's the purpose of using a DSL in the first place) yet not letting it become too magic, so it's still easy to reason about the behavior of the code without understanding all of the meta-programming stuff that's going on in the background. In short, I don't want MoonXML to feel like a Ruby DSL. I hate those.

With all that being said, all suggestions for how user extensions could work are welcome. No need to provide any implementation, I just want to know how people would want to use it.

Lukc commented 5 years ago

Well, if we’re going in the realm of suggestions, here’s how I’d want to use a DSL library:

renderer = require("moonxml").new!

-- Tag registration "hand-building" its body.
renderer\register_tag "something", (options, content) ->
  raw "<something>"
  content!
  raw "</something>"

-- More automated tag registration, using provided tools.
renderer\register_tag "example", (options, content) ->
  options.class or= {}

  table.insert options.class, "is-important"

  element "example", options, content

-- All registered tags should be available somewhere public, so that environments can be duplicated easily.
with renderer.registered_tags
  .ex = .example

renderer.render ->
  html ->
    body -> …

The rationale behind this being:

Creating new renderers allows one to have as many isolated environments as needed. Special constructors could be used to create environments that have some initial custom tag definitions inside them (for example, .xml and .html, just like we already have).
Having to instantiate renderers mean you can pass them around and that you should be able to reuse them if needed.
Handling parameters and providing them as a table of options (possibly empty) and a content callable makes it easier to create new tags, or widgets, components, and so on. In this pseudo-code example, example is a function that always has an "is-important" class, but in real-life it could be containing other tags and custom options used to generate their contents.
In this pseudo-code example, I used raw instead of print because I really don’t like shadowing debug facilities on such a large scale, but this is more of a detail point than anything else. Using raw instead of print would probably bring some more compatibility with Lapis as well, but this doesn’t really matter in practice and I’m just mentioning it for the record.

I feel like this is still overall pretty simple. Or at least, it does look simple from a user’s perspective.

I don’t know what’s your vision about the future of the library though, so maybe this is not what you have in mind?

DarkWiiPlayer commented 5 years ago

To be honest, I'm not a huge fan of that object-oriented pattern where you instanciate an object that really has no associated data and just a single significant method. Another problem I see is that sooner or later it will be necessary to move the DSL code into its own file/string and load it with a set environment, because debug.setupvalue() just isn't stable enough in 5.3. At the very least it would only work in 5.1/JIT and throw an error in 5.2/5.3, with the preferred load implementation being the more portable one.

Another approach could be to string.dump a function and load it, which might be slower, but actually generates a copy of the function. This gets complicated because upvalues though.

As for the environment thing, it's really a complicated question. Originally, there was just HTML, then I made a new project for XML, merged them because they were too similar and now the question of custom extension arises.

My idea, and I am very much still unsure of how good it is, would be to have two predefined environments (XML and HTML) and an easy mechanism to "inherit" from them (set __index metamethod) to a new environment, that the user can customize.this environment could then be a new first argument to a function that renders the template / generates a renderer from a function.

Overall I'm pretty fond of the idea that a renderer is a closure, not an object.

In the end, I imagine it could look somewhat like this:

moonxml = require("moonxml")
moonxml.environment.new("my_environment", "html")
moonxml.environment.my_environment.doctype = ->
  print "<!doctype html>"
template = moonxml.template "my_environment", ->
  doctype!
-- or, alternatively
template = moonxml.template moonxml.environment.my_environment, ->
  doctype!
  html -> body -> h1 "Hello World ♥"

Obviously this example is quite verbose, but a bit of import and with could easily tame that verbosity.

-- templates.moon somewhere in the project directory structure
env = moonxml.environment.new("my_env", "html")
-- Let's say this function returns the new environment, which would make sense anyway
setfenv(1, env) -- or _ENV=env in 5.2+ or use `with env` and dots
export doctype = ->
  print "<!doctype html>"

-- somewhere in the application
import template from require "moonxml"
template "my_env", "templates/file.whatever"
-- Alternative to providing a function: path to a file

-- templates/file.whatever
html ->
  doctype!
  body ->
    img src: "path/to/image"
  -- etc.

Maybe you could just pass a table to environment.new instead:

moonxml.environment.new "new_name", "parent",
  doctype: -> print "<!doctype html>"
  title: => h1 @
  -- etc.

Lukc commented 5 years ago

To be honest, I'm not a huge fan of that object-oriented pattern where you instanciate an object that really has no associated data and just a single significant method.

Well, it does have associated data: the registered tags and their definitions. I guess some options for some definitions could also be there (the list of escape sequences, for example, changes between HTML and XML, so it could be stored in that object).

Another problem I see is that sooner or later it will be necessary to move the DSL code into its own file/string and load it with a set environment, because debug.setupvalue() just isn't stable enough in 5.3. At the very least it would only work in 5.1/JIT and throw an error in 5.2/5.3, with the preferred load implementation being the more portable one.

What about using moonscript.utils for this? IIRC it redefines a few functions for compatibility purposes, like loadfile or so. Of course, this would mean a runtime dependency on MoonScript, which has its own set of issues, or importing the code, which is also less that ideal.

Another approach could be to string.dump a function and load it, which might be slower, but actually generates a copy of the function. This gets complicated because upvalues though.

No idea about this.

As for the environment thing, it's really a complicated question. Originally, there was just HTML, then I made a new project for XML, merged them because they were too similar and now the question of custom extension arises.

I’m interested in rendering XHTML, which has a bit of both. That’s the actual reason I’m interested in creating new environments. :p

My idea, and I am very much still unsure of how good it is, would be to have two predefined environments (XML and HTML) and an easy mechanism to "inherit" from them (set __index metamethod) to a new environment, that the user can customize.this environment could then be a new first argument to a function that renders the template / generates a renderer from a function.

Overall I'm pretty fond of the idea that a renderer is a closure, not an object.

In the end, I imagine it could look somewhat like this:
moonxml = require("moonxml")
moonxml.environment.new("my_environment", "html")
moonxml.environment.my_environment.doctype = ->
  print "<!doctype html>"
template = moonxml.template "my_environment", ->
  doctype!
-- or, alternatively
template = moonxml.template moonxml.environment.my_environment, ->
  doctype!
  html -> body -> h1 "Hello World ♥"

I don’t like the idea of having new objects you create be stored in a global environment, that could be overwritten or interfered with from other parts of a software using the same module. Creating new environments through some kind of inheritance also looks good, however, as long as you can pass it around and that it’s not global… but this would basically end up the same way (I think?):

moonxml = require "moonxml"

my_env = moonxml.clone "html"
my_env.doctype = ->
  print "<!doctype html>\n"

This would still look fine as far as I’m concerned, although cloning an “empty” environment (and not just xml or html) could still be useful, very much so if the default/unknown tags can be overwritten. But maybe this is getting out of the scope of this project? :p

As for the last example, I really have no idea what would be best as an import mechanism. I, personally, either wouldn’t need one or I’d roll my own (with either require or loadfile, and some library to provide independence from Lua version, probably).

DarkWiiPlayer commented 5 years ago

I don’t like the idea of having new objects you create be stored in a global environment, that could be overwritten or interfered with from other parts of a software using the same module.

True, that goes completely against modularity. I guess a nicer way to do things would be like this:

my_env = xml.environment.html:derive()
my_env.div = -> os.exit() -- let's play a prank on our coworkers

This allows for more modularity when needed, but if desired, you can also just write your newly created environment into xml.environment (useful for smaller projects)

An empty environment might be nice, but the biggest problem is the formatting and behavior of empty tags.

In HTML there's a whitelist of tags that don't get closed, like <br>, while in XML every tag without children is treated that way, and the / is added to the opening tag. Until I find a good mechanism of abstraction to remove those differences from the core code, it will be quite hard to really find a nice way to solve the environment problem.

DarkWiiPlayer commented 5 years ago

After thinking about it for a while, I think the best option might be to add an additional layer to the system. At the top level, there'd be the language, which defines how a tag is built. Within each language, there's environments, which can define custom tags. Then there's templates and lastly content (no change there).

Pros:

Easy to support new languages (Do I see a JSON in the distance?)
More modular code
Reduced Complexity in some parts

Cons:

Requires rewriting maybe 70% of the code
Increased complexity in some parts

Thoughts:

print should be set and reset before and after rendering, so when no print is given, it falls through to the global environment. This actually reduces complexity and allows for more advanced use.
The syntax should be defined by a single function that accepts
- environment
- name (string)
- attributes (table<string, string>)
- content (function())
when defining a new language, you'd get an "empty" environment.
- All tags are their defaults
- Can be extended easily by adding functions

Lukc commented 5 years ago

In case you’re expecting an answer, I also think this would be a very good option. :+1:

DarkWiiPlayer commented 5 years ago

Small update on this: I've rewritten large parts of the code and have pushed some what I have so far to the new exp20 branch (experimental 2.0). So far only the core is complete and the tests will most likely all fail, but it gives an idea of what things will look like and can be played around with.

DarkWiiPlayer / moonxml-old