Rethink reader macros

[gilch]

Reader macros originally required fully-qualified names. I've since added unqualified names ending in a hash in the _macro_, and seem to be using those a lot more.

It's a design goal of Hissp that everything be available in-line without advance imports. This is important for lissp -c usage, and any other embedding that just does short snippets. Maybe I should add that to the README. Unfortunately, this isn't possible for the prelude definitions, but lissp -c does imply the prelude. Fully-qualified names work for all importable runtime objects, and for compiler macros. The original fully-qualified reader macros also fit this requirement.

But, it's pretty awkward that using a reader macro that already ends in a hash requires an escape of that hash. E.g.

hissp.._macro_.b\##"Fully qualified b# macro at read time."

It would be nicer if

hissp.._macro_.b#"Fully qualified b# macro at read time."

worked, perhaps by assuming the hash is part of the name. Maybe we don't have to require it to live in a _macro_ namespace anymore, just end in #. But consider

builtins..ord#Q

Now this won't work, because it's ord, not ord# in builtins. But,

.#(ord 'Q)

still would. Maybe we can stop here.

---

This is both better and worse. It's nice that we don't have to fully qualify it (although we can), but it's too bad we have to wrap it in (), which could force a line break in standard style. ord was never meant to be a reader macro, we're just invoking it that way, so maybe an inject makes more sense.

But consider the alias macro

(hissp.._macro_.alias M: hissp.._macro_)
M:#!b"Read-time b# via alias."

(alias B builtins.)
B#!ord Q

also from the quick start. As a proof of concept, a custom reader macro could be made to work the same way, on any symbol resolvable at read time:

as-reader-macro#!builtins..ord Q
as-reader-macro#!ord Q

But, we'd have to "import" as-reader-macro somehow. The whole point of this form was to make everything available inline without advance imports.

One solution might be to extend the built-in inject macro .# to accept optional extra arguments, as aliases do.

.#!builtins..ord Q
.#!ord Q

This would allow any read-time resolvable callable to be used as a reader macro, but wouldn't necessarily require a fully-qualified name. This frees up the fully-qualified reader macro syntax to assume the name ends in #, so we don't have to repeat it with an escape, with an overhead of only a few characters: ., !, and maybe a space. (Not that fully-qualified reader macros were ever short.)

[gilch]

One illustration of why the current inject isn't sufficient, and why we'd need an as-reader-macro of some kind as a replacement, is that some objects are not pickleable.

#> .#(lambda :)
>>> # CompileError

(>   >  > >><function <lambda> at 0x000002229386C9D0><< <  <   <)
# Compiler.pickle() PicklingError:
#  Can't pickle <function <lambda> at 0x000002229386C9D0>: attribute lookup <lambda> on
 __main__ failed

#> 'builtins..repr#.#(lambda :) ; This works, but we're about to remove this syntax.
>>> '<function <lambda> at 0x000002229386C8B0>'
'<function <lambda> at 0x000002229386C8B0>'

#> '.#(repr '.#(lambda :)) ; The obvious direct alternative fails.
>>> # CompileError

repr(
  (>   >  > >><function <lambda> at 0x000002229386C4C0><< <  <   <)
  # Compiler.pickle() PicklingError:
  #  Can't pickle <function <lambda> at 0x000002229386C4C0>: attribute lookup <lambda> 
on __main__ failed
  )

#> 'B#!repr .#(lambda :) ; But an alias can do it!
>>> '<function <lambda> at 0x000002229386E0D0>'
'<function <lambda> at 0x000002229386E0D0>'

[gilch]

Aliases already check for _macro_ in the module name and only append the QzHASH_ before lookup in that case. Fully-qualified reader macros could be made to work the same way. This is probably simpler than adding extras to Inject.

The one case this can't handle is when you want to use a compiler macro (living in a _macro_ namespace, but named without the trailing #) as a reader macro. Aliases have the same issue, but we could at least fully-qualify it as a workaround now. If fully-qualified macros were made to work the same way, we'd lose that workaround. Inject is still available as a workaround, but again, may have to go through pickles, and not everything is pickleable. The final workaround is to assign it a new name. Not ideal, but seems pretty rare. This is all that escaped hash is buying us. It hardly seems worth it.

[gilch]

Back when reader macros were single argument, appending a # to distinguish them from normal macros seemed so elegant. It seems less so now that the number of arguments can vary. Typical usage of @# is written @##, for example. A combined namespace has the advantage of a single defmacro being able to define both types. There's also only one special namespace that needs to be initialized.

But given the possibility of renaming _macro_ (#257) and additional reader macro args (#187), I'm considering separating the namespaces altogether. The combined namespace makes alias more complex than is probably necessary and required the special casing in #175. Using an alias instead of a shorter name for _macro_ (as discussed in #257) seems like it might require re-implementing alias in Python along with all the handling of special cases, even though the common case of a normal macro would be pretty trivial.

Dropping the requirement of a # suffix seems like it would simplify a lot, but this doesn't really work since the same symbol would then need to work as both a tag and as a normal macro, e.g. @ for both 'list of' and 'decorator'. There may be other hacky ways to distinguish them, like an attribute, but it feels like they really want to be separate namespaces. I'm not sure which is worse.

Now where to put them? They could both be globals. _macro_ and _tag_? _macro_ and _QzMacro_? Or the tag namespace could be nested in _macro_ somehow. That would be more contained, but flat is better than nested. Usage might need another layer of attribute access.

[gilch]

I don't think I'm going to be making further changes around this issue before the next release. Untangling it now is too much.

Fully qualified imports are always available. The three main ways to avoid them (for macros) are prelude, alias, and maybe attach, although alias should be preferred with attaching to _macro_ used sparingly (I should add that to the style guide).

prelude and alias (via defmacro) will initialize the _macro_ ns. attach doesn't. Splitting the namespaces complicates this picture.

If Hissp had an ns macro like Clojure, module initialization could be done cleanly there. Clojure's ns macro is one of its legacy mistakes. It's not strict enough to nail down a standard style (even in indentation and bracket types) and it's more complex than in needs to be. Parts are effectively deprecated in modern style but retained for backwards compatibility. I really wanted to avoid an ns macro for Hissp.

I want short fragments of Hissp to work for things like lissp -c and readerless. I don't want the user to have to ceremoniously initialize a module just to compile some Hissp. That's why defmacro doesn't require _macro_ to exist. Unfortunately, module initialization is not entirely avoidable. You do need a module path if you want imports to resolve correctly, although a snippet not being used by anything else can think it's in main and things should still work. Ensue from the prelude is too long for inlining to be reasonable. engarde and enter are much easier to define at the top level (in an exec), which shouldn't be inlined either. lissp -c assumes the prelude for you. Readerless doesn't require you to use it. You can fully qualify things and use some other functional library, or dump the prelude in a namespace somewhere once and import from there, even in snippets.

Hebigo's def is quite a bit more powerful than Lissp's bundled define. You can def into a namespace, not just globals. That means one form could define functions, macros, tags, and globals/"constants". It's too complicated to bundle, but maybe it could inspire a solution.