lib:langextra (or a better name)

[remexre]

I'm thinking about refactoring lib:monto a bit before it's merged, to try to put everything that's "not compilation" under a single grammar. Tentatively planning:

• Existing things
  • docgen backend -> lib:langextra:docgen
    • By backend, I mean the parts that go RootSpec -> genFiles
    • It might also make sense to rework it a bit when it gets moved -- it would make sense to me to only parse comments as Markdown, then generate everything else directly as HTML.
    • The reason why this would be somewhere "global" instead of in Silver is mainly for the third bullet point under lib:langextra:hoogle below; we probably shouldn't use Jekyll as a documentation generator, but rather something made for that purpose; dumping decls to JSON is a pretty good way to let us change how the actual HTML is generated without needing to do "annoying" stuff (i.e. implement Markdown/CommonMark parsing in Silver instead of using an existing implementation).
    • If we're generic over what a decl is, we might be able to reuse this to e.g. act as a compiler-supported documentation generator for ableC. (I'm guessing that e.g. Doxygen isn't happy with getting already preprocessed sources, but I don't know for sure.)
  • lib:monto -> lib:langextra:monto
    • This would probably get split so everything that's only for the Monto protocol lives in lib:langextra:monto, and everything reusable goes in lib:langextra.
• "for the future"
  • Hoogle backend in lib:langextra:hoogle
    • This is just dumping function and type definitions as Haskell values, so it should be relatively easy?
    • See Roogle, which is basically the same thing for Rust -- the first section of its README.md shows the Haskell values needed.
    • Alternatively (this is what Roogle does), have the doc tool dump decls as JSON and then have a separate program (part of the Hoogle/Roogle server) convert the JSON to Haskell values.
  • HTML highlighting in lib:langextra:html
    • If it's just wrapping every token in a <span> with the lexer classes as HTML/CSS classes, then doing the styling with external CSS, this is almost trivial.
  • LaTeX highlighting in lib:langextra:latex
    • This is also theoretically pretty easy; the minted LaTeX package for syntax highlighting uses Pygments, but allows setting a custom version of that executable with \MintedPygmentize.
    • Making it should just require faking pygmentize's command-line interface.
    • Alternatively, it'd also be reasonably easy to output the actual LaTeX ourselves.

Also, I don't have any strong attachment to langextra as a name. I'm also considering langservices, but any other suggestions are more than welcome!

---

Also, I'm thinking about trying to convert Silver's Message to the langutil one in the feature/monto-in-silver branch -- it's a bit unergonomic to need to have two versions of every function that works on messages, one for Silver and one for everything else.

---

Thoughts?

[tedinski]

Remember #152, so lib isn't a good prefix. The bit at the end of that issue is the major thing: we someday want package names that play well with the Java ecosystem. I suggested maybe silver:langutil:monto but that would assume we'd distribute one versioned langutil artifact. Perhaps we'd want it as a separately versioned artifact. silver:support:monto? I'm not sure we want anything else starting with lang but I'm not having a lot of ideas at the moment.

RE: multiple Messages: I think the only impediment there is that langutil has pp be a Document while Silver wants it to just be a string. Converting over might be overly painful?

RE: everything else... I'm not really sure what you're referring to with a lot of this stuff. I'm in favor of refactoring as we go, so if we merged it all in one grammar and broke things out over time, that's probably preferable. Can you link to code you're calling docgen backend?

[remexre]

silver:support sounds better to me, yeah.

If we can use the pp"foobar" syntax in Silver, I think it'd get a lot less painful to switch that over.

Yeah, I probably could've worded the docgen stuff better. This would mostly be silver/grammars/silver/extension/doc/driver/BuildProcess.sv and the toMarkdown stuff in silver/grammars/silver/extension/doc/core/RootSpec.sv, as well as any other parts that deal with Markdown directly. The advantages I see from this are:

• We can generate "better" documentation
  • See Go or Rust for what I consider "better"
  • Mainly links to referenced things (including inter-module links) and search.
  • Rust having different colors for different things is nice too
    • e.g. for Rust, struct names are orange and enum names are green
    • e.g. for Silver, function names are $color1, abstract productions are $color2, and concrete productions are $color3
    • This is admittedly less useful than Rust's case, but still nice
• We can make it more generic
  • Every "real" programming language (e.g. not CSS) I can think of has the concepts of modules, functions, and types, so if we start with an interface for these concepts and add the ability to define language-specific things (e.g. struct members for C, attributes and terminals for Silver, etc.), we end up with something that gives high-quality documentation for arbitrary languages
    • hopefully :)
• We can reuse generic documentation support for "get the documentation of the thing my cursor's on" in Monto

Actually doing this would involve a refactor of how we currently do documentation, which I'm going to declare out-of-scope for this issue. This issue is mainly "what should I call this thing" and "should I unify the Message types while I'm at it"

[tedinski]

I'll have to look at the docgen stuff in a couple of days, I'm really busy right now.

RE: pp in Silver: I'm probably in favor of this, so long as it doesn't lead to increased ugliness of any code. HOWEVER, note that the silver translation often does // ${blah.pp} when generating code, so we'd actually have a problem making sure that newlines don't show up... Now that I think about it, that's maybe a blocker to bothering to change Silver over to using Document.

...Mayyyybe we should switch Silver to use unparse instead of pp? That's just a rename of the attribute, then... but some things might already have an attribute called unparse? Not sure.

[ericvanwyk]

This all seems like silver:langutil to me. These sub-grammars, e.g. monto, have functions and datatypes for different language related activities. silver:util is for general purpose data structures like tree maps and such, right?

silver:support reminds me of the support directory holding Silver scripts and editor modes. I like the distinction of langutil and util - it seems more clear what their intended uses are. Where util and support sound like the same thing. Just a thought.

Regarding documentation, this directory silver:langutil:docgen or whatever it is called has only language-independent things in it. There are things that Silver, ableC, or other languages would use to support documentation generation. So I don't think the BuildProcess.sv file or its ilk would be here. The functionality that Silver uses to generate Silver documentation belongs somewhere else, maybe where it is now.

If prepending all appropriate Strings in Silver with pp then maybe this isn't too bad.

[ericvanwyk]

I like the idea of renaming pp to unparse since that what this attribute really is - not much 'pretty' about it.

Does this then make slotting in the langutil Message very easy then - the impediment is just getting pp out of the way?

[remexre]

Sorry, I'm a bit confused now -- what part of Message requires that compilation use Document instead of String? I just meant that the pp"" syntax would make it easier to change how Messages were constructed, not necessarily that we should do a complete changeover of compilation. (If we do want to do that though, I have no objections.)

I'll take a look at doing this after feature/monto-in-silver gets merged; I really meant this issue on "where should the generic support service things be located," and kinda got sidetracked when writing the OP.

If we're enlarging langutil, then it'd make sense to put this stuff there.

[tedinski]

what part of Message requires that compilation use Document instead of String? I just meant that the pp"" syntax would make it easier to change how Messages were constructed

My understanding of the difficulty of switching over to using langutil in Silver is the name clash between silver's pp and langutil's pp. Silver uses a string, langutil a Document.

Messages shouldn't use Document. There's already a s"" syntax, if that's what you're looking for! :)

[remexre]

Hah, I've been working on Monto's networking stuff long enough that Silver's stuff is "out of cache" :). I could just do an import-only, right?

[tedinski]

As for where it gets put:

Eric, the only motivation I have for silver:support:monto is that we'd get a separately versioned artifact later on when that's a thing. i.e. we'd have

silver.compiler-1.0.0.jar
silver.langutil-1.0.0.jar
...etc...
silver.support.monto-1.0.0.jar

as distributed artifacts. My thought was only that it might be valuable to distribute separately, instead of it being bundled with langutil (and everything else we might choose to put under there being bundled together. e.g. eclipse support too.)

It's possible we might put the "docgen backend stuff" under langutil, once that gets factored out, but put monto support in a different artifact?

[tedinski]

import ... only would work, but you'd have to put it in a LOT of grammars. Pain.

If we mass renamed "pp" to "unparse" (after verifying that'd work) in Silver, we could then just exports langutil from silver:definition:core and that'd be easy peasy...

[ericvanwyk]

Ted, the idea being that not everyone want the Monto stuff? Or that there may be different versions of Monto stuff that people want? I think you mean the former.

Docgen stuff would stay in langutil because Silver source files had doc-gen syntax in them? Otherwise dog-gen and Monto seem like similar concerns one might choose to not use for a specific language.

[remexre]

It's possible we might put the "docgen backend stuff" under langutil, once that gets factored out, but put monto support in a different artifact?

Why would the docgen stuff need to be in langutil? Couldn't we just declare it to be a dependency of the documentation extension's "binary"?

---

Long-term, I'm envisioning (keeping #152 in mind):

+---------------+                                     +--------------------+
|silver:compiler+---+                               +->silver:support:monto|
+-------------+-+   |                               | +--------------------+
              |     |    +------------------------+ |
              |     +---->silver:langutil:LangInfo+-+ +---------------------+
              |          +------------------------+ +->silver:support:docgen|
              |                                     | +---------------------+
              |                                     |
              |                                     | +--------------------+
              |                                     +->silver:support:latex|
              |                                       +--------------------+
              |                                             |
              |           +------------------------------+  |
              +----------->silver:compiler:composed:latex<--+
                          +------------------------------+

where LangInfo (again, naming is up for discussion) is a nonterminal with a bunch of function attributes, for e.g. parsing, getting errors, getting decls (with a Maybe<documentation>?)

That way, the only thing that has a hard dependency on a silver:support:* grammar is the relevant composed grammar.

[tedinski]

@ericvanwyk : Both and more. Think about it this way: a lot of artifacts will depend on the langutil artifact. You don't want it to start depending of lots of other things, like a monto runtime.

I don't know exactly what the "docgen backend" consists of, so maybe I'm wrong about that. But I imagined just data structures for representing info used in documentation generation. Kinda like how we have Message for errors.

[remexre]

I don't know exactly what the "docgen backend" consists of, so maybe I'm wrong about that. But I imagined just data structures for representing info used in documentation generation. Kinda like how we have Message for errors.

Yeah, that's essentially what I mean. What I'd keep in langutil on the LangInfo nonterminal would be a docDecls attribute which has basically anything needed to be able to generate docs for both C and Silver. (They're sufficiently different that I feel like anything that works for both is generic enough to work for "anything")

[ericvanwyk]

@tedinski: That makes sense. In the short term everything is still in the Silver repo somewhere under the silver name space but in the long term we can split these into different packages (as jars) that get distributed based on the need for specific things. So, things that come with extra runtime stuff (ZeroMQ jars for Monto for example) naturally live in silver:support (or something) and things that don't need runtime support are in silver:langutil and are presumably more commonly used. I think doc-gen and Monto would have similar usage patterns (or let's assume so for the sake of this discussion) and then the different is really about runtime support or not. Right?

[tedinski]

and then the different is really about runtime support or not. Right?

I don't understand this question, but the rest of the comment is right.

Actually, let me make a guess at what you mean: we'd probably continue to distribute everything along with silver, but the idea is that the user's artifacts (e.g. ablec?) would not need to distribute all of what comes with silver, but only what it needs. Also, it'd offer us the possibility of eventually dropping things from what we distribute, by being able to make them available elsewhere. Not being under silver:langutil means it could be factored into a separate repo, if such a thing were ever needed.

BTW, I'm open to a different prefix than silver:support for the "other stuff we want to distribute as separate individual artifacts" if you don't like the name.

[ericvanwyk]

I think you've figured out what I meant.

So something like ableC would need silver:langutil and optionally things in silver:support, but not silver:definition and other similar grammar defining the Silver language. This would suggest factoring not only silver:support but also silver:langutil into separate repos from Silver.

I think we can hash this out on Thursday, if you're in then.

In the meantime Nathan can continue with silver:support:monto, with the caveat of maybe renaming support.

[tedinski]

This would suggest factoring not only silver:support but also silver:langutil into separate repos from Silver.

Such things would become possible but not necessary. We'd probably always keep langutil within silver's repo. We'd probably keep most things under silver:support in-repo with silver. But it means that later, if we decide something is deprecated or has grown into such a big project we want it separate, we can move it to a different repo with no problems. (i.e. it can keep its name silver:support:whatever.)

In the meantime Nathan can continue with silver:support:monto, with the caveat of maybe renaming support.

Yes.

I'll be in Thurs as usual.

[ericvanwyk]

Sounds good.

Maybe silver:support is not such a bad name. We have editor modes for emacs and vim, I think, in the silver/support directory. Some "support" things are not Silver grammars and live in silver/support and some "support" things are grammars and live in silver:support. Or is this too confusing? But again, Thursday...

[krame505]

Did this ever get discussed/resolved? Anyway, silver:support seems like a fine name to me.