New api for creating static documentation

[peter1000]

Initial start of a new Documentation Layout with some docs #110

Example:

require(joinpath(Pkg.dir("Lexicon"), "src", "Document", "Document.jl"))
using Document
out = Layout(
        LayoutConfig(),
        Section(
            SectionConfig(),
            Page{:api}(),
            Page{:index}()
            ),
        Section(
            SectionConfig(),
            Page{:manual}(),
            Page{:api}(),
            Page{:index}()
            )
        )

[MichaelHatherly]

Can we avoid parameterising Page? Do we need separate LayoutConfig and SectionConfig? What about PageConfig?

I'd prefer lowercase for section, page, and config. Layout -> document as well.

Thanks for getting the ball rolling. I'll have some time later this week or perhaps next to look at the implementation properly.

[peter1000]

I can make the changes no problem.

Do we need separate LayoutConfig and SectionConfig?

I think so. If we find out they are not needed I will merge them into one which should be quite easy if they are the same.

I will continue push here - so if you want to see any procress or comment on it.

What about PageConfig?

I thought parameterising Page would take care of this. But as you said: Can we avoid parameterising Page? maybe a PageConfig? could be added but at the moment I would say that would probably nullify what you said about the section: section would be used to separate the documentation into different parts that could have different config settings.

[MichaelHatherly]

but at the moment I would say that would probably nullify what you said about the section

page will need to have the possibility for adding config data to it. Would you be able to sort and filter the docstrings in a page without having a config(sorter = ..., filter = ...)?

I think so. If we find out they are not needed I will merge them into one which should be quite easy if they are the same.

A config passed to a document and a config passed to a section can just be adjusted internally by the system to what's needed though without the user having to care about two different names. Or can't they? Having a very small interface to learn is quite important I'd say.

We should be able to add multiple configs to a single node (document, section, or page) with later configs overriding earlier ones. Also arbitrary keyword arguments for nodes that override all config objects for a node would be good as well. Such as

page(
    "reference.md",
    config(
        a = 1,
        b = 2
    ),
    Docile,
    "Some intermediate text.",
    Docile.Cache,
    Docile.Formats,
    config(
        a = 2,
    ),
    b = 3,
)

resulting internally in

page([
        "<contents of reference.md>",
        Docile,
        "Some intermediate text.",
        Docile.Cache,
        Docile.Formats,
    ],
    config(
        a = 2,
        b = 3,
    ),
)

Adjacent Module objects would have their docstrings merged when filtered/sorted/grouped for rendering.

We then provide additional custom config objects such as an index() that causes a page to only render the :summary metadata for each docstring and their name.

(These are just ideas though for the moment.)

[peter1000]

We should be able to add multiple configs to a single node (document, section, or page) with later configs overriding earlier ones

haven thought about this situation but sounds good.

[MichaelHatherly]

Also, a reference to each parent node would be useful to have for some of the rendering code, ie:

type Section
    parent :: Union(Document, Section)
    ...
end

type Page
    parent :: Section
    ...
end

[peter1000]

related questions;

if I have something like:

abstract DocumentNode
abstract Config

type Document
    children    :: Vector{Union(Section, Config)}    # not defined error
end

type Section <: DocumentNode
    parent      :: Union(Document, Section)
end

How do you usually handle the

ERROR: LoadError: UndefVarError: Section not defined

[peter1000]

Would something like this be the correct way to do it?

abstract DocumentNode
abstract Config

abstract SectionNode <: DocumentNode

type Document
    children    :: Vector{Union(SectionNode, Config)}    
end

type Section <: SectionNode
    parent      :: Union(Document, Section)
end

[MichaelHatherly]

Vector{Union(SectionNode, Config)}

A Document should only have one Config so it can go in a separate field, not as one of the children.

Section <: SectionNode

Let's not worry about subtyping anything at the moment until it's definitely needed.

Would something like this be the correct way to do it?

Yeah, it's a bit tricky getting the definitions in the right order. For the moment lets just go with something like:

type Config
    data :: Dict{Symbol, Any}
end

type Document
    children :: Vector
    config   :: Config
end

type Section
    parent   :: Union(Document, Section)
    children :: Vector
    config   :: Config
end

type Page
    parent   :: Section
    children :: Vector
    config   :: Config
end

If we happen to run into performance issues, which seems unlikely with this one I'd think, we can always revisit later. Perhaps adding a sprinkling of typeasserts, ::, where needed.

[peter1000]

Just had a realization and a question to clarify: I had an idea that maybe I misunderstood you here

other_pages = [page(m, title  = t,
                       filter = obj -> isa(obj, Function)) for (m, t) in things]

out =
    document(
        config(...),
        section(
            # The manual section of the documentation.
            config(...),
            page(...),
            page(...),
            section(...))
        section(
            # The function reference section.
            page(...),
            other_pages...))

save(pdf("out.pdf", out))
save(markdown("out", out))
...

The two config(...) below document and below the first section.

I understood that this both refere to complete different configurations:

a DocumentConfig and a complete different SectionConfig.

Now I just had a thought that maybe you meant: One general config which can be adjusted for a document and also separately for whole section.

If that is the case than I see that we misunderstood each other: example here

MichaelHatherly: Do we need separate LayoutConfig and SectionConfig?

Anyway, was just a thought and I would be interested to know if you had in mind:

• complete different configurations or (not overlapping fields) that is how I understood it first.
• just different palces to adjust one general configuration.

Cheers

Have a bit less time at the moment.

[MichaelHatherly]

Anyway, was just a thought and I would be interested to know if you had in mind:

• complete different configurations or (not overlapping fields) that is how I understood it first.
• just different palces to adjust one general configuration.

Each node, Page, Section, or Document would have a single .config :: Config field that stores that node's configuration. When constructing nodes you could combine multiple configs to produce the final Config which is stored for that node. This will be a bit more clear when I get a chance to actually play around with writing something :)

Have a bit less time at the moment.

Yeah, same here.

[peter1000]

I did a rough demo implementation of the more complexed : config options I mentioned here did not upload was just playing around.

Will play with a more simple one: I guess it will take me a couple of days or a week till I will be able to upload an example code.

BTW: I saw someone had reported a bug so I asked them if they wanted to be added to the Docile Project page. TradingLogic.jl

[peter1000]

I simplified most: using an inherited configuration which I think is quite a nice way to do it.

Basic idea: - who knows it might be what you had in mind anyway ;)

• One default configuration which can be customised and the final config is pass on to children.
• All children of a node inherit the same parents configuration but each child node can customise it to be even more specific.

docs Config

CONFIG EXAMPLE with comments

out = document(
    # at the `document` root level customize the default Config more general
    # for the whole document: this is inherited by all child Sections.
    config(a=1500, c=565656, include_internal=false),
    section(
        # at the `section` node one can further adjust the inherited config
        # this is inherted by all child pages of this section.
        config(a=0),
        page(
            # at the `page` node one can further adjust the inherited config
            # this is inherted by all child content nodes of this page.
            config(c=300),
            title("Lexicon"),
            objs(Lexicon, :macro),
            title("Docile"),
            # at the `content` node one can further adjust the inherited config
            # for this single content node
            objs(Docile, :macro; include_internal=true),
            ),
        ),
        # this `page` node subtree uses just the inherited configuration
        # from the parent section
        page(
            title("Lexicon"),
            objs(Lexicon, :macro),
            ),
        ),
    )
    # this `section` node subtree uses just the inherited configuration
    # from the parent document
    section(
        page(
            title("Lexicon"),
            objs(Lexicon, :macro),
            ),
        ),
    )

[peter1000]

Aside of the Config:

New ContentNodes can easily be defined outside of the package and allow for package authors to customise how content is integrated into the page sub-tree.

There is still much room for changes but the basic idea of ContentNodes is there. Page ContentNode

[peter1000]

As time permits I will do some examples ...

QUESTION: do you still have in mind do have something like: include_internal or just leave it all up to the document author?

Does the new Docile provide an easy way to filter such (exported/internal) without using the legacy Interface?

EDIT: Found it: Cache.getmodule(m).metadata[:exports]

[peter1000]

Did a base implementation with a couple of high level section/page nodes for a autogenerted packge documentation:

Still missing parts like: rendering of obj, sorting, etc. but the basic ideas are there

• low-level pages to
• high-level multi-module (package) sections
• extensible options

see also example folder and TODO.md which does only note some points.

e.g. High-level package documentation

out = document("TestDocumentation",
    section("Introduction",
        page("index",
            textfile("index.md"),
            ),
        ),
    section("Manual",
        preformat("Manual",
            ("Overview",   "manual.md"),
            ("Syntax",     "syntax.md"),
            ("Metamacros", "metamacros.md"),
            ),
        ),
    section("API",
        packagemodules(Docile),
        ),
    )

save(markdown("out", out); remove_destination=true)

e.g. Low-level partial package documentation

out = document("TestDocumentation",
    section("Introduction",
        page("index",
            textfile("index.md"),
            ),
        ),
    section("Manual",
        page("Overview",
            textfile("manual.md"),
            ),
        page("Syntax",
            textfile("syntax.md"),
            ),
        page("Metamacros",
            textfile("metamacros.md"),
            ),
        ),
    section("API", config(style_title="#", style_header="##", style_subheader="###"),
        page("Docile", config(style_subheader="*"),
            title("Docile"),
            header("Modules"),
            objs(Docile, :module),
            header("Methods & Macros"),
            subheader("Methods"),
            objs(Docile, :method),
            subheader("Macros"),
            text("Some intermediate text describing more about macros."),
            objs(Docile, :macro),
            ),
        page("Docile.Cache",
            title("Docile.Cache"),
            header("Modules"),
            objs(Docile.Cache, :module),
            header("Methods & Macros"; style_header="#"),
            subheader("Methods"),
            objs(Docile.Cache, :method),
            subheader("Macros"),
            text("Some intermediate text describing more about macros."),
            objs(Docile.Cache, :macro),
            ),
        page("Docile.Collector",
            title("Docile.Collector"),
            header("ALL documented objects"),
            objs(Cache.objects(Docile.Collector)),
            ),
        ),
    )

save(markdown("out", out); remove_destination=true)

[MichaelHatherly]

Let's not use an Experimental folder for this like we did for Docile -- especially since the markdown folder's been duplicated. Makes for quite a lot of changes to review.

[peter1000]

hi,

updated PR without that Experimental folder: still a good number of changes.

• only changes in folders: src/examples: left all others as they are so that there are not too many additional changes at once.
• the two examples in: folder examples are runable but there are still missing features: e.g. no index, no rendered output of objects, no permalink etc.

---

• folder static should be removed and if needed readded when html is redone
• folder test should be cleaned out ready for new tests: using [skip ci] for now
• folder docs should probably also be cleaned and redone as code is added.

---

maybe adding an Experimental branch where things could be merged into would be good?

I could rebase this than against the empty branch? might be easier.

Maybe remove all the old stuff and only add new clean once as we go along. When most of the planned feature work replace the current Lexicon at once?

[peter1000]

I added a low level type: ObjList which I thought should be able to hold any mixture of unfiltered/filtered - sorted/unsorted documented objs and later render them for the selected 'save' format.

e.g.: select all objects of 5 very small modules. something like:

objects = []
for m in  [mod1, mod2, mod3, mode4, mode5]
    append!(objects, Cache.objects(m))
end

QUESTION: is there a way to get the meta or without having the reference to the module I would like to too later somthing like:

for obj in objects
      obj_meta = getmeta(obj)
      obj_paresed = getparsed(obj)
end

ERROR: LoadError: MethodError: `getparsed` has no method matching getparsed(::Method)

I know I can append to the objects = [] also a tuple (obj, modname) and we did something like that a lot in the old Lexicon - but I just wonder if you have an idea for something like the above.

[peter1000]

Split it into 2 separate commits so it easier to view.

[MichaelHatherly]

maybe adding an Experimental branch where things could be merged into would be good?

Well, this is a branch for experimenting already. Do we need a different one rather? One in this repo rather than just on your fork?

Maybe remove all the old stuff and only add new clean once as we go along.

The current api has to remain intact until a 0.3 tag to allow for a deprecation period, so everything really should remain intact if at all possible.

is there a way to get the meta or without having the reference to the module

Everything expect for :typealias and :global should be possible currently. I've though of maybe storing those two as (ModuleName, :myconst) rather than just :myconst in Docile -- might make things a bit easier for this.

I'm not sure about all the title, header, objs, etc. I'd rather have as few node names as possible. (Without loosing expressiveness though.) I'll have a think about it.

[peter1000]

Well, this is a branch for experimenting already. Do we need a different one rather? One in this repo rather than just on your fork?

I think it would be better do have an official branch where approved parts can be merged in? But I'm fine with the current option too?

The current api has to remain intact until a 0.3 tag to allow for a deprecation period, so everything really should remain intact if at all possible.

I did not express myself well: I meant: in an Experimental branch start from scratch and only make a switch if all needed functionalities are implemented. Does not matter to much anyway.

I'm not sure about all the title, header, objs, etc. (Without loosing expressiveness though.)

I had thought also maybe have for all the content type only one content or whatevername. But somewhere the distinction should come in that a header is rendered automatically with the config header setttings and so all the others.

[peter1000]

Everything expect for :typealias and :global should be possible currently. I've though of maybe storing those two as (ModuleName, :myconst) rather than just :myconst in Docile -- might make things a bit easier for this

seealso User experience: get metadata

---

I started on auto-generating the proper mkdocs.ymlfiles and will later start playing with the rendering of objects (and signature)

[MichaelHatherly]

I'm not sure about all the title, header, objs, etc. I'd rather have as few node names as possible. (Without losing expressiveness though.) I'll have a think about it.

More specifically:

textfile("filename.ext") should really just be "filename.ext" and let Lexicon find the actual content by itself. Just like Docile does with docstrings.

text("...") could just be "...". Any specific reason for using text?

objs(Docile.Cache) should be Docile.Cache and provide additional filtering and sorting via config to the enclosing section or page.

If we rather use a :format => MarkdownFormatter like with Docile's metadata as part of each node's config then I don't think we need header(...), subheader(...), etc. They could just be specified as "# ...", "## ...", etc. and parsed internally. That also removes the need for config(style_header...) I'd think.

If I've missed the point completely on any of these do let me know. I have quite a few other commitments stealing my attention currently :)

[peter1000]

I will be of to lunch - I will look at it later or tomorrow.

[MichaelHatherly]

I think it would be better do have an official branch where approved parts can be merged in? But I'm fine with the current option too?

Here's a nearly blank slate to work against. https://github.com/MichaelHatherly/Lexicon.jl/tree/lexicon-next

I've done all the major deleting so PRs to that branch should be easier to review now.

[MichaelHatherly]

I've pushed a sketch of module structure the lexicon-next. I guess you'll mostly be working in Elements and Render for the moment?

[peter1000]

I've pushed a sketch of module structure the lexicon-next

:+1:

I will re-try a new version with the suggestion of yours: removing the title, header, objs, etc. and see how that looks like: probably by later tomorrow I shall report back..

[peter1000]

objs(Docile.Cache) should be Docile.Cache and provide additional filtering and sorting via config to the enclosing section or page.

I re did most of it as per your suggestion: except I wanted to keep the config options for the content items and not only to the enclosing section or page.

Anyway I will split it in a couple of commits and update the PR in a couple of hours after lunch.

---

just a quick side question: is there an easy way to write to file the result of the markdown of getparsed(module, obj) without the old entries?

[MichaelHatherly]

just a quick side question: is there an easy way to write to file the result of the markdown of getparsed(module, obj) without the old entries?

Does

text = "# ..."
ast  = Markdown.parse(text)
open("out.md", "w") do f
    writemime(f, "text/plain", ast)
end

do what you're wanting?

[peter1000]

do what you're wanting?

thanks a lot - was looking for such.

[peter1000]

New PR against the new brach lexicon-next you created.