Allow "Packages/Plugins" for Franklin

I'm not clear if this functionality is available or not. If it is, then I haven't been able to find a way to implement it.

I effectively wrote a small package for Franklin. It uses custom julia code (lx_* functions), franklin-markdown (newenvironments and newcommands), and css scripts. It allows for theorems using a syntax reminiscent of amsthm in LaTeX, including automatic numbering, labeling, and referencing. It is not clear that this belongs on the Franklin.jl repository.

I would like to make this open source, but I don't see any way to make this installable for other users.

Ideally, I would like to have a folder like _franklin_libraries/ or _fkl_libs/. Within that folder, I should have a directory custom_package_name\, with julia scripts and franklin-markdown scripts in it. Those scripts are evaluated before utils.jl and config.md. This would make sharing/installing packages very easy on the user end. From what I can tell, nothing like this is implemented.

Progress on a workaround:

I would be happy if there was a way for me to just share a folder full with one .jl, one .md, and one .css file, such that other people could write "include statements" somewhere in their code that included these libraries. For instance, I have code in three places:

utils.jl I can move the custom julia code to a new file called "custom_package.jl", and simply write include("custom_package.jl") in the utils.jl file. This is a nice way to "install" the julia script without having to copy-and-paste my full julia code into the user's utils.jl file. This is fine.
_css/custom_package.css The person installing a package just needs to move a css file into the css directory and update style.html. This is fine.
config.md I did not find any way to pull my package's config.md definitions into a new file. Rather, I did not find any way to include some file called custom_package.md into my config.md file. This would force users to copy-and-paste my whole franklin-markdown code into their config.md file, which seems like a real problem.

In order to get third-party / open-source packages, in the short term, there should be some easy way to include one markdown folder into the config.md file. This specifically has to support \newcommand and \newenvironment codes, which would be made available to all pages.

\textinput didn't seem to do this. I also tried writing a custom lx_* function that ran convert_md with isconfig=true, but this didn't work either.

Help on establishing a way for people to create open source and easily installable plugins would be greatly appreciated!

Hello! this is great and exactly what I was hoping that people would ultimately build (there is another such "plugin" package being built by @kellertuer: https://github.com/kellertuer/FranklinLiszt.jl) and I'm definitely keen to help such plugins be easily available to users.

Current high-level perspective

devs make a package "PluginX.jl" which exports a bunch of lx*, hfun* and env* functions, say for instance lx_theorem and register it
users just put using PluginX in their utils.jl
users can call all of those function, e.g.: \theorem{...}

Note: maybe it helps that these packages would all be named FranklinPluginX.jl 🤷

An example of this is the FranklinLiszt package above, and also the meta package https://github.com/tlienart/FranklinUtils.jl which helps defining "Julia-function-like" syntax such as:

\command{opt1=value1, opt2=value2, ...}

what I mean here is that the utils.jl file is meant to be the entry point for whatever custom code, be it actual commands (lx*, hfun*, env*) or "meta" commands which help define such commands.

Here's a (WIP) example where I use FranklinUtils for instance: https://github.com/tlienart/ft-academic-resume/blob/master/utils.jl . FranklinUtils doesn't export any lx*, hfun* or env* but does export macros that make their definitions easier.

In your case you could have for instance FranklinAmsMath.jl with (say)

module FranklinAmsMath
using FranklinUtils # some of these might be convenient to use but you don't have to

export lx_theorem, env_definition

@lx function theorem(; name=..., color=...)
 ...
end

@env function definition(md; name=..., color=...)
...
end

end

Custom CSS

If your package requires custom CSS, then, a bit like the latex .sty file, this one would have to be added in the _css folder and included in the _layout/head.html for instance.

Making stuff easier for the user

Maybe one thing we could do is have the statements be in config.md indicating that the user wants to use package XYZ and Franklin would look at the statement and potentially ensure that the files and statements are at the right place but I'm not yet convinced this simplifies stuff enough to warrant the effort.

Does that kind of make sense? Also if you point me to the repo with your code, I will gladly give this a shot with you so we can see whether there's anything specific beyond what's mentioned here that should indeed be added.

In any case, thanks a lot for working on this!

Ah sorry you do mention one additional thing which is for people to be able to inject "custom config" in the config.md. Hmmm ok I see what you're saying, I'll think a bit about that one to see what would be the best way forward. In theory you could "just" have a function append "your" config.md to the current one (a bit like Unix programs do the the .bashrc file) but this might require a bit more thinking. I'll re-read what you suggested and mull over this a bit.

Thanks for responding so fast!

I'm all for having a script that appends a plugin's config.md into the true config.md.

This is what I spent most of my time trying to do, since using and include make the Julia side easy, and adding CSS files is easy in Franklin.

In case it's helpful, here's three different approaches I tried, none of which worked. It's not clear to me why any of these didn't work, and they all just produced the same error message:

Franklin.LxObjError("Command or environment 'definition' was used before it was\ndefined.")

That is, all of these scripts ran, but none of them brought new definitions into scope.

In all cases, I have my custom franklin-markdown file at _fkl_libs/thm_config.md

Attempt 1: \textinput{_fkl_libs/thm_config.md} I tried adding this to either the config.md file, or at the very top of my actual webpage.md file, but neither case worked. I didn't find any very clear documentation saying if \textinput is supposed to be fully parsed by Franklin, so I don't know if this approach made sense at all.

Attempt 2: Returning a body of franklin-markdown text with an lx* function I tried to just paste the contents of the plugin's franklin-markdown file into the config.md by using an lx* function:

function lx_usepackage(lxc, _)

    # Parse Inputs
    args, kwargs = lxargs(lxc)

    # Grab the given name of the package, like "thm"
    package_name = string(args[1])

    # Make a path out of the package name, like "./_fkl_libs/thm_config.md"
    config_path = "./_fkl_libs/" * package_name * "_config.md"

    # If the file exists
    if isfile(config_path)
        # Print a debug line (to the command line) proving that the file was found
        print("Loaded!\n")
        # Return the file, as a string
        return read(config_path, String)
    else
        # Print an error (on the command line) if the file does not exist
        print("Error!\n\n")
    end
end

So that the command \usepackage{thm} would just return the complete contents of _fkl_libs/thm_config.md.

Attempt 3: Running Franklin.convert_md I tried to replicate the way that Franklin approaches the config.md parsing, by slightly modifying the definition of process_config:

function lx_usepackage(lxc, _)

    # Parse Inputs
    args, kwargs = lxargs(lxc)

    # Grab the given name of the package, like "thm"
    package_name = string(args[1])

    # Make a path out of the package name, like "./_fkl_libs/thm_config.md"
    config_path = "./_fkl_libs/" * package_name * "_config.md"

    # If the file exists
    if isfile(config_path)
        # Run the same subroutine as process_config()
        Franklin.convert_md(read(config_path, String); isconfig=true)
        # Print a debug line (to the command line) proving that convert_md has run
        print("Loaded!\n")
    else
        # Print an error (on the webpage) if the file does not exist
        return "~~~ERROR : " * config_path * "~~~"
    end
    return ""
end

So that in my config.md file can simply include the line \usepackage{thm} to (hopefully) import the plugin's franklin-markdown data.

In all cases, I don't get any error messages besides

Franklin.LxObjError("Command or environment 'definition' was used before it was\ndefined.")

So I'm unclear if there's anything obvious I'm missing or doing wrong, of if some aspect of Franklin's parsing of \newcomand makes the above approaches fundamentally doomed.

Ok, here's a working example that you can try to copy, please let me know how it works for you: https://github.com/tlienart/FranklinPluginExample.jl

To use this a user would have to add in utils.jl:

using FranklinPluginExample

FranklinPluginExample.plugin_config()

(after installing the relevant plugin package, in this case, you have to clone it bc I didn't register it)

The only extra step would be to add the CSS if there's specific CSS required, that's pretty easy though.

Would you mind trying this out and letting me know what you think?

Note: this is very close to what you tried to do, the only difference is that you might not have been aware of when global variables are reset, which likely caused the issues you saw.

Note2: there's a remaining question about ordering, here the stuff that's defined in the plugin config.md would be loaded after the user's config.md. This may not be desirable in all generality bc that would mean that the imported config could overwrite some of the stuff the user defined. E.g. if you define a command \R and the user wants a different one; yours will be the one that's used. This is easy to fix though, we just have to load the utils.jl before the config.md i.e. swap those two lines: https://github.com/tlienart/Franklin.jl/blob/4a870b9e62f487902696189bcc7b94d61381aa7c/src/manager/franklin.jl#L208-L209; I'll do that and test it separately.

This is beautiful! This worked like a charm! I'll get to work on making my franklin package now!

Now that this is clearly possible, I'd like to follow up with a less important "due process" type question:

The source for process_config sets FD_ENV[:SOURCE] = "config.md". If I understand correctly, doing this ourselves should correct what files are printed in Franklin's error messages? Should we do this for both the .jl and .md files?

In fact, the plugin_config function you wrote looks almost exactly like process_config. Should file_utils.jl just be updated to include this as a built-in function? In principle, this would hide any bookkeeping (like FD_ENV[:SOURCE]), and allow the error checking to be more consistent (like calling an equivalent of config_warn).

Syntax wise, in utils.jl, the user would write something like

using FranklinPluginExample
Franklin.include_external_config(FranklinPluginExample.config_path())

Where FranklinPluginExample.config_path() just returns a path to its config.md file. Further, this would allow people to import other config files they might be using (like a really long tex-style preamble full of math definitions).

I don't see any clear downsides to implementing something like this unless Franklin has some other plans for plugin syntax down the road. I'm new to Franklin, so I hesitate to implement this without asking first....

If I understand correctly, doing this ourselves should correct what files are printed in Franklin's error messages? Should we do this for both the .jl and .md files?

Yes that's correct. It's mostly relevant for the .md file though (if we work with the assumption that plugins aren't buggy 😛)

I don't see any clear downsides to implementing something like this unless Franklin has some other plans for plugin syntax down the road. I'm new to Franklin, so I hesitate to implement this without asking first....

Your suggestion makes sense to me. What I'd suggest is that you do this on a branch of Franklin, work on your plugin until you're happy about the ways both work together (feel free to ping me from that repo) and then we can merge the required changes inside Franklin?

Thanks!

Alright, I'm pretty happy with the core of it:

My plugin
The Franklin fork & branch with the new function, with pull request #781.

There's a couple other points where I wonder if existing tools would make it easy to improve the experience a bit:

1. Automatically reloading when plugins are edited.

When editing a plugin's markdown or julia files, Franklin does not automatically recognize the edits to those files, so I have to restart the server every time I want to test an edit to my plugin. It's a minor inconvenience for plugin developers. This uses using PackageName in utils.jl, so I'm not sure if something like Revise.jl would be needed/helpful?

2. Javascript workflow

Even in the long term, it makes sense for the CSS file to not be automatically included by the plugin, since I would expect people to fine-tune CSS to their specific webpage styles. This is less obvious to me from a Javascript perspective though.

Any idea on a nice way to do this in the short term? My current best solution is to include the javascript at the end of a webpage explicitly with a \newcommand that just returns a <script>...</script> block. I suppose I could also use the if-end blocks in foot.html. This requires the user to do more work, which I would rather avoid.

Long-Term Plugin Support

This also got me thinking about the long-term infrastructure for plugins. How can we make Franklin handle most of this setup for us? One idea is to have plugins provide a function plugin_setup_config(), which returns a dictionary that defines what files should be added & edited in order to enable a plugin. Then, when a user decides they want to include a plugin, they navigate to the Franklin site directory, and in their julia repl they run

using Franklin
using FranklinPluginName
Franklin.install_plugin(FranklinPluginName.plugin_setup_config())

Then, Franklin would read this dictionary, and perform initial setup needed to enable a plugin. For instance, this would:

Add the import lines to the utils.jl
Add the default CSS script to \_css\
Add the default javascript code to \_libs\
Update the \_layout\ directory to include the javascript (maybe? not sure if this is possible in general. maybe just create a new file analogous to foot_katex.html?)
Other things I haven't thought about yet

Any thoughts on this long-term design of/for plugins?

tlienart / Franklin.jl