Homebrewery API

[JosefUtbult]

I would like to integrate homebrewery with other editors, an Obsidian plugin in my case as that is where I personally create my adventures. For that I would like to have an API where I can send a request with the markdown content and some metadata, and get back either a rendered website (HTML with CSS links to the homebrewery site, and not local files) or a PDF.

I can take on this task by myself, but I would like to know if this is something that would be desired to have, or if you guys think it’s unwanted. I’m in the developer chat as @CPUToast and can also be contacted here. I will try to start up a conversation about the idea, and would appreciate some feedback.

[G-Ambatte]

I hadn't heard of Obisidian before; I'd seen references to people trying to integrate the Homebrewery standalone CSS into the Atom editor, and there is a fork to use Homebrewery as an Electron app.

It should be possible to duplicate the existing download function, modifying it to instead send the raw HTML text of a rendered brew; similarly, it should be possible to call this function via a new route that processes POST'd data without saving it to the local database.

Hypothetically I think it's certainly possible. Whether it's something that's desirable for the main project is another question entirely - @calculuschild is probably best positioned to answer that.

[calculuschild]

We have actually been trying to remove all Markdown parsing from the server-side and moving that to the client, because we've occasionally had cases where a user has malformed markdown that is broken in just the right way that it crashes our server. Better to have that one person's browser crash.

Having an API return parsed markdown would require undoing a lot of that and re-exposing our server to malicious markdown. In other words, we don't want to execute foreign code on our servers if we can help it.

There may be some way to send the markdown parser itself back as part of the API response, which is then executed on their end, but I'm not really sure if that's feasible.

Those are my initial thoughts anyway.

[calculuschild]

Could be we create some sort of bundled "Mini homebrewery script" that people could import not from an API but just as a script they can embed on their page that basically just contains Marked plus the modifications we made.

[JosefUtbult]

That’s an interesting idea. I can look into it a bit and see if I can figure out a way to get a parser sent to a client. If the markdown parsing is already moved to client side, it feels like it shouldn’t be that hard to get it sent over a request. In my mind it should be quite similar to getting the parser in the homebrewery editor, but with a few less steps. I’ll get back to you if I figure something out, and we can discuss further from there. Is it possible to get the css from the site in some way, instead of it being local in the homebrewery editor?

[JosefUtbult]

I have successfully added the following to a fork.

• The homebrewery markdown renderer is packaged with all its dependencies into a standalone js file using Browserify. This single file is then accessible from localhost:8000/api/homebreweryRenderer.js. This file can either be downloaded or included in a html page, where it's functionality can be accessed.
• A json object containing all theme names, their paths and the names/paths of the themes snippets are available from localhost:8000/api/themes
• All themes css files are accessible from localhost:8000/api/themes/<theme path>/style. These stylesheets can also be included in a html file.
• All fonts are acessable from localhost:8000/api/themes/fonts/5e/<font name> and can be imported via the stylesheets
• All theme snippets can be requested to run on the server and its result is sent back in the request. These snippets are accessible from localhost:8000/api/themes/<theme path>/snippets/<snippets group name>/<snippet name>

My idea is that a client application can first download the standalone renderer. Then it can requests a list of all available themes. At the moment these are only the themes present in the source code, but if we implement user themes that are public, we can include them here to (and then is when this API structure will shine). When a user has selected a theme in their application, the client can request that themes stylesheet from the server. The application also has all available snippet names for that theme, from the list of themes, that the application can display in some nice menu. When a user selects a snippet, the application will be able to request the result from that snippet from the server, which will execute the selected snippet and return the result which the application can display however it wants.

I'm still working on a few problems, but I have successfully made a demo application in nodejs that integrates with the homebrewery API. It looks like this at the moment.

Notice that this application doesn't have any internal knowledge of which themes that exists or what options should be available in the snippets menu. These things are accessed over the API from the homebrewery instance running on my computer.

There are still some things I think we need to discuss.

Should the standalone renderer be accessible from the homebrewery site? It is a quite large file as it contains the code for all required modules, so it might take to much bandwidth for our taste if a bunch of clients tries to load it all the time. It might be a solution to build the standalone renderer and put it in the github repo instead.

Is this solution with an API something we are interested in? I have proved it works, but is it something that we want to have on the homebrewery site?

Should we add functionality to access brews saved in homebrewery, and even let applications save and publish brews?

I'd appreciate some suggestions. If you want to try it out, you can get the homebrewery branch from my fork, and the API test application from my repository.

[calculuschild]

Would you like to turn this into a PR so we can more easily compare the changes you are proposing?

[JosefUtbult]

Absolutely! Just don't merge it yet, as its not completely finished.

[G-Ambatte]

I've opened PR #2268 so we can see the current status of the work. Looks like it was forked from master back in December, so there's a few changes to be applied, just to bring it up to date with the current master.

[JosefUtbult]

Thanks. I forked it a while ago, yes. The branch is made from SwappableThemes-ReorganizeFolderStructure, so it might be better to make the pull request towards that branch.

I'm working on getting everything merged to the latest version of SwappableThemes-ReorganizeFolderStructure at the moment, so i can get a real PR to work later.

[JosefUtbult]

I got everything merged to the latest version of SwappableThemes and made a pull request towards that, so that it is easier to see my changes. Check out PR #2269 to see the current status.

[calculuschild]

@JosefUtbult If you need, you can build off of the second-to-last commit https://github.com/naturalcrit/homebrewery/commit/442c5e2d8a84e35936f01868cbad00be64831aee

[JosefUtbult]

Great. Thats probably a good idea. I had to do some ugly fixes on the journal theme to get it to work (i didn't push them). But I'll see if I can get it back to that commit

[khalid-talakshi]

Question: I noticed the API idea changed from the start to where it is currently, and was wondering if you had any idea about how somebody could use this API to auto-generate a pdf from a markdown file? For context, I want to use GitHub actions to generate a pdf of a series of markdown docs.

[G-Ambatte]

Question: I noticed the API idea changed from the start to where it is currently, and was wondering if you had any idea about how somebody could use this API to auto-generate a pdf from a markdown file?

In effect, this is a callable parser that will allow Markdown files to be parsed using the Homebrewery-specific extensions and customizations, returning a HTML page that a browser (nominally Chrome) can interpret and display. In theory, a headless Chrome installation could be used to programmatically parse files and then push the result to a printer, and said printer could theoretically be a logical "Print to PDF" printer rather than a physical device.

However, such an installation would fall wildly outside the scope of the project - in fact, PDF generation alone falls outside the scope of the project, as this has always been handled by the browser's print functionality.

[ericscheid]

Theoretically, there is a spectrum of implementations of an API

1. client submits brew code → PDF response (not gonna happen)
2. client submits brew code → pure html response, rendered on server (undesirable load, risk of DOS vector)
3. client submits brew code → /print resource response, with brew and js embedded, which when executed renders html

Currently, the /print endpoint (e.g. https://homebrewery.naturalcrit.com/print/G5mKYTJircSY ) performs as (3) above (with a hidden quirk which is deprecated and due for removal).

Another approach is to have an API of submit brew code → get saved brew id, where that brew id could then be utilised via the existing /print/ endpoint.

4. client submits bew code → tiny json response containing a brewId, do with that what you will

This will have a side-effect of creating a saved brew somewhere (mongo, or gdrive), which is not wholly desirable because any API has the potential to create a huge amount of junk brews which eat into storage. This could be mitigated by requiring gdrive storage, but at the expense of needing to code the API to be aware of the API user's gdrive credentials (and coping with errors via the API, since gdrive errors won't be reported via the /edit/ UI).

On the gripping hand we could spin up an API endpoint that simply generates the current bundle.js resource but with a function other than start_app(), which when called with the brew code as a parameter (and some other parameters) will return a text result of the needful html (i.e. instead of injecting into <div id="brew"></div>).

5. client submits API credentials (but no brew) → code bundle with function for rendering brew locally

This last option (5) sounds easiest from both THB and client perspectives, and could also result in much less load on THB servers (since the bundle.js could be downloaded once and re-used repeatedly). Also, no need to HTTP-SEND an entire brew code, only for it to be simply ignored and round-tripped.

(The THB project could also be coded to be run from the command line, receiving brew code, emitting html code, and intended to be run locally. No comment.)

6. run as command line app on local machine

[G-Ambatte]

(The THB project could also be coded to be run from the command line, receiving brew code, emitting html code, and intended to be run locally. No comment.)

I mean, such a beast could theoretically be used to process the files created by using the "Download" option from the live site. It could need to return the HTML used to dangerously set inner HTML from the BrewRenderer, if I remember correctly.

[ericscheid]

Yup. The only saving grace for the headache of option (6) is that because it's running locally, the output can be fully rendered html (i.e. effectively "render on server", but where "server" is "your machine, not THB"). That saves a pump-brew-thru-browser-js step.

It also doesn't need to do any storage handling. It takes input, it produces output.

[JosefUtbult]

Sorry for late answer, but I've had a lot of other stuff. As I see it at the moment (and as I've implemented it) a client aren't able to request brew code to be rendered on the server. The idea is that the actual renderer itself is accessible for a client. So the result is that rather then a client sending a brew to the server, the client will request the renderer from the server and parse the brew code by itself. I do like the Github actions idea though @khalid-talakshi. I can look into if that is a possibility.

But as @calculuschild said a bit further up; we don't want to render clients brews on the server.

[JosefUtbult]

I had some time to implement my changes in the newest version of the main branch now! I made a new pull request if you want to look at it, and we can discuss it from there.

If you want to try it out, you can download that branch, add "enable_themes" : true in config/default.json and then access it through the Homebrewery API Example repository, which will act as a client.

[JosefUtbult]

I'm also working on an Obsidian Homebrewery plugin if that is of any interest for somebody

[Sparks01]

Did this every go anywhere? I've been searching for Obsidian Hombrewery integration. Thx.