[docs] show how to use Weave from Franklin

tlienart commented 4 years ago

basically similar to using Literate.

explain differences in the docs (weave is monopage, can output PDF, ...)

(could add that as example in the extended sandbox)

EvoArt commented 2 years ago

Is there any advantage to doing this (using weave) as opposed to just using live code evaluation with Franklin.

EvoArt commented 2 years ago

Now that I think of it, I definitely like the idea of keeping my reports as weavable jmds. But it would be nice to no how to do this within Franklin to have the nav bar etc.

tlienart commented 2 years ago

Is there any advantage to doing this (using weave) as opposed to just using live code evaluation with Franklin.

Currently there are two that I can think of:

you get a simple unified experience, you don't have to mess with HTML files etc, all reports will look good from the start
you can output a PDF

(1) is not really an issue with Franklin; you can use one of the template, those will get better as time goes (hopefully), and we could have one that's copied from Weave for single page stuff.

(2) is something that will eventually be addressed in Franklin, there's already a working prototype but it's very much not ready for a release.

Now that I think of it, I definitely like the idea of keeping my reports as weavable jmds. But it would be nice to no how to do this within Franklin to have the nav bar etc.

This you can already do. Basically Franklin can always do the following:

Here some markdown

{{my_insert some parameters}}

here some more markdown

the my_insert corresponds to a function you'd code yourself (hfun_my_insert, see this part of the docs if you're not sure what I'm talking about) and would insert arbitrary HTML. That HTML can, for instance, be generated by Weave.

So a viable path is to write such a function that gets a path to a .jmd file, calls Weave to transform it to HTML, possibly trims that HTML down (e.g. you'd want to remove the head and foot), and then plugs it in. It might seem more complicated than it actually is which is why I wanted to have a demo for it.

Same stuff for Pluto notebooks (#813 #833)

tlienart commented 2 years ago

This came up on Slack, @TheCedarPrince came up with the following solution:

function hfun_weave2html(document)
    f_name = tempname(pwd()) * ".html"
    weave(first(document), out_path = f_name)
    text = read(f_name, String)
    final = "<!DOCTYPE html>\n<HTML lang = \"en\">" * split(text, "</HEAD>")[2]
    rm(f_name)
    return final
end

TheCedarPrince commented 2 years ago

Thanks @tlienart - this solution right now has some issues as it stands:

Syntax highlighting is missing right now. I do not know how to bring back the highlighting at this moment.
As mentioned on slack, the footer is still here, but I think could easily be removed. I'll tinker and post back later. 😄

Glad this can serve as a first pass of a possible solution with Weave + Franklin compatibility!

tlienart commented 2 years ago

I don't know how weave encodes code blocks, you should check the produced html. If the code tags have no class then that's the issue, it needs to be given the class "language-julia" and then highlighting should work

TheCedarPrince commented 2 years ago

Tried taking another pass at this - maybe you have some thoughts @tlienart ?

Here is the code block I am trying to get properly highlighted:

using CSV
using DataFrames
using Latexify
using PrettyTables
using PyPlot
using Statistics
using TimeSeries
using Weave

Here is how weave renders blocks of code:

                <pre class='hljl'>
<span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>CSV</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>DataFrames</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>Latexify</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>PrettyTables</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>PyPlot</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>Statistics</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>TimeSeries</span><span class='hljl-t'>
</span><span class='hljl-k'>using</span><span class='hljl-t'> </span><span class='hljl-n'>Weave</span>
</pre>

Here is my attempt at trying to fix the HTML classes:

 
<pre class='hljl'>
  <span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>CSV</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>DataFrames</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>Latexify</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>PrettyTables</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>PyPlot</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>Statistics</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>TimeSeries</span><span class='language-julia'>
  </span><span class='language-julia'>using</span><span class='language-julia'> </span><span class='language-julia'>Weave</span>
  </pre>

Sadly, no syntax highlighting yet. Any thoughts as to what I need to change to make highlighting work?

TheCedarPrince commented 2 years ago

Ah figured it out! With a bit more tinkering, here is the new and improved function that:

Takes in a raw julia markdown file
Converts it via weave to HTML
Strips the weave formatting out from the HTML file head section
Strips the weave syntax for code blocks out
Replaces the weave code block syntax with Franklin code block syntax
Returns the final Franklin compatible HTML file into the call location
Makes a nice Franklin formatted page like you would expect!

Here is the new function; note that this is most likely brittle, could break easily, and is perhaps not the most robust/fast solution - but now I leave that to folks here to investigate and optimize as it works as is:

function hfun_weave2html(document)
    f_name = tempname(pwd()) * ".html"
    weave(first(document), out_path = f_name)
    text = read(f_name, String)
    final =
        "<!DOCTYPE html>\n<HTML lang = \"en\">" * split(text, "</HEAD>")[2] |> # Splits the weave document on the head block
        x ->
            replace(x, r"<span class='hljl-.*?>" => "") |> # Removes weave code block syntax
            x ->
                replace(x, "</span>" => "") |> # Removes weave code block syntax
                x ->
                    replace(
                        x,
                        "<pre class='hljl'>\n" => "<pre><code class = \"language-julia\">", # Replaces weave code block syntax with Franklin's
                    ) |> x -> replace(x, "</pre>" => "</code></pre>") # Replaces weave code block syntax with Franklin's
    rm(f_name)
    return final
end

Here is an example julia markdown file

# Hello World! 

```julia
2 + 2

The above was a cell written in Weave! How cool is that?


Here is how one could use it in creating a new Franklin page:

```julia
+++
title = "Weave Example"
hascode = true
date = Date(2021, 7, 15)
rss = "Weaving a tapestry of fun with Franklin and Weave.jl"
+++
@def tags = ["literate", "julia"]

{{weave2html path/to/file.jmd}}

Then here is how it could look like rendered:

Current limitations to this solution:

Didn't test other programming languages
Not extremely resilient or robust at this point
Creates a temporary file in the conversion process (more a limitation of Weave as I could not find a way to write to a buffer)
Didn't look into automatically updating the sidebar

If anyone wants to come after and improve on this, be my guest! But perhaps this addresses the issue folks were asking about? I am CC'ing @tlienart @EvoArt and @jvaverka to see if this looks good as a starting point.

jvaverka commented 2 years ago

Great to see this working end-to-end. Awesome work @TheCedarPrince !

This works for me given that the weave2html function is the only one on the page. When including this function to insert page tags the weaved doc does not render.

However, just as you said - not robust at the moment. Still great to be able to reuse my Julia Markdown Docs!

EvoArt commented 2 years ago

Thanks @TheCedarPrince. I cant seem to get it working though. Works fine when I use serve. But not when I push to github. Only works with serve if i include using Weave in utils.jl, but this makes the github deployment fail. Not sure what I'm doing wrong.

Did you test this with a github deployment?

tlienart commented 2 years ago

@EvoArt is Weave in your site Project.toml?

EvoArt commented 2 years ago

Yes, but on the remote repository, it's only in the master branch project toml.

EvoArt commented 2 years ago

It's just a toy site that I'm using to figure out Franklin. I think I'm going to clear everything out and start again. When I originally added using Weave I hadn't added it to my site toml. Think I may have messed something up by doing that, since now the deployment fails no matter what I do.

tlienart commented 2 years ago

if it still fails after trying this, ping me from the relevant repo and I'll take a look

EvoArt commented 2 years ago

Thank you @tlienart ! please see here

tlienart commented 2 years ago

I finally had a crack at this Weave thing,

here's a demo note that the top bit is Franklin Markdown up until the horizontal line then it's all weave generated, similar effect to what would happen if you used Literate (btw I still encourage people to just use Literate if they don't use Weave specific stuff like PDF output)

here's the function I used

here's the CSS I used this is just copy pasted from what Weave uses

Some notes:

I use PyPlot for the plots backend instead of GR; for some reason I had a QT issue with GR when using the GR backend and deploying on GitHub, maybe this can be worked around though it's not really an issue..
Weave comes with its own CSS that expects a single page layout, I copied some of it to get the same highlighting etc but if you want further correspondence with Weave CSS you have to do some more work, basically your options are:
- keep the whole weave page as a standalone HTML page that you link to, you place this HTML page yourself in the hfun_* in __site/some/appropriate/path.html
- do something like I did, and do more work in _css/weave.css to adjust smaller details, that's the path you should take if you want to use a Franklin layout around the weave content
- create a custom Weave template as per their docs : https://github.com/JunoLab/Weave.jl/blob/master/templates/md2html.tpl
Weave uses Mathjax, I didn't change that so the demo page uses mathjax as well, see also the conditional header: https://github.com/tlienart/Franklin.jl/blob/f4451675084c080c437bd5cc9a9e9a8df6cd4b86/demos/_layout/head.html#L13-L22

Steps required to get this to work:

in the deploy script, the matplotlib needs to be there so that you can use PyPlot (https://github.com/tlienart/Franklin.jl/blob/master/.github/workflows/deploy.yml#L18)
in the Project.toml, note that Weave and PyPlot are there (https://github.com/tlienart/Franklin.jl/blob/master/demos/Project.toml#L6, https://github.com/tlienart/Franklin.jl/blob/master/demos/Project.toml#L8)
in the utils.jl there's the using Weave obviously https://github.com/tlienart/Franklin.jl/blob/f4451675084c080c437bd5cc9a9e9a8df6cd4b86/demos/utils.jl#L9
in the config.md declaration of a variable weave = false which is set to true in the weave.md page.

That's it, no other fancy business required.

TheCedarPrince commented 2 years ago

Hey @tlienart - I just tried out your solution for Weave.jl-based Franklin websites and I have to say, "IT WORKS!" This is fantastic! Thank you so much for helping us all with getting it working. I haven't yet deployed the Franklin website to a service such as GitHub Pages, but with your directions, it seems very straightforward! Here is how it looks on my end:

tlienart commented 2 years ago

tenor

EvoArt commented 2 years ago

Thanks all. This works for me! I've tried tinkering with the css with some success. The only thing I cant figure out is how to wrap code inside of code blocks. I've tried replacing overflow-x: auto; from line 6 of weave.css with overflow-wrap: break-word; (and other variants) but just end up with text continuing outside of the code block. Is it possible that css/style decisions in other files are overriding weave.css?

tlienart commented 2 years ago

could you deploy this on a draft page so that we can inspect the CSS? + show a screenshot here?

One way to force things to be exactly as you intend them in CSS is to add !important after the property. It's not very nice but it usually does a good job.

tlienart / Franklin.jl

[docs] show how to use Weave from Franklin #372