tlienart / Franklin.jl

(yet another) static site generator. Simple, customisable, fast, maths with KaTeX, code evaluation, optional pre-rendering, in Julia.
https://franklinjl.org
MIT License
963 stars 114 forks source link

[FR] Allow for on-the-fly eval of code blocks + relative paths improvements #152

Closed tlienart closed 5 years ago

tlienart commented 5 years ago

See https://github.com/tlienart/JuDoc.jl/issues/147 for context

With respect to scripts, I think a simple solution to keep things organised would be to allow

code (executed):
```julia:phys/script1
println(exp(1im*pi))

input: \input{output}{phys/script1}

code (non-executed):

println("hello")

The second one would amount to the "no-eval" option you're mentioning and doesn't create a file. The first one would indicate a relative path in `assets/scripts` so that things stay well organised within the assets folder:

. ├── infra │   ├── favicon.png │   └── ... └── scripts ├── output │   ├── phys │      └── script1.txt ├── phys    └── script1.jl


I'm more in favour of all this happening in the `assets/` folder but I guess an extra syntax like 

`````markdown
code (executed):
```julia:./code/script1
println(exp(1im*pi))

input: \input{output}{./code/script1}


could indicate that `script1` is contained in a local subfolder (which seems to be what you want) i.e. the structure would be

src └── pages └── phys ├── page1.md    ├── code   │  └── script1.jl └── output # generated └── script1.txt

Invarianz commented 5 years ago

First, thanks for clarifying the copy mechanism when putting assets into webpage/src/pages instead of webpage/assets/ it is now clear that it is much more sensible to put images, pdfs, etc. into webpage/assets/. Now that I thought about reorganizing my webpage for a bit I came up with the following solution influenced by my own experiences. Usually single webpages do not have that many images, code or maybe PDFs attached to them. When the webpage gets too long one tends to split logical parts into separate webpages. Let me show this with my specific example:

src/
├── index.md
├── config.md
├── _css/
│   └── ..
├── _html_parts/
│   └── ..
└── pages/
    ├── julia.md
    ├── judoc.md
    ├── physics.md
    ├── rpg.md
    ├── julia/
    │   └── tips_tricks.md
    ├── physics/
    │   ├── math/
    │   │   └── statistics.md     
    │   ├── tests/
    │   │   ├── hubbard_model.md
    │   │   └── interacting_dot.md
    │   └── theory/
    │       ├── cdw.md
    │       ├── hartree_fock.md
    │       └── tdhf.md
    └── rpg/
        └── broken_hope.md

To keep the logic in the webpage/assets/ folder I would propose a structure that reflects everything happening below src/ i.e. compared to the above folder structure the structure in webpage/assets/ would look like this:

assets/
├── index/
│   └── .. (All images, pdfs, scripts in index.md)
└── pages/
    ├── assets/
    │   ├── judoc/
    │   │   └── .. (.. in judoc.md)
    │   ├── julia/
    │   │   └── .. (.. in julia.md)
    │   ├── physics/
    │   │   └── .. (.. in physics.md)
    │   └── rpg/
    │       └── .. (.. in rpg.md  )
    ├── julia/
    │   └── assets/
    │       └── tips_tricks/
    │           └── .. (.. in tips_tricks.md)
    ├── physics/
    │   ├── assets/
    │   │   └── (empty)
    │   ├── math/
    │   │   └── assets/
    │   │       └── statistics/
    │   │           └── .. (.. in statistics.md)
    │   ├── tests/
    │   │   └── assets/    
    │   │       ├── hubbard_model/
    │   │       │   └── .. (.. in hubbard_model.md)
    │   │       └── interacting_dot/
    │   │           └── .. (.. in interacting_dot.md)
    │   └── theory/
    │       └── assets/
    │           ├── cdw/
    │           │   └── .. (.. in cdw.md)
    │           ├── hartree_fock/
    │           │   └── .. (.. in hartree_fock.md)
    │           └── tdhf/
    │               └── .. (.. in tdhf.md)
    └── rpg/
        └── assets/
                └── broken_hope/
                    └── .. (.. in broken_hope.md)

So basically every markdown page in the src/ folder gets its own subfolder in the assets/ folder and with this one can reconstruct the layout of the src/ folder. Now I believe that having this per page ordering negates the need for specific scripts/ img/ etc. folders in assets and also follows the logic of the src/ tree. So now one can just dump everything related to the markdown page into the specific folder and if needed can still create a substructure inside the per page folder (if one needs to further split between scripts, images etc.). The only problem I see with this approach is that the links to files inside the markdown document could become quite long. This could be circumvented if every link in the markdown document (to scripts, images etc.) would point to its respective subfolder. E.g. if i write ![Charge density wave](peierls.png) in cdw.md it automagically resolves to webpage/assets/physics/theory/assets/cdw/peierls.png. Or one could add a special variable for the subfolder belonging to a markdown page e.g. ![Charge density wave](ASSETSDIR/peierls.png). The same approach could be applied to scripts.

I hope this post was not too long, I wanted to make sure to cover weird corner cases with subfolders.

tlienart commented 5 years ago

Ok thanks for the insight, what you're suggesting makes sense but I'll need to think a bit about how to do this best. Ideally I'd like to not impose a folder structure too much onto people as some people (e.g. you, possibly me) like a lot of logic in their folders and others might not want to bother.

One thing as well is that two pages may call the same assets and this should be easy to do.

Maybe one approach could be to keep things as they are but offer a slightly different syntax for "relative paths" that would correspond precisely to what you suggest so you could have something like:

![Charge density wave](/assets/physics/theory/cdw/peierls.png)  <!-- the current way -->

![Charge density wave](./cdw/peierls.png) 

in the second case, there would be the assumption that you have assets/[subpath]/peierls.png where subpath would match the subpath in pages/[subpath]/cdw.md.

So to make things explicit:

.
├── assets
│   └── physics
│       ├── maths
│       └── theory
│           └── peierls.png
└── src
    ├── _css
    ├── _html_parts
    └── pages
        ├── index.md
        └── physics
            ├── maths
            └── theory
                └── cdw.md

If you wanted to further nest things in assets so that they're in their own cdw folder, you would write ![Charge density wave](./cdw/peierls.png) with this tree:

.
├── assets
│   └── physics
│       ├── maths
│       └── theory
│           └── cdw
│               └── peierls.png
└── src
    ├── _css
    ├── _html_parts
    └── pages
        ├── index.md
        └── physics
            ├── maths
            └── theory
                └── cdw.md

What do you think?

A side note

as a side note, there is a simple way to already do all of this which you may want to use and it's to just define a command.

\newcommand{\imcdw}[2]{![#1][/assets/physics/theory/cdw/!#2]

And then you can just do

\imcdw{peierls.png}

It's not exactly what you're asking for but it may help you to currently avoid having to write paths everywhere :) (I use something similar to have simple between-pages links in my website:

\newcommand{\cvx}[1]{/pub/csml/cvxopt/!#1}

Something that we [saw before](\cvx{mda.html})

which seemed nicer than to have to write [saw before](/pub/csml/cvxopt/mda.html) .

Invarianz commented 5 years ago

I think the "dot" syntax i.e. ![Charge density wave](./cdw/peierls.png) is an excellent solution that would keep the freedom of structuring the assets/ folder and still offer the convenience of using the "logical" approach. However, there is one small problem. I believe that one has to create sub assets/ folders (in webpage/assets/) to avoid name clashes between markdown page names and subfolder names in the src/ tree. E.g. in the folder tree I have posted above I have a pages/physics.md webpage and a subfolder called pages/physics/ where physics.md is basically the "sub landing page" for that particular folder. That is why, for every subfolder below src/, I create an additional assets/ folder in webpage/assets/ where I then create folders for every markdown page. E. g. for the name clash problem above, the markdown page data is in webpage/assets/pages/assets/physics/, whereas the subfolder src/pages/physics/ is related to webpage/assets/pages/physics/ .

Thanks for the side note, I did not think of using LaTeX to partly solve this problem, I will certainly use it. :)

tlienart commented 5 years ago

E.g. in the folder tree I have posted above I have a pages/physics.md webpage and a subfolder called pages/physics/ where physics.md is basically the "sub landing page" for that particular folder.

Ok so if I understand your context properly, you could have:

.
├── assets
│   ├── im_index_1.png
│   └── pages
│       ├── im_phys_1.png
│       └── phys
│           └── math
│               └── im_noether_1.png
└── src
    ├── index.md
    └── pages
        ├── phys
        │   └── math
        │       └── noether.md
        ├── phys.md
        └── rpg

And with the dot syntax you'd do:

In `index.md`
![](./im_index_1.png)
![](./pages/im_phys_1.png)

In `phys.md`
![](./im_phys_1.png)
![](./phys/math/im_noether_1.png)

In `noether.md`
![](./im_noether_1.png)

If you wanted something "higher up" you'd have to write the full path 
(we could allow `..` but I doubt it would be clear)

![](/assets/pages/im_phys_1.png)

And if you want even more structure in assets, well you can always add subfolders; the dot syntax would just amount to mimicking the path to the original .md file and you could add whatever path you wanted after that. So for instance you could alternatively use

.
├── assets
│   ├── im_index_1.png
│   └── pages
│       └── phys
│           ├── im_phys_1.png
│           └── math
│               └── noether
│                   └── im_noether_1.png

and then you'd do

In `phys.md`
![](./phys/im_phys_1.png)

In `noether.md`
![](./noether/im_noether_1.png)

I think both approaches are reasonable as long as you stays consistent with one or the other? what do you think? I'll try to work on this soonish

Thanks for the side note, I did not think of using LaTeX to partly solve this problem, I will certainly use it. :)

The idea for allowing the definition of LaTeX-like was precisely to allow people to basically write whatever shortcuts they would see helpful so that they could think primarily about the content and not have to bother too much about weird syntax or paths or whatever more than once 😉

Invarianz commented 5 years ago

So before I go on, thanks again for the fast replies and active development of the package. I think this is getting better everyday :).

To come to your answer: It is not quite what I had in mind but I think we could discuss the pro and contras of the approach you offered and mine.

So let me start with the problem I see with your approach and let me take your folder tree as an example and add a second page into pages and remove the empty rpg folder:

├── assets
│   ├── im_index_1.png
│   └── pages
│       ├── im_phys_1.png
│       ├── im_julia_1.png
│       └── phys
│           └── math
│               └── im_noether_1.png
└── src
    ├── index.md
    └── pages
        ├── phys
        │   └── math
        │       └── noether.md
        ├── phys.md
        └── julia.md

The only problem I have with doing this directly is that page content and the subfolders mimicking the structure in src/ get mixed up. The proposal I made created one more folder for all the content:

├── assets
│   ├── im_index_1.png
│   └── pages
│       ├── content
│       │   ├── im_phys_1.png
│       │   └── im_julia_1.png
│       └── phys
│           └── math
│               └── content
│                      └── im_noether_1.png
└── src
    ├── index.md
    └── pages
        ├── phys
        │   └── math
        │       └── noether.md
        ├── phys.md
        └── julia.md

So if you e.g. write ![](./im_phys_1.png) in phys.md it points to /assets/pages/phys/content/im_phys_1.png instead of /assets/pages/phys/im_phys_1.png. In the end this boils down to a more or less stylistic choice on the order in the /assets/ folder. With your syntax I could still write ![](./content/im_phys_1.png) which would not make the relative path to type much longer but, as mentionend, content and subfolders would get mixed as a "default".

On the other hand your syntax has the advantage that I could refer to images further down in the source tree, e.g. I could write ![](./phys/math/im_noether_1.png) in phys.md. With the approach I have proposed and since there is no .. operator, this would only be possible by using the full path.

I think you need to decide what suits your package best. :)

tlienart commented 5 years ago

We're on the same page it seems (and thanks too for the back and forth, it's very helpful).

Right so we have two possible approaches: first is that the dot syntax called from a page [subpath]/page.md would map to

/assets/[subpath]/

second is that it maps to

/assets/[subpath]/page_assets/

where the added _assets would avoid the name clash you raised earlier.

I think I'm leaning towards the first one just because it seems (to me) that suggesting identical structure for the source folder and the assets folder is appealing.

One way to have our cake and eat it: I understand what you're saying with "get mixed as a default", but users who do not want this to happen could do one thing that's not far from the reverse of what you were suggesting but that, I think, is more consistent: just put noether.md in its own folder.

├── assets
│   ├── im_index_1.png
│   └── pages
│       ├── im_phys_1.png
│       ├── im_julia_1.png
│       └── phys
│           └── math
│                └── noether
│                   └── im_noether_1.png
└── src
    ├── index.md
    └── pages
        ├── phys
        │   └── math
        │       └── noether
        |            └── noether.md
        ├── phys.md
        └── julia.md

what do you think? (of course there remains a possibility for mess for the "landing pages" (phys.md and julia.md but I would tend to think that these landing pages will typically not have a crazy number of assets? and they could also have their own folder in assets I guess or be in their respective folder (so move phys.md to phys/phys.md)

Ultimately I think I'd like to suggest an "intuitive" default and offer the possibility for whatever customisation either through latex commands or through the config file which could determine how this works for instance (but I'm a bit reluctant to have the config file be too complicated).

PS: I've started work on the eval of code blocks, will probably split this into another issue or whatever but there will be news soon.

Invarianz commented 5 years ago

Since the drawbacks of your approach are very, very minor and your approach offers more flexibility I would also lean towards your suggestion to map [subpath]/page.md to /assets/[subpath]/. From this point onward the user, depending on his demands, can order files in [subpath] as they wish and create subfolders. As you said, the landing page issue can also be solved by the user; adding an additional subfolder in /assets/[subpath].

I am really looking forward to the evaluation feature, this will also draw interest by all the Julia Bloggers that somehow try to integrate Weave.jl with Jekyll, Hugo etc. . I am especially referring to this thread: Blogging with Julia code blocks

tlienart commented 5 years ago

I'm excited to announce that this is now available on branch #eval. It's probably a bit rough around the edge still so feedback would be super welcome.

I'll merge after a bit more testing, fixing any comments, fixing the JuDocTemplates to use the new syntax and add an entry in the doc

Here is a small example that can be tested on that branch:

@def title = "Title"
@def hascode = true

# Eval'ing code blocks

We can write a code block like so (note the naming)

```julia:./code/ex1
f(x) = x^5
print(f(7))

This will be executed on the fly unless it's already been executed (and the content hasn't changed) and an output file exists. We can show the output like so:

\input{output}{./code/ex1}

I've also added a shortcut for plain output (syntactic sugar for the above command):

\output{./code/ex1}


### result

![Screen Shot 2019-05-28 at 6 58 35 pm](https://user-images.githubusercontent.com/10897531/58465305-bd187280-817a-11e9-83eb-4c85fb7cc5fc.png)

----------------------

You can change for instance the `7` into a `4` and see it being live-updated. Here since the code is very simple this is all very fast.

In terms of folders, the page was `src/pages/pg1.md` and this generates

. ├── assets │   ├── pages │   │   └── code │   │   ├── ex1.jl │   │   └── output │   │   └── ex1.out └── src ├── _css ├── config.md ├── index.md └── pages ├── pg1.md



note the matching hierarchy with the added `/code/` that was specified in the markdown. (This corresponds to the implementation of the dot syntax from the discussion above).
tlienart commented 5 years ago

this is now implemented #156 and documented.