Make possible to weblorg-copy-static copy other files than the static directory

[semente]

Would be interesting if weblorg-copy-static could copy things outside a predefined static directory. For instance, I use this in org-publish:

("assets"
  :base-directory "."
  :base-extension "jpg\\|png\\|txt\\|sh\\|el"
  :exclude '("^_" "^[.#]" "~$")
  :recursive t
  :publishing-directory ".build"
  :publishing-function org-publish-attachment)

With this I can keep post assets in the same directory of the posts or in any other directory of my choice. Once again, as mentioned in other issues I have open, I want to be able to easily browse my project files using Emacs in the same way as in a web browser.

With weblorg I currently copy these files using rsync but weblorg-copy-static could support something like this:

(weblorg-copy-static
 :base-dir "."   ; other than theme/static
 :input-pattern '("**/.jpg" "**/*.png" "**/*.txt" "**/*.sh" "**/*.el")
 :output "output/{{ file }}"
 :url "/{{ file }}")

Besides the custom input-pattern and base-dir, note the multiple input values using a list.

[JensAc]

Hey, I want to add something on this. Would it make sense to have a directory per per post. Let's say I have code blocks, which export figures and I have a lot of posts doing this. Then, I maybe do not want to have all the images in the same directory, as this would be really confusing, when browsing the directory. Perhaps, I even do not want to name the images differently. Would that contradict the overall concept of weblorg?

[mrmechko]

Pull requests adds :data-dir option to weblorg-copy-static to allow copying individual static directories over into the final weblorg site.

The problem I was having was that simply providing :base-dir resulted into a translation to a theme directory within :base-dir. However, that is a very useful feature, so instead of messing with the theme translation, I just added the :data-dir option. If you have a :data-dir, that particular static route will copy files in :data-dir. You can still use :input-filter etc as you see fit.

(weblorg-copy-static
 :name "cs141-static"
 :data-dir "~/docs/141/static"
 :output "output/cs141/static/{{ file }}"
 :url "/static/{{ file }}")

[clarete]

I'm interested in this use-case. I'm under the impression that we're missing a small piece to make it all work, but I wanted to see if my assumptions are correct.

If I am getting this right, you want something that 1. goes through a directory recursively, 2. finds files matching a list of patterns, 3. copy them to a destination directory that follows the same directory structure. If that is correct, I think the weblorg--copy-static that we currently have is short of an option like :recursive t, just like org-publish does. @semente what would be the difference between your suggestion and something like (regexp-opt '(".jpg" ".png" ".txt" ".sh" ".el"))? do you think we could keep :input-pattern the way it is if we use it with regexp-opt?

And, more objectively, would :recursive t solve this issue for you guys?

[mrmechko]

I particularly want to be able to move static files that are not theme files (eg linked images, or slides). :data-dir specifies a file that overrides the location finding function weblorg--path. Is there better way to do that?

[semente]

@semente what would be the difference between your suggestion and something like (regexp-opt '(".jpg" ".png" ".txt" ".sh" ".el"))?

hi @clarete! yea, in combination with :recursive t it would work just fine. My thought was since :input-pattern already support the globstar ** it would make more sense than adding another option (:recursive).

So I think we already have an recursive t if we can use **/*. I would just filter in few files based on extensions (or regex).

[clarete]

@semente what would be the difference between your suggestion and something like (regexp-opt '(".jpg" ".png" ".txt" ".sh" ".el"))?

hi @clarete! yea, in combination with :recursive t it would work just fine. My thought was since :input-pattern already support the globstar ** it would make more sense than adding another option (:recursive).

So I think we already have an recursive t if we can use **/*. I would just filter in few files based on extensions (or regex).

ohh, that's pretty interesting. I'm going to see if the eshell function that I use to resolve the glob expression can expand recursively as well. That would save us from adding a new parameter, which is more expensive in terms of maintenance/documentation/etc!

[clarete]

I particularly want to be able to move static files that are not theme files (eg linked images, or slides). :data-dir specifies a file that overrides the location finding function weblorg--path. Is there better way to do that?

after learning about this globstar thing, which I didn't really fully understand, my plan is to try to use just the :input-pattern parameter as a way of finding which files need to be copied. That will take removing the hardcoded mention to the "static" directory there. If the globstar thing doesn't work, we'll probably have to add the :recursive t parameter alongside with the fix for using `:input-pattern' here.

[semente]

@clarete weblorg use eshell-extended-glob, which has globstar capability. See for instance (eshell-extended-glob "/etc/**/*.conf"), it will match all the /etc/*.conf, /etc/*/*.conf, /etc/*/*/*.conf files, and so on.

[clarete]

Hi hi hi 👋🏾 this has taken some time to sink in for sure. Really sorry about that and thank you for the patience, but I think I've finally got an idea worth sharing 😄

It really took me trying to setup a few different static websites until I really saw what you guys were saying about wanting to keep "the original filesystem structure" of the source of a website. That is a very sophisticated feature that I didn't really have in mind while designing the first version of weblorg. This is a very good idea and very powerful feature to be included but it might take a bit more of thought and some emacs lisp hacking to get done, so I feel like it's fair to say that it'd be a feature for the 0.3 release. (0.2 is sorta coming 😄).

With all that said, this is what I thought that we could do to achieve what I think we all want here:

(weblorg-route
 ;; Name of the route.  That is what we use to refer to this
 ;; route within other routes via `url_for' for example.
 :name "index"

 ;; Find files ending with all these extensions
 :input-pattern "posts/*.(jpg|png|css|org)"

 ;; Declare a different handler for each
 :input-handle '((:pattern "*.(jpg|png|css)"
                  :handler #'weblorg-handle-static)

                 (:pattern "*.org"
                  :aggregate #'weblorg-input-aggregate-each
                  :handler #'weblorg-handle-orgmode
                  :template "post.html"
                  :output-filename #'weblorg-output-filename-slugified))

 ;; All input files will have the same base directory as the
 ;; output.  That said, you can still customize how the file
 ;; name of each input handle type will look like.
 :output "output/{{ output_file_name }}"
 :url "/")

The above proposal of including the new :input-handle parameter that allows one single route to have different handlers. That's how we'd split the pipeline for different inputs. The splitting of the list of input files with a matching of a regular expression against the file path of the input that matched :input-pattern.

Also, notice the :output-filename parameter. That is a response to the issue #27, so you'd be able to configure how the file name of each handler would be created.

Curious to hear what you guys think 😄

/cc @semente @guilhermecomum

[semente]

@clarete :D

Yea, I really like this!

Really sorry about that and thank you for the patience, but I think I've finally got an idea worth sharing smile

There is no rush! Don't worry. Thank you

[nanzhong]

I like the described approach!

I use a similar structure in my personal site, with each post/page having its own directory that includes the org file and assets the org file links to. My current approach is to ignore the static assets in weblorg and have an out of band process to copy those over preserving the output structure that weblorg uses. This feels like it could replace that entirely.