Feature: Incremental compilation (changed only)

[holyjak]

Fixes (partly) cryogen-project/cryogen-core#52, cryogen-project/cryogen#67.

Adds a new compile-assets argument, changeset, passed from the watcher and used to only compile the pages/posts that have changed (or everything, if theme html changes). There is also a new watcher function that passes the changeset to the provided callback. (If changeset is empty, everything gets compiled.)

Note: This cuts my compilation time from 60 to 3s despite the fact that tags and archives are still always re-compiled. We could speed up further if desirable by skipping those.

Requires a change to the generated blog code, i.e. replacement of start-watcher! (that expects a 0-arg action) with start-watcher-for-changes! (expecting a 1-arg callback) to take effect - see cryogen-project/cryogen#220.

[holyjak]

~## Alternative solution 1 (no blog code change needed)~

~Pass the changeset from the watcher to compile-assets via a dynamic variable / atom.~

~I don't like this because it is very opaque and non-FP. But on the other hand it is far more efficient than (re-)scanning the filesystem and users can benefit from it without changing their code.~

[holyjak]

~## Alternative solution 2 (no blog code change needed)~

~If (:incremental-compile? config) then compiler itself checks whether the page/post has changed via "timestamp of the target file < ts of the source file" and compiles only if it has.~

~It is somewhat slower because we have to ask the file-system for the timestamp of each source & target file but I assume it is still fast enough. I actually like this solution since it requires no work on the side of the blogger other than a config change. A complication is that we need to know not only whether the current page/post has been updated but also whether any theme HTML has changed so we also need to first scan the theme directory for any html changes. I am not sure whether the additional complexity if this doesn't make this solution unworthy...~

[bombaywalla]

There is a minor typo that causes the tests to fail. There is an extra double-quote at the end of a line. Here is a diff.

diff --git a/src/cryogen_core/compiler.clj b/src/cryogen_core/compiler.clj
index 3454582..e02e260 100644
--- a/src/cryogen_core/compiler.clj
+++ b/src/cryogen_core/compiler.clj
@@ -580,7 +580,7 @@
      - `changeset` - as supplied by
                    [[cryogen-core.watcher/start-watcher-for-changes!]] to its callback
                    for incremental compilation, see [[only-changed-files-filter]]
-                   for details"
+                   for details

   Note on terminology:
    - `article` - a post or page data (including its title, content, etc.)

[bombaywalla]

Would you please remove the change to the version in project.clj for now? I think it is best that PRs do not update the version. We will discuss on Slack.

[holyjak]

Thanks a lot! You are doing an awesome job! Sorry for the typo, that was bad merge. I should have run tests before pushing.

[bombaywalla]

Thanks for reverting the version.

Would you please re-check the typo? I am still getting the same error regarding the double quote.

I did the following:

git clone https://github.com/holyjak/cryogen-core
cd cryogen-core/
git checkout --track origin/feat/fast-compile
lein test

[holyjak]

Commits are immutable. Don't look at those, look at https://github.com/cryogen-project/cryogen-core/pull/149/files

On Sun, 29 Nov 2020, 01:33 Dorab Patel, notifications@github.com wrote:

Actually, I may be doing something wrong. If I look at the 31fcf39 https://github.com/cryogen-project/cryogen-core/commit/31fcf39ff56b6d24e8957df40958e078e9a74ed2 commit, I see that you have made the change. But if I clone your repo (as shown above) I do not see the change. What am I doing wrong?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cryogen-project/cryogen-core/pull/149#issuecomment-735310283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYSPVSYAJFNWSDXBXN7QDSSGJDVANCNFSM4S6GTEYA .

[holyjak]

OMG 😅 I forgot to push the typo fix. Amended now.