ddhendriks commented 2 months ago

LS,

My usecase is a technical report/overview of projects and I would like to transclude the last subtree of a particular tree. For each project I keep a journal and in a particular overview node I want to automatically transclude all the most recent project journal-entries. I have looked around in the documentation, but it did not become entirely clear to me whether this is actually possible and how I would do this.

MWE:

when pointing to a tree

* journal
** 2024-05-13
example
** 2024-05-14
example 2

I want the transclusion to show me only

** 2024-05-14
example 2

optionally without the subheader

Is this possible? sounds like it fits the usecase listed in the docs.

Kind regards,

David

akashpal-21 commented 2 months ago

One trivial way I could suggest is using the #+transclude: [[file:<filepath>::*2024-05-14]] :only-contents

ddhendriks commented 2 months ago

Hi @akashpal-21 ,

Thank you for your suggestion, but I am looking for a solution that dynamically finds the last entry (perhaps triggered by some refresh, turning org-transclude-mode on and off, or something).

For example, if I update my above example file to

* journal
** 2024-05-13
example
** 2024-05-14
example 2
** 2024-15-21
example 3

I want the transclude to pick that up automatically and display

** 2024-15-21
example 3

I have attempted some solutions with custom functions that find the last subtree in a given heading, and then generate the entire transclude statement (including the last header, hardcoded), but that still required a lot of manual work (i.e. removing the previously generated ones).

I think this has to be some function within org-transclude to work nicely, but i may be wrong here.

nobiot commented 2 months ago

@ddhendriks

I think this has to be some function within org-transclude to work nicely, but i may be wrong here.

I think you are right. We don't have an easy way within Org-transclusion at the moment as far as I remember/tried. What's the function you use to dynamically find the last tree in a given heading?

I don't think I can do much until mid-late June, but I am considering something that may help with this.

ddhendriks commented 2 months ago

hi @nobiot

With the help of chatGPT I created a function that given a header, finds the last subheader (optionally given a maximum allowed depth). A second function then builds up the transclude statement. The two functions are as follows

(defun my/find-last-entry-in-subtree (file subtree-heading &optional max-depth)
  "Find the last entry in the subtree under SUBTREE-HEADING from FILE.
Print and return the heading of the last entry. If MAX-DEPTH is provided,
limit the search to that depth."
  (let ((last-heading nil)
        (max-depth (or max-depth most-positive-fixnum)))
    (with-current-buffer (find-file-noselect file)
      (goto-char (point-min))
      (if (re-search-forward (format "^\\*+ %s" (regexp-quote subtree-heading)) nil t)
          (let ((start-level (org-current-level)))
            (message "Subtree '%s' found at level %d." subtree-heading start-level)
            (forward-line)  ; Move to the next line to start checking for headings
            (while (and (re-search-forward org-heading-regexp nil t)
                        (> (org-current-level) start-level))  ; Ensure we stay within the subtree
              (let ((current-level (org-current-level)))
                (when (and (<= current-level (+ start-level max-depth))
                           (> current-level start-level))
                  (setq last-heading (org-element-property :raw-value (org-element-at-point)))
                  (message "Updated last heading to: %s at level %d" last-heading current-level))))
            (if last-heading
                (message "Last entry heading within depth %d: %s" max-depth last-heading)
              (message "No entry found within depth %d under subtree '%s'." max-depth subtree-heading)))
        (message "Subtree '%s' not found." subtree-heading)))
    last-heading))

(defun my/org-transclude-last-entry (file subtree-heading &optional max-depth)
  "Transclude the last entry of a subtree under SUBTREE-HEADING from FILE.
If MAX-DEPTH is provided, limit the search to that depth."
  (let ((last-heading (my/find-last-entry-in-subtree file subtree-heading max-depth)))
    (if last-heading
        (let ((transclude-command (format "#+transclude: [[file:%s::*%s]]\n" file last-heading)))
          (message "Transclude command: %s" transclude-command)
          (insert transclude-command))
      (message "No last entry found in subtree '%s' within depth %d." subtree-heading max-depth))))

I suppose that first function could be used somewhere in org-transclusion.

I should say, my lisp knowledge is little and there may be things wrong with this function, so beware using it blindly.

nobiot commented 2 months ago

@ddhendriks

What I had in mind and tested a little is simple. I assume that the structure of the journal in your example above is fixed -- ie the date entry is always level 2.

Then you can simply evaluate this function and add a custom org link.

(defun my/org-last-entry (path arg)
  (org-link-open-as-file path arg)
  (goto-char (car (last (org-map-entries #'point "level=2" 'file)))))

(org-link-set-parameters "last-entry" :follow #'my/org-last-entry)

It enables this last-entry custom link, which navigate to the "last entry" of the tree in FILE.org.

[[last-entry:FILE.org]]

I wanted to create a new wrapper function within Org-transclusion that can transclude an arbitary link like this. The idea is, if the link can navigate to the target, then it can also be turned into a transclusion by simply adding #+transclude keyword.

Would any part of the idea above work for your case?

I am not yet certain if enabling transclusion for any arbitrary link will actually work technically -- but this is something I am thinking of trying in late June (in addition to continuation of some bug fixes).

ddhendriks commented 1 month ago

Hi @nobiot, apologies for the late reply. Thank you for looking into this a bit already and thinking along.

I assume that the structure of the journal in your https://github.com/nobiot/org-transclusion/issues/247#issuecomment-2122051045 is fixed -- ie the date entry is always level 2.

the situation is not entirely like this but:

in a given file/node the depth of the journal entries are consistently the same, but not necessarily at a depth of 2 relative to the root of the file.
a file containing a tree with journal entries can also contain other text

Given the above considerations, if we would be able to:

provide a header in which to look for the entries
provide a depth of the entries.

The code I shared did more or less that.

In your example,

[[last-entry:FILE.org]]

it doesnt look like it's easy to provide additional configurations, whereas there already exists some things that look relevant and maybe have a better way to provide these configurations?

#+transclude: [[file:path/to/file.org::*Headline]] :level 2

if we could just add something like :latest-only to this then that would be a clean syntax and flexible.

I have no clue if there is anything preventing something like this, though.

ddhendriks commented 2 weeks ago

@nobiot have you had the chance to look at any of this? I will pick up trying some things out in the coming weeks again

nobiot commented 2 weeks ago

Yes looked at your suggestion. There are ways to do this but not directly in the way you suggested.

I am specifically referring to this syntax:

+transclude: [[file:path/to/file.org::*Headline]] :level 2

:level property is already used by org-transclusion for a different purpose.

I can make more nuanced comments about what would work but I cannot elaborate right now -- would need to come back, later today or this week. Sorry a little hectic these past few weeks outside Emacs.

akashpal-21 commented 2 weeks ago

Try the following - we cannot use level because it is used up to mean the output level of the transcluded content

Since Nobiot is working on a better name - for the time being - you can use :headlinelevel to mean it -- I present the basic changes you'd need to make to implement it inside the transclusion protocol


;; ensure :headlinelevel is part of insert protocol
(setq org-transclusion-keyword-value-functions
  '(org-transclusion-keyword-value-link
    org-transclusion-keyword-value-level
    ;; HHHH---------------------------------------------------
    org-transclusion-keyword-value-headlinelevel
    ;; HHHH---------------------------------------------------
    org-transclusion-keyword-value-disable-auto
    org-transclusion-keyword-value-only-contents
    org-transclusion-keyword-value-exclude-elements
    org-transclusion-keyword-value-expand-links
    org-transclusion-keyword-current-indentation))

;; Convert :headlinelevel from string to plist
(defun org-transclusion-keyword-value-headlinelevel (string)
  "It is a utility function used converting a keyword STRING to plist.
It is meant to be used by `org-transclusion-get-string-to-plist'.
It needs to be set in
`org-transclusion-keyword-value-functions'."
  (when (string-match ":headlinelevel *\\([1-9]\\)" string)
    (list :headlinelevel
          (string-to-number (org-strip-quotes (match-string 1 string))))))

;; Make sure on removal - :headlinelevel keyword is inserted back
(defun org-transclusion-keyword-plist-to-string (plist)
  "Convert a keyword PLIST to a string."
  (let (;;(active-p (plist-get plist :active-p))
        (link (plist-get plist :link))
        (level (plist-get plist :level))

    ;; HHHH---------------------------------------------------
    (headlinelevel (plist-get plist :headlinelevel))
    ;; HHHH---------------------------------------------------

    (disable-auto (plist-get plist :disable-auto))
        (only-contents (plist-get plist :only-contents))
        (exclude-elements (plist-get plist :exclude-elements))
        (expand-links (plist-get plist :expand-links))
        (custom-properties-string nil))
    (setq custom-properties-string
          (dolist (fn org-transclusion-keyword-plist-to-string-functions
                      custom-properties-string)
            (let ((str (funcall fn plist)))
              (when (and str (not (string-empty-p str)))
                (setq custom-properties-string
                      (concat custom-properties-string " " str ))))))
    (concat "#+transclude: "
            link
            (when level (format " :level %d" level))

        ;; HHHH---------------------------------------------------
        (when headlinelevel (format " :headlinelevel %d" headlinelevel))
        ;; HHHH---------------------------------------------------

            (when disable-auto (format " :disable-auto"))
            (when only-contents (format " :only-contents"))
            (when exclude-elements (format " :exclude-elements \"%s\""
                                           exclude-elements))
            (when expand-links (format " :expand-links"))
            custom-properties-string
            "\n")))

;; Implement logic for processing :headlinelevel keyword
(defun org-transclusion-content-insert (keyword-values type content sbuf sbeg send copy)
  "Insert CONTENT at point and put source overlay in SBUF.
Return t when successful.

This function formats CONTENT with using one of the
`org-transclusion-content-format-functions'; e.g. align a table
for Org.

This function is intended to be used within
`org-transclusion-add'.  All the arguments should be
obtained by one of the `org-transclusion-add-functions'.

This function adds text properties required for Org-transclusion
to the inserted content.  It also puts an overlay to an
appropriate region of the source buffer.  They are constructed
based on the following arguments:

- KEYWORD-VALUES :: Property list of the value of transclusion keyword
- TYPE :: Transclusion type; e.g. \"org-link\"
- CONTENT :: Text content of the transclusion source to be inserted
- SBUF :: Buffer of the transclusion source where CONTENT comes from
- SBEG :: Begin point of CONTENT in SBUF
- SEND :: End point of CONTENT in SBUF"
  (let* ((beg (point)) ;; before the text is inserted
         (beg-mkr (set-marker (make-marker) beg))
         (end) ;; at the end of text content after inserting it
         (end-mkr)
         (ov-src (text-clone-make-overlay sbeg send sbuf)) ;; source-buffer overlay
         (tc-pair ov-src)
         (content content))
    (when (org-transclusion-type-is-org type)
      (with-temp-buffer
        ;; This temp buffer needs to be in Org Mode
        ;; Otherwise, subtree won't be recognized as a Org subtree
        (delay-mode-hooks (org-mode))
        (insert content)
        (org-with-point-at 1
          (let* ((to-level (plist-get keyword-values :level))
                 (level (org-transclusion-content-highest-org-headline))

         ;; HHHH---------------------------------------------------
         (by-level (plist-get keyword-values :headlinelevel))
         ;; HHHH---------------------------------------------------

                 (diff (when (and level to-level) (- level to-level))))

        ;; HHHH---------------------------------------------------
        (when by-level
          ;; go to org-level bylevel here
          ;; then narrow to subtree

              ;; Go to org-level bylevel here
              (goto-char (point-min)) ;; Possibly redundant(?)
              (while (and (re-search-forward org-heading-regexp nil t)
              (not (= (org-outline-level) by-level))))  ;; loop exit cond
              ;; Narrow to subtree
              (org-narrow-to-subtree)
          ;; Reset markers for :level processing
          (setq level (org-transclusion-content-highest-org-headline)
            diff (when (and level to-level) (- level to-level))))
        ;; HHHH---------------------------------------------------

        (when diff
              (cond ((< diff 0) ; demote
                     (org-map-entries (lambda ()
                                        (dotimes (_ (abs diff))
                                          (org-do-demote)))))
                    ((> diff 0) ; promote
                     (org-map-entries (lambda ()
                                        (dotimes (_ diff)
                                          (org-do-promote)))))))

            (setq content (buffer-string))))))
    (insert
     (run-hook-with-args-until-success
      'org-transclusion-content-format-functions
      type content (plist-get keyword-values :current-indentation)))
    (setq end (point))
    (setq end-mkr (set-marker (make-marker) end))
    (unless copy
      (add-text-properties beg end
                           `(local-map ,org-transclusion-map
                                       read-only t
                                       front-sticky t
                                       ;; rear-nonticky seems better for
                                       ;; src-lines to add "#+result" after C-c
                                       ;; C-c
                                       rear-nonsticky t
                                       org-transclusion-type ,type
                                       org-transclusion-beg-mkr
                                       ,beg-mkr
                                       org-transclusion-end-mkr
                                       ,end-mkr
                                       org-transclusion-pair
                                       ,tc-pair
                                       org-transclusion-orig-keyword
                                       ,keyword-values
                                       ;; TODO Fringe is not supported for terminal
                                       line-prefix
                                       ,(org-transclusion-propertize-transclusion)
                                       wrap-prefix
                                       ,(org-transclusion-propertize-transclusion)))
      ;; Put the transclusion overlay
      (let ((ov-tc (text-clone-make-overlay beg end)))
        (overlay-put ov-tc 'evaporate t)
        (overlay-put ov-tc 'face 'org-transclusion)
        (overlay-put ov-tc 'priority -60))
      ;; Put to the source overlay
      (overlay-put ov-src 'org-transclusion-by beg-mkr)
      (overlay-put ov-src 'evaporate t)
      (overlay-put ov-src 'face 'org-transclusion-source)
      (overlay-put ov-src 'line-prefix (org-transclusion-propertize-source))
      (overlay-put ov-src 'wrap-prefix (org-transclusion-propertize-source))
      (overlay-put ov-src 'priority -60)
      ;; TODO this should not be necessary, but it is at the moment
      ;; live-sync-enclosing-element fails without tc-pair on source overlay
      (overlay-put ov-src 'org-transclusion-pair tc-pair))
    t))

Tested lightly.

akashpal-21 commented 2 weeks ago

Introduces a nasty regression in that it breaks :level -- we have to reset the markers relative to the level and inverse the timing of operations between :level and :headlinelevel

        ;; HHHH---------------------------------------------------
        (when by-level
          ;; go to org-level bylevel here
          ;; then narrow to subtree

              ;; Go to org-level bylevel here
              (goto-char (point-min)) ;; Possibly redundant(?)
              (while (and (re-search-forward org-heading-regexp nil t)
              (not (= (org-outline-level) by-level))))  ;; loop exit cond
              ;; Narrow to subtree
              (org-narrow-to-subtree)
          ;; Reset markers for :level processing
          (setq level (org-transclusion-content-highest-org-headline)
            diff (when (and level to-level) (- level to-level))))
        ;; HHHH---------------------------------------------------

        (when diff
              (cond ((< diff 0) ; demote
                     (org-map-entries (lambda ()
                                        (dotimes (_ (abs diff))
                                          (org-do-demote)))))
                    ((> diff 0) ; promote
                     (org-map-entries (lambda ()
                                        (dotimes (_ diff)
                                          (org-do-promote)))))))

Will update the above snippet

nobiot commented 2 weeks ago

@akashpal-21, thank you for the code. It has given me some better ideas. @ddhendriks, here is what's in my mind.

Original idea -- prefer generic solution

Using your suggest as an example (adding @akashpal-21's change):

#+transclude: [[file:path/to/file.org::*Headline]] :headlinelevel 2

This, to me, extends Org's built-in link functionality. [[file:path/to/file.org::*Headline]] is the built-in link syntax, and :headlinelevel2 is the extension.

This extension is specific to your use case.

Where possible, I prefer to try to work out a more generic way to achieve the same end result. One idea was to use the Org's built-in feature to create your own custom links, like this.

[[last-entry:FILE.org]]

If you can create a custom link that lets you specify the subtree and level, for example (hypothetical custom link just for illustration):

[[subtree:FILE.org*Headline::2]]
::2 specifies the first level-2 subtree within a subtree named "Headline" in FILE.org.

This can be more generic functionality, and hence this idea I suggested below.

I wanted to create a new wrapper function within Org-transclusion that can transclude an arbitary link like this. The idea is, if the link can navigate to the target, then it can also be turned into a transclusion by simply adding #+transclude keyword.

If implemented, it would let you use the following transclusion syntax (and, in theory, any custom links that you may create).

#+transclude: [[subtree:FILE.org*Headline::2]]

Specific solution -- extending @akashpal-21's idea

This would be a customizing. I think you could avoid overriding the functions.

A couple of suggestions:

keyword <-> plist conversion :headlinelevel

Have a look at how [org-transclusion-src-lines.el](https://github.com/nobiot/org-transclusion/blob/main/org-transclusion-src-lines.el) is implemented, especially around the use of hooks org-transclusion-keyword-value-functions and org-transclusion-keyword-plist-to-string-functions.
Avoid overriding the function org-transclusion-content-insert.

Instead, I would first try adding a new filter. See this part.

For example, it may make sense to have a filter like :only-level -- it would probably bring ALL level 2 subtree under Headline if there is more than one.

#+transclude: [[file:path/to/file.org::*Headline]] :only-level 2

The challenge for me is that currently it's not easy to add a new custom filter as a customizing. I think I have made some attempt in the course of discussing this PR -- I remember I refactored relevant parts, but never merged this to main.

Back to original request

@ddhendriks

In either of the 2 options above, it will take some efforts.

akashpal-21 commented 2 weeks ago

The other problem is that there is not an easy way to make all the other elements play nice. The problem with my code is that it is only meant to be used in conjunction to specifying the headline with a search parameter in the link, so that using the keyword argument w/o the search parameter would result in the first headline level found to be included only.

We could work around it using a slightly more complication involving org-element-map and making some more adjustment, but this adds way too much complexity in the end to allow for a niche usecase.

Using custom links make more sense

nobiot / org-transclusion

Transclusion of last entry of subtree only #247

+transclude: [[file:path/to/file.org::*Headline]] :level 2

Original idea -- prefer generic solution

Specific solution -- extending @akashpal-21's idea

Back to original request