meedstrom
commented
3 months ago Opinions @emacsomancer?
Closed meedstrom closed 2 months ago
meedstrom
commented
3 months ago Opinions @emacsomancer?
emacsomancer
commented
3 months ago I've only done very basic things with capture templates in general. I had basic ones for plain Org back in the day, and still not very complicated Org-Roam ones. I think they are useful, and could be more useful perhaps if one thought about one's typical patterns of use (or, alternatively, better patterns of use even).
I haven't delved too deeply into all of the underlying mechanics of org-roam-dailies-capture, org-roam-capture, or org-capture, or their exact relations.
I have a very basic regular org-roam-capture thing for entries (probably just copied from the manual more or less):
(setq org-roam-capture-templates
`(("d" "default" plain "%?" :if-new
(file+head "%<%Y%m%d%H%M%S>-${slug}.org"
,(concat "#+title: ${title}\n"
"#+date: %U\n\n"))
:unnarrowed t)))(Though I have a more complicated hook that goes back and adds additional properties like AUTHOR, and LOCATION (with functions that either check what WiFi network I'm on and set location appropriate, or interface with GPS and look up the location on OSM).)
I have very slightly more complicated/useful org-roam-daily capture templates:
(setq org-roam-dailies-capture-templates
`(("d" "default" entry "* %?" :if-new
(file+head "%(concat org-roam-dailies-directory \"/%<%Y-%m-%d>.org\")"
,(concat "#+title: %<%Y-%m-%d>" "\n"
"#+filetags: :daily_journal:\n\n")))
("j" "journal" entry "* journal :personal:
:PROPERTIES:
:ID: %(org-id-new)
:END:
%?"
:if-new
(file+head "%(concat org-roam-dailies-directory \"/%<%Y-%m-%d>.org\")"
,(concat "#+title: %<%Y-%m-%d>" "\n"
"#+filetags: :daily_journal:\n\n")))))The first being likely the default one for new note in today's daily and the second a specialised one for "journal" entries in today's daily [adding a tag and an org-id to the heading to make it a node].
org-read-dateI've managed never to use org-journal, and I'm wary about trying to get too deep into it, especially given what you've said about some things being unstable or behaving in unexpected ways.
Personally, I think the ideal would be to try to depend mainly on plain Org things as much as possible (which might mean rethinking the capture templates thing, using plain Org capture templates instead, rather than Org-roam ones? though that might just end up reinventing the wheel?). I think (org-read-date) is quite useful - and you can see that in the commits in https://github.com/meedstrom/org-node/pull/25 that's actually the thing doing all the real work.
Org-roam's dailies capture and navigation things ultimately rely on org-read-date, which will pop up the little calendar thing if it's not given a starting date. (I found Sacha's Org Mode date arithmetic useful for figuring out how to work with org-read-date.)
Personally, I like having a generalised "goto-date" function, and then pre-set things for going to "the previous daily entry" and "the next daily entry". (I don't actually use the literal "yesterday" and "tomorrow" commands in practice.) It's not about extra keystrokes per se, but if there is a command that does "goto previous entry" or "goto next entry", then after activating it once, I can then just call repeat (which I have bound to a single keystroke) and easily cycle forward or backward through dailies.
(For keybindings, I have the "note taking" ones all under a C-c n prefix; e.g. C-c n n for a new daily note; C-c n f to find a node; C-c n i to insert a link to a node; C-c n y to go to the previous journal entry ("Yesterday"); C-c n t to go to the next journal entry ("Tomorrow"); and so on.)
And, in the current version of org-node, the "previous" and "next" commands are simple, e.g. (org-node-goto-daily "--" 1 "d") for "previous", just relying on the logic in org-node-goto-daily.
(Of course, the current logic of the function that actually finds the previous or next entry is a bit naive, as it tries, for instance, to see if org-node has a hash table entry for the day before, and if not, then sees if it has a hash table entry for the day before that, and so on. It would be smarter to do something where it looked directly at what dailies org-node has indexed and sort them and just pick the one preceding the current one, whatever it was - that would be much more computationally efficient. (Imagine the case where you stopped journalling for five years and then started again and went to look up the "previous entry": currently org-node--find-next-daily-iterate is going to check org-node for each possible date until it finally finds the daily from five years ago.))
There is some aspect of re-inventing org-roam-dailies-goto-date, but it's fairly simple anyway and really ultimately really relies on org-read-date anyway.
The real question, I think, is about the "next" and "prev" type commands, because org-roam-dailies-goto-next-note is more complicated. It operates (in a sense more smartly than the current org-node-goto-daily, since it actually knows what entries are available) via org-roam-dailies--list-files, which is looking at what files exist in the org-roam-dailies-directory (and then uses cl functions to operate over a list). I'm adverse to this (though it makes sense) for the reason of avoiding repeating filesystem-related operations.
One way of handling look-up of dailies in this (directly file-based) fashion would be to accept (the inevitable) initial listing of all dailies by querying the filesystem and then not repeat the operation but rather cleverly modify a hash when a new daily was created (or deleted?).
Though since dailies should all be already indexed by org-node, cleverer still, I think, would be just to look them up in the main org-node hash table, filtering out things whose path does not include the dailies directory.
Which raises/touches on the other issue of how org-node should (if at all) handle dailies. I think my current inclination has been to introduce an org-node-dailies-directory variable (which could default to the value of org-roam-dailies-directory if extant).
But the directory-independent logic could be better. Filtering based on a tag or a directory is presumably the same cost for hash table look-ups. And then the user could set what their "daily" tag is (mine is daily_journal).
Though I have a question here: I have a couple thousand daily files at least, and several thousand non-daily files, and then other org-id bearing entities within all of those: how costly is filtering 20,000 nodes for either :file-path or :tags? Would it, in fact, end up more efficient simply to have another hash table that only contained things in a specific directory? And that would then not require filtering? Or would such hash-table filtering be very cheap, even with tens of thousands of nodes?
(I suppose the separate "dailies hash table" might run into the same look-up problems/cost unless the hash-table itself encoded the "position" of each entry [i.e. directly encoded the ordering relation between hash table "daily" entries], which then would raise an issue of how to update the table when a new daily was created, in a way that would not require re-indexing the entire directory each time.)
I'm not sure I like the "creation-date" (:CREATED:) property logic, for the reason that I sometimes anachronistically create dailies.
One possibility — assuming that the hash-table filtering approach is viable for a hash table with 20,000+ entries — is just to have the potential for the user to choose from different options: it could be based on :file-path property or a :tags property. But this is about filtering and not finding (by finding, I mean something like "give me the entry preceding the current one, which may or may not be actually the preceding day").
For finding nodes: the possibility of look-ups based on something like :file-title or :CREATED: is back to the question of how "next"/"previous" type things are actually handled. Obviously org-read-date is useful is calculating calendrical dates, but not "give me the previous or next entry, however temporally distant it may be".
If there are, say, 20,000 nodes in the hash table, which are then filtered based on :file-path or :tags to just the "dailies" (assuming this isn't too expensive), what would be the best way of getting "the previous" or "the next" daily? Pulling out all of the :file-title or :CREATED: properties into a list and sorting it and then doing something like what org-roam-dailies-goto-next-note does? (I.e., use cl-position to find the position of the daily in the list and then use nth to find the appropriate next or previous extant date.) Or is there a better way?
Sorry this is disorganised. But, to try to pull it together:
org-journal, but I could be mistaken here. And also org-node being non-reliant on (but compatible with) org-roam seems desirable.org-read-date interface (which is still useful for picking particular dates, of course). And, moreover, the basic org-read-date interface doesn't directly solve the issue of "what is the next/previous extant entry?" :file-path property in the hash table) or a tag (:tags in the hash-table), or potentially some other hash-table property. But I don't know the computational cost for filtering a hash-table with tens of thousands of entries based on properties.org-node and the user is advised to use other existing packages to deal with dailies.org-journal and/or org-roam are leveraged in some way, if the solutions for these packages are deemed accepted. (I'm dubious here; esp. org-roam's solutions seem to end up costly/slow in many cases, at least when dealing with significant numbers of nodes.)org-node and org-node-specific solutions are devised which are computationally-efficient when dealing with large numbers of nodes.I'm obviously a bit biased towards the last of these myself.
meedstrom
commented
3 months ago Re. leveraging org-roam, I suggest an override for org-roam-dailies--list-files at https://github.com/meedstrom/org-node?tab=readme-ov-file#tip-on-very-slow-filesystems. Does it fix the performance issues you have on Termux?
I agree "next" and "prev" commands are important to have.
I am, perhaps ironically, not concerned with performance because I am sure I can find some efficient implementation, no matter what! So it's a question of whether there's an unexplored niche here, a new mental model that on retrospect makes more sense than the existing solutions. Always nice when you can find such things :)
It's already partly possible to do what you touched upon about doing away with Roam's abstractions, thanks to org-node-capture-target, but I feel the absence of a capture target to a daily-note. So it seems we do need an in-house concept of what is a daily note, and a way to find it.
(And it seems that org-journal doesn't ship any capture-related stuff... so it's pretty incomplete. It has "carryover" which is basically an automated refile to the next day, but still no general "refile into GIVEN DAY" command, which should exist.)
Pretty sure a lot of people have rolled their own journaling systems with org-capture, so it's not a new idea, but I wonder how they do it. It's not like you can put a filename "%Y-%m-%d.org" in a vanilla capture template, it has to invoke a function.
(Well -- you can keep the template simple if you keep all dailies in one hardcoded file "dailies.org")
And it's extra credit to make a way to jump to any given date ... if I was an amateur, I would just use Dired to go to where I want. Which is perfectly sane. But once you start wanting to refile into dailies, it starts looking like someone should package a collection of related functionality. And not the way org-roam does it :)
I'm tempted to see what core Org has to offer. Did you ever use a datetree? Org has some builtin functions for them, like org-datetree-find-date-create.
And there's code on the net for refiling to a datetree and probably more.
One nice thing is they can be very compact, no ID, no properties drawer. So a question: do dailies really need IDs?
The way I've come to use them, yes. I let some dailies be pages on my website, and I use a daily-page sometimes as a "link hub" so the backlink-to-a-daily-page can be a way to find related work. But perhaps that's just me molding myself to my tools.
Actually I've been a bit bothered that I can't just go to a daily-page and see backlinks to everything that was created on that date, no matter if there exists an explicit link or no.
If there were just no "daily-pages", but we could still "go visit a date", what would that mean? Maybe an indirect buffer, like... the builtin diary... or, you know, I believe the agenda can search for Org timestamps everywhere? Maybe a timestamp like [2024-08-10 Sat] can be considered a pseudo-link already, no need for an ID-link to a daily page for that date.
meedstrom
commented
3 months ago Imagine that instead of browsing the dailies dir, you open the agenda, turn on org-agenda-log-mode, navigate backwards in time, and then see on the days where you have a daily-note that you made a note that day. As well as any other notes made that day. (and with Memacs, any photos you took and any emails you sent and so on, but that's extra credit)
Pretty cool, and maybe the main thing that's acftually missing is that backlinks still need to show up once visiting any of these files, for the "link hub" effect. But backlinks to what? Maybe they'd be pseudo-links that open the agenda buffer.
emacsomancer
commented
3 months ago Yes, I'm trying the overrides and they do seem to help a lot in combination with the cache. I think the cache and other changes may make using Org-Roam(ish) notes much more reasonable on Android/Termux, so this is all great! :)
I definitely agree on mental models. I wonder if one way is trying to design the system with flexibility wherever possible, which could itself enable the discovery of niches.
Yes, clearly a "daily"-sort-of-thing-related capture feature in org-node is needed - and, at least if one is looking at what some other systems have done (like org-roam), the daily-related capture function is at least potentially related to the other issues, like "next"/"prev" things.
No, the plain Org capture templates don't seem to have built-in functionality for capturing to "%Y-%m-%d.org", but surely creating/invoking a function to produce a "%Y-%m-%d.org"-style target is not a major obstacle.
I haven't really worked with org datetree before, so I'm not sure how they would be used here. But it sounds like something to potentially explore.
I think for my workflow I do want actual daily files, for a couple of reasons: one, it acts as a place to put notes that I don't want to have to think about where they go (at least at that moment, maybe I'll refile them or copy information into named nodes later); two, I like having a place where I intentionally put date-associated notes. (I do like the idea of being able to have to things auto-populated from other places, but I think I want also intentionally logged things to be treated specially.)
meedstrom
commented
3 months ago Made some toy code, ended up letting an user-provided lambda determine whether a node is a "daily", so now I'm generalizing the concept into node "series", where dailies are just one of many possible kinds of series... It seemed the natural thing to do...
emacsomancer
commented
3 months ago Ah, excellent!
I spend a bunch of time yesterday playing with a bunch of things to do with "series" myself. A lot of it was dead ends and I felt a bit frustrated with myself, but, and it may turn out irrelevant given your new code. But.... maybe there's something of use it. I suppose we'll see. In any case:
1) At least for me, currently org-node-list-roam-files and org-node-list-roam-daily-files always return nil. In my online fork, I have the fixes that worked for me (see https://github.com/meedstrom/org-node/compare/main...emacsomancer:org-node:master ; essentially just making sure to use truename. These trivial changes I haven't done a pull request for, partially because I'm not sure it's the appropriate fix or not. But it works for me at least.)
2) Even fixing org-node-list-roam-daily-files, trying to use it to work for series of dailies (i.e. to try to do something a bit like what org-roam does for finding "next" and "prev" was too slow, even on my fast desktop). And so the below code doesn't actually depend on the "truename" fixes in the not-yet-a-pull-request changes in my github fork.
3) Instead, I just decide to work directly with org-nodes and query properties, and this works much better and has no speed issues, I think. I haven't cleaned it up fully and so it's not in the github fork yet (and maybe is superseded by what you've done, I haven't looked at that yet), but here's a preview of the new org-node-goto-daily function and its current only two "helper" functions (other than org-node--daily-not-found):
(defun org-node-goto-daily (&optional direction range span)
"Go to a daily node with a time `offset' from the current buffer.
If called interactively, find the daily-note for a date using the calendar,
creating it if necessary."
(interactive)
(let* ((offset
(if (and direction range span)
(concat direction (number-to-string range) span)
nil))
(must-be-in-daily ;; if prefix makes reference to base date
(if offset (let ((prefix
;; ('Some people, when confronted with a problem, think
;; "I know, I'll use regular expressions."
;; Now they have two problems.' —Jamie Zawinski)
"\\`\\(?:\\+\\{2\\}\\|-\\{2\\}\\)"))
;; = two or more `+' or `-' at beginning of string
;; = (rx (: bos (or (repeat 2 2 "+")
;; (repeat 2 2 "-"))))
(string-match prefix offset))
nil)))
(if (and
(not (org-node--daily-note-p))
must-be-in-daily)
(user-error "Not in a daily-note.")
(if (member direction '("--" "++")) ;; if next or prev & missing node
(org-node--locate-offset-node
(file-name-base buffer-file-name)
direction range span)
(let* ((current-node
(if must-be-in-daily
(org-time-string-to-time
(file-name-base buffer-file-name))
nil))
(target-node
(org-read-date nil nil offset nil current-node))
(node-hash (gethash target-node org-node--candidate<>node)))
(if node-hash
(org-node--goto node-hash)
(org-node--daily-not-found target-node)))))))
(defun org-node--locate-offset-node (node direction range span)
"Given a reference `node', a `direction' for offset, a `range'
of distance and a `span' [currently `span' is ignored], open the
daily journal entry file of the appropriate date in the current
window. This function is currently really just used for
`org-node-goto-prev-day' and `org-node-goto-next-day' commands."
(unless (org-node--daily-note-p)
(user-error "Not in a daily-note"))
(setq n (or range 1))
(setq n (* n (if (equal direction "--")
-1 1)))
(let* ((dailies (org-node--calculate-list-of-dailies-dates node))
(position
(when node
(cl-position-if (lambda (candidate)
(string=
node
candidate))
dailies))))
(if position
(progn
(pcase n
((pred (natnump))
(when (eq position (- (length dailies) 1))
(user-error "Already at newest note")))
((pred (integerp))
(when (eq position 0)
(user-error "Already at oldest note"))))
(setq note (nth (+ position n) dailies))
(org-node--goto
(gethash note org-node--candidate<>node)))
(user-error "ERROR"))))
(defun org-node--calculate-list-of-dailies-dates (node)
"Calculate a list of known dailies, adding
`node' as part of the list in case `node' actually
represents the date of an unknown/nascent dailies."
(delete-dups ;; takes care of (a) the case that `node' actually exists
;; and (b) cases of squiggle~ or #autosave or sync-conflict files
;; that `org-nodes' may have entries for
(sort ;; sort into calendrial order
(cons node ;; add in the current node in case it's not actually
(let ((keys ()) ;; (yet) extant
(titles ()))
(maphash (lambda (k v) (push (cons k v) keys)) org-nodes)
(let ((titles
(cl-loop
for key in keys
collect (org-node-get-file-title-or-basename (cdr key)))))
(cl-loop
for title in titles
when
(string-match-p ;; match for YYYY-MM-DD nodes only
"^\\(?:[0-9][0-9]\\)\\{1,2\\}-\\(?:1[012]\\|0?[1-9]\\)-\\(?:0?[1-9]\\|[12][0-9]\\|3[01]\\)$"
title)
collect title))))
'string<)))It still probably needs slightly more testing, but has the following good properties:
1) unlike org-roam's "next" and "prev", it doesn't get 'confused' by sync-conflict or other 'spurious' duplicate files.
2) unlike org-roam's "next" and "prev", it doesn't require that the current daily be saved to disk before operating. This has a couple of advantages. I'm always irritated by org-roam when I open today's dailiy, but haven't saved any entries in it yet, and just want to see yesterday's notes. I don't want to have to save today's ('empty') daily first. The above code solves this. The other advantage is this: say I want to see some of my early notes, say things I have for 2015, but I don't know when they are. I can use org-node-goto-daily in 'interactive' mode to choose <2015-12-31>, which probably doesn't exist, and then just use org-node-goto-prev-day to find the last daily that I do have for 2015, without first having to save a completely empty daily for <2015-12-31> (which I don't want to do, since it doesn't really exist).
3) It's instantaneous, as far as I can tell, at least on desktop, and nothing touches the filesystem except for the actual opening of dailies.
emacsomancer
commented
3 months ago Also:
(defvar-keymap org-node--daily-repeat-next-prev
"-" #'org-node-goto-prev-day ;; "decrease date"
"y" #'org-node-goto-prev-day ;; "yesterday"
"p" #'org-node-goto-prev-day ;; "previous date"
"+" #'org-node-goto-next-day ;; "decrease date"
"t" #'org-node-goto-next-day ;; "tomorrow"
"n" #'org-node-goto-next-day ;; "next date"
)
;;;###autoload
(defun org-node-goto-prev-day ()
"Go to the first extant daily before the day of the current buffer."
(interactive)
(org-node-goto-daily "--" 1 "d")
(set-transient-map org-node--daily-repeat-next-prev nil nil
"Use `-', `p', or `y' to continue going back in time.\
Use `+', `n', or `t' to switch to moving forward in time."))
;;;###autoload
(defun org-node-goto-next-day ()
"Go to the next extant daily after the day of the current buffer."
(interactive)
(org-node-goto-daily "++" 1 "d")
(set-transient-map org-node--daily-repeat-next-prev nil nil
"Use `+', `n', or `t' to continue moving forward in time.\
Use `-', `p', or `y' to switch to going back in time."))(ok, created a pull request for these, just in case.)
meedstrom
commented
3 months ago unlike org-roam's "next" and "prev", it doesn't require that the current daily be saved to disk before operating. This has a couple of advantages. I'm always irritated by org-roam when I open today's dailiy, but haven't saved any entries in it yet, and just want to see yesterday's notes. I don't want to have to save today's ('empty') daily first. The above code solves this. The other advantage is this: say I want to see some of my early notes, say things I have for 2015, but I don't know when they are. I can use org-node-goto-daily in 'interactive' mode to choose <2015-12-31>, which probably doesn't exist, and then just use org-node-goto-prev-day to find the last daily that I do have for 2015, without first having to save a completely empty daily for <2015-12-31> (which I don't want to do, since it doesn't really exist).
Oh, interesting. I was wondering how one would jump to an arbitrary date. I agree with that behavior.
Although I face an additional obstacle as I have my Emacs setup to save on every buffer change. I suppose after I jump to <2015-12-31>, I can immediately undo to clear out the auto-inserted text, and then it's just an unmodified empty buffer as far as emacs is concerned.
Great minds think alike! I also came up with the sorted list of date strings ;)
And after I generalized a concept of series, I saw the need for a transient keymap. I'm thinking a Transient menu that shows every series of which the node is part. I'd like to be able to type let's say "M-s M-s" to bring up the menu, then choose "d" for dailies, then spam "n"/"p" for next and previous.
And just realizing as I type this, but it'd also be so cool if you could also see a cycling list of breadcrumbs in that Transient! But it's out of my skill level with Transient for a few months probably, that's a hard library to learn.
BTW. Here's how I think an user will define arbitrary series:
(setq org-node-series
'(("d" "Dailies"
org-node--default-daily-classifier
org-read-date)
("s" "Star Trek episodes"
(lambda (node)
(when (member "startrek" (org-node-get-tags node))
(org-node-get-title node)))Basically, the classifier function does the magic: it should return a string to sort on (like YYYY-MM-DD), or nil if the node is not part of the series.
Code's up on "dev" branch, but still refactoring.
A lot of it was dead ends and I felt a bit frustrated with myself,
You ok now? It is a pretty non-straightforward problem, I found. Dead-ends are necessary :)
Thanks for spotting the truename issues! They don't happen on my end. I don't much like to use file-truename or expand-file-name unless I understand why, because they are slow (at least the latter, not sure about the former).
I wonder how it fails. Take a line like (string-prefix-p org-roam-directory file). Presumably org-roam-directory is "~/roam/" or whatever? So then what is FILE, I wonder, for it to fail. Oh, must be a symlink, right? But org-roam-directory itself could involve symlinks too, so one should actually wrap that in file-truename too just in case.
But symlinks shouldn't usually be a problem, if one is using roam to jump to a file in roam directory, the buffer will use the symlink name. Likewise when one uses directory-files-recursively on the roam directory, the results will be children of it. I think. But ohh, the issue might be ~ getting mixed up with /home/$USER!
Ok, I have a test for you. Does it work if you use abbreviate-file-name instead of file-truename?
Scratch that -- does it work if you use abbreviate-file-name on org-roam-directory, and do nothing with FILE?
The file paths in Org-node's tables have all been run through abbreviate-file-name, to be compatible with org-id.
emacsomancer
commented
3 months ago I think (still-testing) it works, if I run abbreviate-file-name on the org-roam directories.
(Oh, you just noted that.)
meedstrom
commented
3 months ago Nice. That makes sense!
meedstrom
commented
3 months ago I wouldn't advise cloning dev at the moment fwiw. Got stuff half-done in there.
emacsomancer
commented
3 months ago Although I face an additional obstacle as I have my Emacs setup to save on every buffer change. I suppose after I jump to <2015-12-31>, I can immediately undo to clear out the auto-inserted text, and then it's just an unmodified empty buffer as far as emacs is concerned.
I was thinking about a different issue, but it's related. Even marking "nascent" dailies as unmodified, they still clutter up the buffer-list.
It's a bit tricky though, as I don't think one necessarily wants the auto-inserted stuff be cleared out - what if the user has navigated to a "nascent" daily for the purposes of adding text there?
Would the idea be to clear out the text on a buffer change before (somehow) other buffer-change hooks (say, auto-save) operate?
I was thinking, instead, to just kill the buffer on buffer-change (if it's unmodified, i.e. unsaved, since we marked it as unmodified if it's a "nascent" daily), so it doesn't appear in buffer lists either. Wonder if that would work. Still an issue of order of operations for buffer-change hooks though, I imagine.
meedstrom
commented
3 months ago You know, it's not a bad idea to kill a blank unmodified buffer (especially if it has no file on disk yet). That can be a good initfile snippet. But not sure a package should go so far as to do this for the user.
emacsomancer
commented
3 months ago You know, it's not a bad idea to kill a blank unmodified buffer (especially if it has no file on disk yet). That can be a good initfile snippet. But not sure a package should go so far as to do this for the user.
Maybe if "nascent" dailies are specially marked somehow they would be okay to kill?
I suppose otherwise we just suggest potential things to the user that they could add.
meedstrom
commented
3 months ago Hm. Here you do a regxp on the file-title-or-basename.
(string-match-p ;; match for YYYY-MM-DD nodes only
"^\\(?:[0-9][0-9]\\)\\{1,2\\}-\\(?:1[012]\\|0?[1-9]\\)-\\(?:0?[1-9]\\|[12][0-9]\\|3[01]\\)$"
title)I think I'll nab that for org-node--default-daily-classifier :) I was going to be lazy and expect people with fancy titles to write their own lambdas.
meedstrom
commented
3 months ago But... what is going on in this regexp?
emacsomancer
commented
3 months ago Like a magpie, I stole bits from elsewhere. Mainly from here.
?: is a non-capturing group indicator, but I don't quite grok how it works.
The month part says something like "1 followed by 0,1 or 2 OR 0 followed by a number between 1 and 9". The day part "0 followed by a number between 1 and 9 OR 1 or 2 followed by a number between 0 and 9 OR 3 followed by 0 or 1". I think the year part says "a pair of numbers each between 0 and 9, twice". Then ^ to match string beginning and $ to match string end and two literal -s in-between.
One general lingering question is whether we expect everyone to name dailies the same way. Maybe have something like the monstrous regex (or figure out a better way; maybe via rx; or else some non-regex way; I looked through Org things a bit trying to see if anything would work, but no luck) as a customisable default.
meedstrom
commented
3 months ago Ok, I've figured it out! Pushed to main. https://github.com/meedstrom/org-node?tab=readme-ov-file#daily-notes-and-other-node-series
I made sure that you can undo & avoid saving the buffer ;) Took a page from you and defined the dailies as a series of files by default, so it works for the same reason. That needed some refactoring but now it's even more generalized. Thanks for the inspiration.
Opted not to ship the following commands, even though many people are used to having them on some key. They're not strictly necessary to have, and they'd need a new set of commands for every series.
;; Popular wrapper commands. Prolly won't ship as code, but as readme snippet
(defun org-node-goto-daily ()
(interactive)
(org-node--series-jump "d"))
(defun org-node-goto-today ()
(interactive)
(let* ((series (alist-get (sxhash "d") org-node--series-info))
(item (assoc (format-time-string "%Y-%m-%d")
(plist-get series :sorted-items))))
(when (or (null item)
(not (funcall (plist-get series :try-goto) item)))
(funcall (plist-get series :creator)
(format-time-string "%Y-%m-%d")))))
(defun org-node-goto-next-day ()
(interactive nil org-mode)
(org-node--series-goto-next "d"))
(defun org-node-goto-prev-day ()
(interactive nil org-mode)
(org-node--series-goto-previous "d"))Btw! You had a convenience option for the default daily template key. I envision that users can redefine org-node--default-daily-creator to something that hardcodes that key. Maybe like:
(defun org-node--default-daily-creator (sortstr)
(require 'org-roam-dailies)
(org-roam-dailies--capture
(encode-time (parse-time-string
(concat sortstr (format-time-string " %H:%M:%S %z"))))
t "d"))(Forgive the format-time-string hack, couldn't figure out org-encode-time things)
emacsomancer
commented
3 months ago How should I pass the daily template key now though? Like that?
Also, calling org-node-goto-daily seems to assume I want to capture a note, rather than just visit the daily. I haven't had a chance to look at the actual new code to understand what it does/expects.
meedstrom
commented
3 months ago Yes, well, that snippet I posted would be minimalist if you weren't gonna edit anything else. But realistically I should have posted this, since it lets you take charge of details:
(setq org-node-series
'(("d" :name "Dailies"
:classifier org-node--default-daily-classifier
:whereami org-node--default-daily-whereami
:try-goto org-node--default-daily-goto
:prompter (lambda (_series) (org-read-date))
:creator (lambda (sortstr)
(require 'org-roam-dailies)
(org-roam-dailies--capture
(encode-time (parse-time-string
(concat sortstr (format-time-string
" %H:%M:%S %z"))))
t "d")))))Also, calling org-node-goto-daily seems to assume I want to capture a note, rather than just visit the daily.
Yes, it did not pass a GOTO argument. Fixed on main (or you can use that snippet, which also has the fix).
I didn't notice, because my Roam capture template looks like this. Effectively never capture.
(setq org-roam-dailies-capture-templates
`(("d" "default" entry "* %<%H:%M>\n%?" :if-new
(file+head "%<%Y-%m-%d>.org"
,(string-join '("#+title: %<%Y-%b-%d>"
"#+filetags: :daily:")
"\n")
:immediate-finish t
:jump-to-captured t))))
nickanderson
commented
3 months ago Oh I hope to make time to come back to this thread. I use dailies a lot. Most recurring meetings get their own, I think I am nearing 100. I also find the process of defining the templates cumbersome.
I personally like the yesterday/tomorrow functions and i use the goto/capture date function occasionally.
I have disliked how dailes are their own funnel. I have many daily that are logically related to other collections of notes and i have wished to store those files in closer proximity.
meedstrom
commented
3 months ago I have disliked how dailes are their own funnel. I have many daily that are logically related to other collections of notes and i have wished to store those files in closer proximity.
Oh, then I might be on to something with another series I made, https://github.com/meedstrom/org-node/blob/55b8cb4ce428efdb1c4a57ddad5a0be3456efa62/org-node.el#L1444-L1454
Or not. It depends on those logical relations of yours :) Maybe series are not the answer.
But. While some stuff can't be described as a single linear series, it can be if you have many intersecting series.
Made a short screencast of using org-node-series-dispatch, and flipping between two series, just for an idea of the UX: https://ufile.io/h6hi7vzb
emacsomancer
commented
3 months ago org-node--series-goto-previous doesn't work quite right when the current node hasn't been saved. for instance, if I do a org-node-goto-today for today and don't save the file, when I do (org-node--series-goto-previous "d") it goes to the entry for 2024-08-13 (i.e. two days ago, rather than yesterday).
emacsomancer
commented
3 months ago Also, for the org-node-series, could we implement an ":after" property? Currently there is a "regression" in that nascent nodes are marked as "unsaved", and I would like to be able to do something like (set-buffer-modified-p nil).
meedstrom
commented
3 months ago Interesting, and yea sure. The more lambdas the merrier :P Although, it occurs to me you could do that in the :creator lambda, let it call (set-buffer-modified-p nil) at the end. Let me know if that's not sufficient.
I think it's also possible in your capture template, there's an :after-finalize property or some such.
nickanderson
commented
3 months ago Oh, then I might be on to something with another series I made,
Or not. It depends on those logical relations of yours :) Maybe series are not the answer.
Yeah, I tend to refer to "dailies" as "time series". I might not capture to that series every day but they are stored based on the time YYYY-MM-DD.org usually, but sometimes YYYY-MM.org
Logical relation .... for example, I might have a collection of notes about a specific team and then I might have several time series notes related to that team as well, e.g. daily standups, 1:1s etc. Or I might have a collection of notes about a customer and then time series notes that are related, like monthly, quarterly or whatever periodic status meetings. I have a bunch of notes about org-mode, I might want to store my notes for Emacs ATX meetups closer to my other org-mode notes instead of off burried under dailies.
But, changing where dailies are stored would affect my use (quite limited but still) of org-roam-ui making it harder to get nice screenshots of the graph for marketing :D
Made a short screencast of using org-node-series-dispatch, and flipping between two series, just for an idea of the UX: https://ufile.io/h6hi7vzb
yeah going forward and back across a series isnt something i do often but I like the functionality, goto previous is something i do sometimes use in org-roam.
meedstrom
commented
3 months ago But, changing where dailies are stored would affect my use (quite limited but still) of org-roam-ui making it harder to get nice screenshots of the graph for marketing :D
Oh but you can exclude them from org-roam-ui, based on either file path or tag! It's one of the settings in the web UI.
emacsomancer
commented
3 months ago
org-node--series-goto-previousdoesn't work quite right when the current node hasn't been saved. for instance, if I do aorg-node-goto-todayfor today and don't save the file, when I do(org-node--series-goto-previous "d")it goes to the entry for 2024-08-13 (i.e. two days ago, rather than yesterday).
This is the case for any "goto-previous" event when the current node doesn't really exist (yet), it seems. It skips over the actual previous item in the series and goes to the one preceding that. An off-by-one issue?
[Note: still need to test if the "goto-next" has the same issue.
I've checked and it just seems to be org-node--series-goto-previous (with no optional next arg) that has a problem when the current node is still nascent.
If I change (in org-node--series-goto-previous) (drop (1+ (length head)) (plist-get series :sorted-items)) to omit the (1+ ...), it works when the current node is still nascent (but then fails if it isn't, i.e. it remain on the same node).
Looking at the code org-node--series-goto-previous, I'm not quite sure how to fix this. In theory we could check if the node is nascent I suppose, and then only do (1+ ...) when it isn't. Though there may be a better way.]
nickanderson
commented
3 months ago Oh but you can exclude them from org-roam-ui, based on either file path or tag! It's one of the settings in the web UI.
I will have to look more closely but at least currently dailies is a toggle, probably could do it with file paths or tags just a lot more fiddly.
meedstrom
commented
3 months ago @nickanderson
Logical relation .... for example, I might have a collection of notes about a specific team and then I might have several time series notes related to that team as well, e.g. daily standups, 1:1s etc. Or I might have a collection of notes about a customer and then time series notes that are related, like monthly, quarterly or whatever periodic status meetings. I have a bunch of notes about org-mode, I might want to store my notes for Emacs ATX meetups closer to my other org-mode notes instead of off burried under dailies.
Do you mean you have many files with the prefix 2024-08-14? Like 2024-08-14-quarterly.org? No, I assume they are all nested into one file per day?
What would it mean to store the Emacs ATX meetups closer? That you don't have to link? Say you got a subheading in 2024-08-14.org called "* How Emacs ATX went on 2024-08-14", that has its own ID, and you link to that from some-other-node.org. where relevant. Is that what you mean that's inconvenient?
nickanderson
commented
3 months ago Do you mean you have many files with the prefix 2024-08-14? Like 2024-08-14-quarterly.org? No, I assume they are all nested into one file per day?
No, I use the date as the filename. I have things like this made up example ...
What would it mean to store the Emacs ATX meetups closer?
I mean closer on the file system. I like to sync some notes to some places and it's a bit painful to have things like EmacsATX.org in a completely different tree than the meetings. I'd probably prefer to have:
That you don't have to link? Say you got a subheading in 2024-08-14.org called "* How Emacs ATX went on 2024-08-14", that has its own ID, and you link to that from some-other-node.org. where relevant. Is that what you mean that's inconvenient?
Oh I use links and navigating between the files is not an issue, its more about preferred file storage location and how that impacts my ability to share shards of my notes in different ways. Like this subtree is actually a link over to a completely different directory which does git syncing to a repo. This other subtree is synced via syncthing to my phone.
meedstrom
commented
3 months ago Oh, that's great food for thought.
meedstrom
commented
3 months ago Bit of a breaking change, in case either of you wrote a :prompter lambda. I changed from integers to strings for looking up org-node--series-info. So access with (cdr (assoc... rather than (alist-get (sxhash....
Didn't know how to handle the transition with code, but that's the freedom of having almost no users :) More room to say "fuck it".
Not looking forward to getting on Melpa.
emacsomancer
commented
3 months ago Thanks! Just updated the goto-today function; I think that's all I had.
Suppose could do a melpa-stable branch? Not sure how much that would help as most people just do melpa probably.
meedstrom
commented
3 months ago Actually my "melpa" branch will be relatively stable, and "main" is for people who clone the git repo :)
If I used package.el and needed to roll back a package, I wouldn't do it by "falling back on melpa-stable" anyway as it just lets you go back one version, better how alphapapa does it and check theelpa/ directory into git.
But it doesn't matter. Breaking changes are no less breaking between stable versions. Even if Emacs had a good version infrastructure, having users still limits your freedom to refactor :)
meedstrom
commented
3 months ago Oh. The next time I do a breaking change on series, I'll add a :version property. Easier to write code to handle the upgrade then.
meedstrom
commented
2 months ago So @nickanderson, you got multiple files to represent the same day, in different directories.
A question about this list:
How does navigating "next/previous daily" end up working with org-roam-dailies? I suppose it picks a random one of these, and cycles through the other two, and then moves on to the next day?
And if you goto and choose a date on the date picker, which file does it choose to visit?
The good news with org-node: you should be able to move these files out of the "daily" root directory, and put them wherever. Just have to change the :classifier lambda so it just checks if file-name-base looks like an yyyy-mm-dd, but disregards the directory.
The bad news: :classifier has to put some unique string in the return value. So I am thinking you would perhaps concat the date string with the file-name-directory. And then the :whereami and :prompter... yea... I'll await your answers to the above questions first :)
nickanderson
commented
2 months ago Don't design around my use patterns. I am /very/ slow to adjust my workflow patterns and stuff.
you got multiple files to represent the same day, in different directories.
Yep, it's more organized by type of content and day. So if I have 5 meetings in a day, i will very likely have at least 6 daily files. One for each meeting, and one for my general worklog where I clock time. And those entries typically link over to the specific meeting note (I first capture to the specific meeting template, then capture to worklog and use the automatic backlink %a.
How does navigating "next/previous daily" end up working with org-roam-dailies? I suppose it picks a random one of these, and cycles through the other two, and then moves on to the next day?
It works fine. Each one of those example files above would have it's own daily template. It doesn't pick some random series it picks the previous or next existing file for the series based on the file I am currently visiting. It's not something I use a lot, often I will jump to dired and then pick the day I want. Dired is faster in my experience.
And if you goto and choose a date on the date picker, which file does it choose to visit?
Also works fine in my experience. I use this occasionally to seed a meeting note in the future when I have something special to put on the agenda. After choosing the date, I am prompted to pick a template and it finds the right file and initializes it if necessary as expected.
meedstrom
commented
2 months ago Don't design around my use patterns. I am /very/ slow to adjust my workflow patterns and stuff.
I won't, but I like hearing about use cases! Often leads to some generalization I'm missing.
OK, treating them as totally separate series, that sounds workable.
I was puzzled how org-roam did next/previous behavior as you describe, then I saw that even though they have just one list of files, it's sorted by directory... So if you're at the end of the EmacsATX directory and navigate to the next daily, you'll end up in a different directory, yea?
Don't feel compelled to try this, I'm just toying around, but I think something like this would work for an "Emacs ATX" series that needs not sit in the dailies dir:
(setq org-node-series-defs
(list
'("e" :name "Emacs ATX"
:version 2
:classifier (lambda (node)
(let ((path (org-node-get-file-path node)))
(when (string-search "/EmacsATX/" path)
(let ((ymd (org-node-helper-filename->ymd path)))
(when ymd
(cons ymd path))))))
:whereami (lambda ()
(when (string-search "/EmacsATX/" buffer-file-name)
(org-node-helper-filename->ymd buffer-file-name)))
:prompter (lambda (key)
(let ((org-node-series-that-marks-calendar key))
(org-read-date)))
:try-goto (lambda (item)
(org-node-helper-try-visit-file (cdr item)))
:creator (lambda (sortstr key)
(let ((org-node-datestamp-format "")
(org-node-ask-directory "~/roam/org-mode/EmacsATX/meetings/"))
(org-node-create sortstr (org-id-new) key))))))
Thinking about dailies. One option is to leave all that outside org-node's scope, because org-journal and org-roam-dailies work well enough (mod slowness, which we could hack with shims in the right places).
I am not a power user of dailies, I mainly just go to today's entry (actually a homemade function called
my-last-daily-file) and cycle between next and previous.But perhaps I'm not a power user because I never liked the existing solutions!
org-roam-capture-with too many purposes for my tastes. Plus the word "capture" is reused in many functions that don't do any capturing at all, so it's confusing.Actually, looking at the daily-navigation commands in https://github.com/meedstrom/org-node/pull/25, it taught me that
(org-read-date)has a language! Namely, you can type "+" to yield tomorrow's date and "-" for yesterday. Bit embarrassing that I didn't learn it in ~9 years of Org, but yeah, I never really knew what to do with that little calendar popup. Methinks it could be patched with more obvious hints.I think it's interesting to ask if we can just lean on
(org-read-date)as much as possible. Bare-metal interfaces create opportunities for the user to learn the underlying basics!No actions for today/tomorrow/yesterday?
Let's suppose there's a command
M-x org-node-goto-daily, which shows that little calendar popup. In practice, typingRETwould just go to today, typing+ RETto tomorrow,- RETto yesterday. And there are the more complex strings like+2wwhich goes two weeks in the future, of course.I see I'm reinventing
org-roam-dailies-goto-date(the potential of which I hadn't really understood until now). My proposal is that perhaps we should intentionally ship only this command, and none for today, tomorrow, yesterday. 1-2 extra keystrokes in most cases, comfortable enough.Generalized actions for next/previous?
There should still be commands for next and previous, relative to current buffer, but ya... for that, I've actually been using homemade commands
my-next-file-in-dir,my-prev-file-in-dir, which works everywhere and not only for dailies, so I have the feeling that this is the correct thing to recommend.Directory-independent logic?
Another thing, out of curiosity. Is it possible to just do away with a dailies directory? By looking at... the title, or a
:CREATED:property, or something. Not sure if that's sane, since non-dailies could be mixed in. Though that can be filtered with a tag:daily:or just checking if the directory is "daily/". So you could have an optional "daily" directory that does effective filtering work without ever setting any Lisp variable to point to it.Such a system could potentially be reused to let you just browse any notes by creation date, not only dailies. That doesn't seem that important though. Moreso that it might be easy to make compatible with org-journal's weekly/monthly journal, or a datetree, so that it need not be one file per day.