`org-gtd-process-inbox` skipping every other item

[jlcolbert]

Hello again!

I noticed that, without changing my configuration (and even testing older configurations of mine, just to double check), running org-gtd-process-inbox is starting to skip every other inbox item and thus not processing everything in inbox.org.

I noticed in the CHANGELOG that you maybe have noticed a similar issue and said something about disabling org-edna-mode? Do you have a recommended configuration for the bug?

Thanks!

[Trevoke]

Hmm. This defect showed up a while back. By chance did you update the package, and was it working before the update? (if I can somehow point to a change of mine as breaking the behavior it's much easier).

I know you said that your configuration seemed to not be the issue, but can you share it with me anyway?

And this happens when things get refiled to anything? incubate, project, trash, delegated, etc. ? (I expect the answer to be yes)

[jlcolbert]

This was my config about two weeks ago and was working until yesterday or the day before, when I noticed the issue: https://jlcolbert.github.io/doom-emacs-config/#org79b0e79 I just retried that config, and ran into the same issue.

I tried also following this config, just to see if maybe rearranging things would make a difference: https://github.com/sunnyhasija/Academic-Doom-Emacs-Config

Finally, I tried installing and configuring from the 2.0.0 branch, but that seemed to do some weird stuff with refiling.

[jlcolbert]

And I only noticed the issue when processing the inbox, so yes, I suppose it would be that processing process (ha) and having things refiled to the actionable.org file. Although, maybe it has something to do with the capture template and how things are captured in inbox.org?

[Trevoke]

It could be the capture template, yes -- did you customize those in any way? What do they look like now, if you ask emacs (C-h v org-capture-templates RET) ?

Also, this might sound dumb, but did you restart emacs? the current configuration, which I'm working on simplifying significantly, is kinda load-order dependent, so it might be possible that it loaded a slightly different configuration than expected.

[jlcolbert]

No, I didn't customize the templates, which is why this is so weird to me.

Here's the result from running C-h v org-capture-templates RET:

Value
(("i" "Inbox" entry
  (file "/Users/jaycolbert/gtd/inbox.org")
  "* %?\n%U\n\n  %i" :kill-buffer t)
 ("l" "Todo with link" entry
  (file "/Users/jaycolbert/gtd/inbox.org")
  "* %?\n%U\n\n  %i\n  %a" :kill-buffer t))

Original Value
nil

And yes, I restarted emacs. I try to update everything on my computer every day. If it helps, I'm using the emacs-mac package through Homebrew, and I'm using the Doom Emacs configuration.

Here's my system's info from running doom/info in Doom:

generated    Nov 06, 2021 22:46:46
system       MacOS 11.6.1 Darwin 20.6.0 x86_64 mac
emacs        27.2 ~/.emacs.d/ -> ~/.emacs.d/
doom         21.12.0-alpha HEAD -> develop b4cc1c84d 2021-11-07 01:53:17 +0100 ~/.doom.d/ -> ~/.doom.d/
shell        /usr/local/bin/fish
features     NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS XIM MODULES THREADS JSON PDUMPER LCMS2 GMP
traits       gui server-running envvar-file
modules      :completion (company +childframe) vertico :ui doom doom-dashboard hl-todo modeline ophints (popup +defaults) workspaces zen :editor (evil +everywhere) file-templates fold snippets :emacs dired electric undo vc :term eshell :checkers (syntax +childframe) :tools editorconfig (eval +overlay) lookup lsp pdf :os macos :lang (clojure +lsp) emacs-lisp latex markdown (org +dragndrop +hugo +roam2) sh :config (default +bindings +smartparens)
packages     (org-gtd) (org-transclusion :recipe (:host github :repo nobiot/org-transclusion :branch main :files (*.el)))

And as far as load order, I put them in the order you have in the README.

[Trevoke]

My word it's already been two weeks. Sheesh. I need to add a few inbox processing tests just in case there's a weird combination of processing items in a particular order (e.g. "this is a project, this is knowledge, this is incubated" -> bug) that would cause this to happen, but I haven't come across this yet, so I'm pretty horribly confused.

I wonder if there's something about ... org-mode "heading level" settings that could cause this. but you said you didn't change anything about your configuration...

[jlcolbert]

The only thing I can think of is that the way you have the load order example is perhaps not easily translatable to how it should be done in a Doom config? But I'm not knowledgable enough to know.

[jlcolbert]

Small update: I tried adding in :empty-lines 1 in the capture template, just to see if that helped, and got the same behavior.

[jlcolbert]

And I just tried adding items to the inbox manually. Same skipping behavior. Would it help if I recorded a gif/video of it happening?

[Trevoke]

I think I know what's happening.

Given the following inbox:

* 1
* 2
* 3
* 4

When you process the inbox, you will go through 1 and 3, and it will leave 2 and 4 in the inbox.

Here's why I think this shouldn't be happening (obviously this reasoning is off somewhere):

1. I am using org-map-entries, which goes from heading to heading
2. The documentation clearly states that if I want to preserve continuity despite possibly moving (e.g. archiving, refiling) headings, I need to set org-map-continue-from properly - and I stole the snippet from https://stackoverflow.com/a/27043756/234025 , which I thought was meaningfully enough the same thing that I was doing

So what is necessarily happening is that there's something in the way it stores the point (the cursor) to decide how to resume that isn't quite correct.

[jlcolbert]

Ooooh, that makes sense.

[Trevoke]

@jlcolbert Okay, I'd like you to try something:

1. make sure your inbox is empty
2. quit emacs
3. start emacs
4. add four or five things to the inbox
5. process the inbox [ I expect this will be processed normally ]
6. without saving the inbox buffer, add four or five things to the inbox
7. process the inbox [ I don't think this will be processed normally ]

Let me know if this works that way?

[jlcolbert]

Nope, still not working.

I did a screen recording of the whole process w/ my configuration and me narrating everything.

https://user-images.githubusercontent.com/17013952/143152884-f1f48859-f63d-4d3f-b8d6-c628597b1170.mov

[Trevoke]

Hmm.. Okay. Thanks, that's pretty informative. org is a bit of a complex beast, so there's a lot of possibilities.

I would like you to try one more thing, now that we've established this is broken this way.

TLDR I would like you to edit the code and try something. Don't worry, it's a little bit easier than it may seem, if you're not familiar with coding. All it is is: finding the source code, making the change, and making emacs process the change.

And if what's below still looks like a lost language, let me know, I can do a short recording to show this too.

If you're familiar with coding / emacs lisp:

I want you to edit the code for org-gtd-process-inbox and make the following change:

FROM

     (setq org-map-continue-from (org-element-property
                                  :begin
                                  (org-element-at-point)))

TO

     (setq org-map-continue-from (point-min))

if you're not familiar with coding/emacs

Get to the source code of the package

1. C-h f org-gtd-process-inbox RET -- this opens a help buffer
2. you'll see org-gtd-process-inbox is an interactive Lisp closure in ‘org-gtd.el’. - click on org-gtd.el
3. You might need to scroll up to bring the code of the function into view

Find the code to edit

The function will have a chunk of code that looks like this:

  (org-map-entries
   (lambda ()
     (setq org-map-continue-from (org-element-property
                                  :begin
                                  (org-element-at-point)))
     (org-narrow-to-element)
     (org-show-subtree)
     (org-gtd--process-inbox-element)

We will want to focus on setting the variable org-map-continue-from which spans three lines. We will change the code from:

     (setq org-map-continue-from (org-element-property
                                  :begin
                                  (org-element-at-point)))

from

     (setq org-map-continue-from (point-min))

Once that is done, save the file.

When the file is saved, put the point (the cursor) right after the last parens of the function:

(org-gtd-find-or-create-and-save-files))| where the | is where you should put the point

And finally hit C-x C-e -- this will make emacs evaluate the function and load the new definition in memory.

Once the change is processes

Now.. Try to add a few things to the inbox and process the inbox again.

If this doesn't make the problem go away, then I'm afraid I really need to start from scratch and put together a different solution entirely.

[Trevoke]

(alternatively you can just download the most recent version when it hits melpa, I just pushed that change, I couldn't see what it would break, the tests still passed)

[jlcolbert]

sigh Sadly, still no. I wonder if there are any variables Doom sets that might interfere? Could it have to do with where the point is when I clarify-finalize?

[rickyson96]

Dunno if it helps, but I have a similar function to archive DONE tasks in my config.

(defun my/org-archive-done-tasks-tree ()
  (interactive)
  (org-map-entries
   (lambda ()
     (org-archive-subtree)
     (setq org-map-continue-from (org-element-property :begin (org-element-at-point)))) "/DONE" 'tree))

When archiving the following org tree, it skips over every other element (leaving the * DONE task2 and * DONE task4) in place.

* Test
    * DONE task1
    * DONE task2
    * DONE task3
    * DONE task4

I have tried changing the org-map-continue-from using point-min as suggested, but apparently, the issue remains.

I think this issue happens on the org-map-entries function, but I haven't had time to inspect whether this is coming from doom or maybe something about my org installation.

[jlcolbert]

I don't know if this is relevant, but I saw this in the Doom config code:

;; Fix #462: when refiling from org-capture, Emacs prompts to kill the ;; underlying, modified buffer. This fixes that. (add-hook 'org-after-refile-insert-hook #'save-buffer)

[Trevoke]

Hmm... Maybe? I am not sure how it would impact things, though it certainly might. I would take a look, but I have finished doing a rework of the flow, making it more... emacs-zen, not trapping the user in the flow. This "trap" was something I used to really care about as a way to force you to stay focused on the inbox processing, but that's not really the emacs way.

So, anyway - this is fixed in the 2.0.0 branch, and that's almost done, I "just" need to write the documentation now.

I've got a lot more features I want to add but now I'm a lot more comfortable with the code I do have, and most importantly, it's tested.

[jlcolbert]

Awesome! I'm looking forward to it. I've been snooping around the 2.0.0 branch, and I really like the differences I see. I appreciate you taking the time to work through this.

[Trevoke]

I've released 2.0 RC1 (merged branch into master). There is a change in the order of operations: now as a user you get control to refine the item before you choose what kind of item it is for Org GTD.

Please let me know how this works for you and whether this can be satisfactorily closed!

[jlcolbert]

It still doesn't work for me. I think it has something to do with how the Doom configuration configures org-capture and org-refile, but I'm not knowledgable enough in Emacs Lisp to figure out what's going on. I've tried disabling the Doom hook that sets the default org-capture settings, but it still doesn't work. I have to go in and set the org-capture templates manually to get them to work, and I get a message in my buffer saying with-org-gtd-context isn't a valid function. And once I get things into the inbox, it's still not processing them correctly. I just don't think this package is compatible with Doom, sadly.

[Trevoke]

Okay, something else is up. The org-gtd-context thing is a problem that I thought had been resolved but obviously still exists. Maybe doom uses straight.el, a package manager that has caused me a lot of issues. I think I know how to fix this but I have to get back to a computer first.

If this next fix doesn't help you then I may ask if we can do a video call / screen share at some point.

On Tue, Dec 28, 2021, 11:14 Jay L. Colbert @.***> wrote:

It still doesn't work for me. I think it has something to do with how the Doom configuration configures org-capture and org-refile, but I'm not knowledgable enough in Emacs Lisp to figure out what's going on. I've tried disabling the Doom hook that sets the default org-capture settings, but it still doesn't work. I have to go in and set the org-capture templates manually to get them to work, and I get a message in my buffer saying with-org-gtd-context isn't a valid function. And once I get things into the inbox, it's still not processing them correctly. I just don't think this package is compatible with Doom, sadly.

— Reply to this email directly, view it on GitHub https://github.com/Trevoke/org-gtd.el/issues/72#issuecomment-1002203507, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQSSLSEMM73NYCSTUKOYTUTHV6RANCNFSM5HH47LJQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: <Trevoke/org-gtd. @.***>

[jlcolbert]

Yes, it does use straight.el! https://github.com/hlissner/doom-emacs/blob/develop/docs/getting_started.org#package-management

When one wants to install and configure a package, you put (package! package-name) in packages.el in your ~/.doom/ or ~/.config/doom directory. Then, in config.el in the same directory, you use use-package! as the way to configure, which is a Doom wrapper around use-package; it doesn't use :ensure (which I think comes from straight.el) and I don't tend to see people using :custom but rather just :config and sometimes :init.

And like I said, Doom comes with a lot of things pre-configured, which is really helpful for newbies like me, but breaking those configs can sometimes take a bit of figuring out. And like with org-capture in this case, I can see where the Doom configuration is and why the loading-order is screwing with things, but I'm not always the best with figuring out what to do about it. You can also use the after! macro to configure packages, especially ones already installed with Doom, so that your configurations kick in after the Doom ones; otherwise, the Doom ones overwrite them. It's similar to the :after call in use-package.

So I know with the new way you have the templates kicking in, either you or people using Doom would need to make sure they kick in after Doom loads the +org-init-capture-defaults-h function on the org-load-hook, and then there's probably something to do with how Doom has org-refile configured as well. I know how to set my own templates and not have Doom override them, but I'm not sure about having other functions do it. Org-roam has its own capture templating system that works with Doom (and the person who maintains a lot of org-roam stuff uses Doom), so it might be worth looking there?

Otherwise, I don't know about the weird stuff with things skipping. It worked, and then it didn't, so I'm suspecting it's not necessarily how you had the original package configured (because I tried installing from older commits) but rather how something else in org changed when getting updated or something, either by Doom or by the org folks themselves. I saw someone else in this issue having a similar skipping problem without org-gtd. I tried doing installing and configuring in a blank Emacs (without Doom) and experienced the skipping problem, for what it's worth, even on the previous 1.x.x version.

[Trevoke]

Okay - You can upgrade to the most recent on master. You will want to add (require 'org-gtd) somewhere in your configuration -- org-gtd is definitely not in a code organization position where it's manageable to do pure autoloads, so it's easier to load all the code at once. Let's call it a work in progress..

Let me know if this works for you!

[jlcolbert]

No luck, but I'm probably not putting (require 'org-gtd) in the right spot. I see another person is having issues and uses Doom, so if they get it to work, I'll ask very nicely for how they have it configured. I'm sure right now this is user configuration error on my part. I used the sample config from the docs, adapted to Doom, and tried putting (require 'org-gtd) to load after org, and then after org-capture, and neither changed the behavior. The inbox.org document generated fine, as it always has, but the capture templates didn't set up (I had not done them manually this time), and processing still didn't work, after I added entries to the Inbox manually.

[jlcolbert]

EDIT IT WORKS. For some reason, my new install didn't pull in the latest commits, and I had to make sure I updated the package.

[jlcolbert]

I will play around with where to put the (require 'org-gtd) call so that it doesn't slow down load time too much, but whew! Even processing works!

Edit for fellow Doom people: put (require 'org-gtd) like this:

(after! org
 (require 'org-gtd))

And then:

(use-package! org-gtd
  :after org
  :demand t
  :config
  (setq org-gtd-directory "~/gtd/"
        org-edna-use-inheritance t)
  (org-edna-mode)
  :bind
  (("C-c d c" . org-gtd-capture)
   ("C-c d e" . org-gtd-engage)
   ("C-c d p" . org-gtd-process-inbox)
   ("C-c d n" . org-gtd-show-all-next)
   ("C-c d s" . org-gtd-show-stuck-projects)
   :map org-gtd-process-map
   ("C-c c" . org-gtd-choose)))

(I'm not sure if the :demand t is needed, but it's not hurting, so it stays.)

[Trevoke]

Woohoo! That's fantastic news. I just finished playing around with doom and came to the following conclusion:

1. ~/.doom.d/packages.el : (package! org-gtd) (this will load from MELPA, so you'll have to wait for the next MELPA build)
2. ~/.doom.d/config.el : (require 'org-gtd) (since doom sync installs everything, you can trust everything you need is available to load the package)

If you want to load from the latest commit (not always a great idea), you would do (package! org-gtd :recipe (:host github :repo "trevoke/org-gtd.el"))

What you have obviously also works, so let's stick with that since you also get the keybindings :D

I'll close this issue now and put a link to your comment in the other issue to help the other user. Please open new issues for any other problems... Or join the discord to talk about GTD and give me ideas for further improvements :D

[jlcolbert]

Sweet! Doom can do keybindings without a use-package! wrapper, so I'll try that.