search

ahyatt / ekg

The emacs knowledge graph, app for notes and structured data.
GNU General Public License v3.0
230 stars 19 forks source link

Org-roam import does not work #139

Closed nameiwillforget closed 7 months ago

nameiwillforget commented 8 months ago

If I run ekg-org-roam-import, the titles of the org-roam nodes appear as tags but nothing seems to be tagged by them. I've backed up my org-roam-directory and removed all but one file:

:PROPERTIES:
:ID:       a52075c8-be51-439e-9477-62091fd7c847
:END:
#+filetags: :sein:nichts:
#+title: Werden
[[file:Wissenschaft der Logik.org::*C. Werden][Werden]]
"Das reine Sein und das reine Nichts ist also dasselbe. Was die Wahrheit ist, ist weder das Sein noch das Nichts, sondern daß das Sein in Nichts und das Nichts in Sein – nicht übergeht…

Now, if I run ekg-org-roam-import, the tag werden is created but neither sein and nichts are created and if I run ekg-show-notes-with-tag to view the werden notes, no notes are displayed under it. Similarly, if I create a new note senf using the default template, and run ekg-org-roam-import again, the tag senf appears but no note is associated to it. If I print out the text variable in ekg-org-roam-import, it does print the text of the werden note. If I put

(print (ekg-note-tags note))

under

                                      (let* ((note (ekg-note-create
                                                    :text text
                                                    :mode 'org-mode
                                                    :tags (seq-difference (seq-uniq
                                                                           (cons
                                                                            (ekg-org-roam-import-title-to-tag (org-roam-node-title node) (org-roam-node-tags node))
                                                                            tags-from-links))
                                                                          (seq-union ekg-org-roam-import-tag-to-ignore
                                                                                     ekg-org-roam-import-tag-to-prefix
                                                                                     #'equal)
                                                                          #'equal))))

it prints out

("senf")

("Werden")

If I instead put

(print note)

there, it prints out

#s(ekg-note 33466969820 #(":PROPERTIES:
:ID:       1b62e4e9-59f1-450c-8e47-6b1f336227e8
:END:
#+title: senf

" 0 12 (face org-drawer fontified t font-lock-fontified t) 12 13 (fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 13 17 (face org-special-keyword fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 17 23 (fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 23 24 (fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 24 60 (face org-property-value fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 60 61 (fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 61 66 (face org-drawer fontified t font-lock-fontified t isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show org-fold--spec-org-fold-drawer--1496384421108082184 org-fold-drawer) 66 67 (fontified t) 67 69 (invisible org-modern face org-document-info-keyword fontified t font-lock-fontified t) 69 75 (face org-document-info-keyword fontified t font-lock-fontified t) 75 76 (fontified t) 76 81 (face org-document-title fontified t font-lock-fontified t) 81 82 (fontified t)) org-mode ("senf") 1708790384 1708790384 nil nil)

#s(ekg-note 33466969728 #(":PROPERTIES:
:ID:       a52075c8-be51-439e-9477-62091fd7c847
:END:
#+filetags: :sein:nichts:
#+title: Werden
[[file:Wissenschaft der Logik.org::*C. Werden][Werden]]
\"Das reine Sein und das reine Nichts ist also dasselbe. Was die Wahrheit ist, ist weder das Sein noch das Nichts, sondern daß das Sein in Nichts und das Nichts in Sein – nicht übergeht, sondern übergegangen ist. Aber ebensosehr ist die Wahrheit nicht ihre Ununterschiedenheit, sondern daß sie nicht dasselbe, daß sie absolut unterschieden, aber ebenso ungetrennt und untrennbar sind und unmittelbar jedes in seinem Gegenteil verschwindet. Ihre Wahrheit ist also diese Bewegung des unmittelbaren Verschwindens des einen in dem unterschieden anderen: sind, das aber Werden; durch einen eine Bewegung, Unterschied, der worinbeide sichebenso unmittelbar aufgelöst hat.\"
" 0 12 (font-lock-fontified t face org-drawer fontified t) 12 13 (fontified t) 13 17 (face org-special-keyword fontified t) 17 24 (fontified t) 24 60 (face org-property-value fontified t) 60 61 (fontified t) 61 66 (font-lock-fontified t face org-drawer fontified t) 66 67 (fontified t) 67 69 (invisible org-modern face org-meta-line font-lock-fontified t fontified t) 69 71 (face org-meta-line font-lock-fontified t fontified t) 71 77 (face org-meta-line font-lock-fontified t fontified t) 77 79 (face org-meta-line font-lock-fontified t fontified t) 79 80 (display #(":" 0 1 (face org-hide)) face org-meta-line font-lock-fontified t fontified t) 80 81 (display #(" s" 1 2 (cursor t)) face org-modern-tag font-lock-fontified t fontified t) 81 83 (face org-modern-tag font-lock-fontified t fontified t) 83 84 (display "n " face org-modern-tag font-lock-fontified t fontified t) 84 85 (display #(":" 0 1 (face org-hide)) face org-meta-line font-lock-fontified t fontified t) 85 86 (display #(" n" 1 2 (cursor t)) face org-modern-tag font-lock-fontified t fontified t) 86 90 (face org-modern-tag font-lock-fontified t fontified t) 90 91 (display "s " face org-modern-tag font-lock-fontified t fontified t) 91 92 (display #(":" 0 1 (face org-hide)) face org-meta-line font-lock-fontified t fontified t) 92 93 (fontified t) 93 95 (invisible org-modern face org-document-info-keyword fontified t font-lock-fontified t) 95 101 (face org-document-info-keyword fontified t font-lock-fontified t) 101 102 (fontified t) 102 109 (face org-document-title fontified t font-lock-fontified t) 109 110 (face (rainbow-delimiters-depth-1-face org-link) font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 110 111 (face (rainbow-delimiters-depth-2-face org-link) font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 111 154 (face org-link font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 154 155 (face (rainbow-delimiters-depth-2-face org-link) font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 155 156 (face (rainbow-delimiters-depth-2-face org-link) font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t rear-nonsticky (mouse-face highlight keymap invisible intangible help-echo org-linked-text htmlize-link) htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 156 161 (face org-link font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-description-global org-link-description org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 161 162 (face org-link font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-description-global org-link-description org-fold--spec-org-link-global org-link fontified t rear-nonsticky (mouse-face highlight keymap invisible intangible help-echo org-linked-text htmlize-link) htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 162 163 (face (rainbow-delimiters-depth-2-face org-link) font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show) 163 164 (face (rainbow-delimiters-depth-1-face org-link) font-lock-multiline t keymap (keymap (follow-link . mouse-face) (mouse-3 . org-find-file-at-mouse) (mouse-2 . org-open-at-mouse)) mouse-face highlight org-fold--spec-org-link-global org-link fontified t htmlize-link (:uri "file:Wissenschaft der Logik.org::*C. Werden") help-echo "LINK: file:Wissenschaft der Logik.org::*C. Werden" isearch-open-invisible-temporary org-fold-core--isearch-show-temporary isearch-open-invisible org-fold-core--isearch-show rear-nonsticky (mouse-face highlight keymap invisible intangible help-echo org-linked-text htmlize-link)) 164 831 (fontified t)) org-mode ("Werden") 1708790384 1708790384 nil nil)

I'm not sure what's going wrong.

Edit: Emacs version is 30.0.50, Org-roam version is 20240114.1941, Ekg is freshly installed from Melpa.

ahyatt commented 8 months ago

Thanks for the bug report. I see several things going wrong here, which will need several bug fixes. Please stay tuned, I'll try to commit fixes this weekend.

ahyatt commented 8 months ago

Please check out the fixes in the develop branch. Does it solve your problem?

nameiwillforget commented 8 months ago

I've pulled the development branch and evaluated ekg-logseq.el and ekg-logesq-test.el, then tried ekg-org-roam-import again, but it didn't seem to do anything differently. When I checked the created tags it didn't list the notes underneath them and when I modified the import function to show me the tags of the created notes I received the same output as before.

nameiwillforget commented 8 months ago

Correction: now the tags don't display notes that have been egk-captured either, and this is true even if I remove the develop version from the Emacs path and start a new Emacs instance. Maybe the importing through the develop version corrupted the database somehow? I'll delete the database and try it again with a new one.

nameiwillforget commented 8 months ago

I've removed the old database and created a new one, but I don't think it changed anything. It seems to display tags inconsistently. I'll attach a screenshot. I don't know what's going on. Refreshing the tags doesn't help either, and it shows no notes if I use ekg-show-notes-with-tags-all without any tags (and also if I use-any). But it shows all notes if I use ekg-show-notes-for-today.

ekg

ahyatt commented 8 months ago

I think what has happened is that you removed the database but didn't close it first, which would explain some of the issues you are seeing. Let me explicitly show you how I'm testing on your example, from the develop branch.

  1. M-x ekg-close
  2. (setq ekg-db-file "~/tmp/test-ekg.db") (in *scratch* or in ielm)
  3. ekg-show-notes-latest-captured to verify it is an empty db
  4. M-x ekg-logseq-sync
  5. Refresh latest captured window.

Here's what I see when I do that: image

One note is that for ekg-show-notes-with-all-tags this takes a list of tags and show you notes that have all of them. It does not show notes with all the tags available (I see now that this might be a bit confusing, but the docstring, README and doc should hopefully explain it).

nameiwillforget commented 8 months ago

Alright, I tried that, but it still doesn't work again. I noticed a warning printed though:

⛔ Warning (emacs): EKG: No field could be parsed from metadata line at point 11
⛔ Warning (emacs): EKG: No function found for field tag

When I now capture a test note and have its tag open, it appears without even needing to be refreshed:

test

But when I refresh the tag, it disappears:

testafter

ahyatt commented 8 months ago

Hm, this is really weird. At what point in my above steps do you get that warning message?

About your test tag not showing a note, I'm not sure why, this isn't an error I've seen before. If this is a test data and doesn't have anything personal, feel free to attach the database if possible, so I can have a look.

nameiwillforget commented 8 months ago

Hm, this is really weird. I don't understand it either. I've just tried it on my desktop, where ekg was newly installed, and where I didn't use the develop version, and the result is the same (note that my desktop has almost exactly the same system and emacs config as my laptop). But it had worked already! I questioned whether I was doing something really basic wrong, but I can't see what it could be.

At what point in my above steps do you get that warning message?

I don't know when the error message appeared, I don't think it popped up, I just found it in a tab at some point.

About your test tag not showing a note, I'm not sure why, this isn't an error I've seen before. If this is a test data and doesn't have anything personal, feel free to attach the database if possible, so I can have a look.

Here is the database:

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

On Thursday, February 29th, 2024 at 4:07 AM, Andrew Hyatt @.***> wrote:

Hm, this is really weird. At what point in my above steps do you get that warning message?

About your test tag not showing a note, I'm not sure why, this isn't an error I've seen before. If this is a test data and doesn't have anything personal, feel free to attach the database if possible, so I can have a look.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

ahyatt commented 8 months ago

Thank you, but I don't think your attachment came through, can you add it directly to the page, not via email? It may be more reliable that way.

nameiwillforget commented 8 months ago

I tried but gh said the file type wasn't supported. But maybe I can trick it by removing the suffix of the file name. Alright, it doesn't complain if I call it .txt. test-ekg.txt

ahyatt commented 8 months ago

Thanks! I was able to open your database without issues. image

Since these problems seem weird and unique to your experience, I'm guessing there might be something in your config that is causing the issue. Can you try this with emacs -Q, loading only the triples and ekg modules? I think just opening your database should be sufficient.

Other possibilities: your emacs may be too new (I'm still using 29.1), and perhaps there's something in the newer emacs that is causing sqlite incompatibilities. But as a first step, please try it with a clean config.

nameiwillforget commented 8 months ago

"Andrew Hyatt" @.***> writes:

Thanks! I was able to open your database without issues. image.png (view on web)

Since these problems seem weird and unique to your experience, I'm guessing there might be something in your config that is causing the issue. Can you try this with emacs -Q, loading only the triples and ekg modules? I think just opening your database should be sufficient.

Other possibilities: your emacs may be too new (I'm still using 29.1), and perhaps there's something in the newer emacs that is causing sqlite incompatibilities. But as a first step, please try it with a clean config.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

If started this way, it works! So I guess now the question is what exactly is causing it not to work if my config is loaded. Is there any convenient way to do that?

nameiwillforget commented 8 months ago

"Andrew Hyatt" @.***> writes:

Thanks! I was able to open your database without issues. image.png (view on web)

Since these problems seem weird and unique to your experience, I'm guessing there might be something in your config that is causing the issue. Can you try this with emacs -Q, loading only the triples and ekg modules? I think just opening your database should be sufficient.

Other possibilities: your emacs may be too new (I'm still using 29.1), and perhaps there's something in the newer emacs that is causing sqlite incompatibilities. But as a first step, please try it with a clean config.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

I got a bit closer: the bug appears when I leave this code snippet uncommented in my config:

(use-package forge
   :after magit)

That's a bit unexpected to me. What can forge have to do with ekg? It seems at least that the other stuff in my config is unrelated.

nameiwillforget commented 8 months ago

Getting back to the original problem of org-roam import, it seems to work now if I use the develop version of ekg-org-roam. However, it only imports the notes in my org-roam base directory. I have a system where I'm using org-roam subdirectories as described here. I can import those notes by running ekg-org-roam-import in those subdirectories, but it would be better if it would do that automatically. Before the other bug, I wanted to try to write this myself, and maybe I'll do that, but right now I'm just glad ekg is functioning at all.

ahyatt commented 8 months ago

Great find about the forge package. I looked through it, and I'm guessing there may be some issue due to this package's use of emacsql at the same time you are using ekg through sqlite. I don't know how, but that's my guess at the moment.

About ekg-org-roam-import, the page you linked to describes directory-local variables that allow multiple different directories depending on where you are. The problem is, ekg doesn't have access to all those directory-local variables. We could add something, but anything we add would require you to essentially just give us a list of every directory again, and keep it up to date with all these other directories.

In the meantime, you can do something like this:

(cl-loop for org-roam-directory in '("~/org-roam-directory1" "~/org-roam-directory2" "~/src/project/roam-directory")
  (ekg-org-roam-import))
nameiwillforget commented 8 months ago

Thanks! Since the import seems to work I guess we can close this for now.

nameiwillforget commented 7 months ago

Ah, sorry, I was so happy that I didn't notice that the import from org-roam still doesn't include tags from the imported node. For instance, this file:

20240307194111-test3.txt

(With .txt replaced by .org) still doesn't assign the tag "test" to the note.

I also rewrote the import function to consider folders in org-roam (not several org-roam directories but subfolders of one directory). This function adds the relative path of a node from the org-roam base directory as a prefix to a node and all directories in that path as tags to the node. Might be worth taking into the upstream:

ekg-import-subdirectories.txt

(again replace .txt with .el).

ahyatt commented 7 months ago

Yes, sorry about that - my fix was for logseq, not org-roam. org-roam is something where people may have different ideas on how it should be imported. That said, it seems reasonable to add all the tags in a note, so I've now done that (in the develop branch). Perhaps there was some reason I didn't do it before, but if so, I've forgotten why. Thanks for reporting this.