org-ql-query got error: \"‘CATEGORY’ is a malformed function\"

[beast-pro]

org file and query:

** TODO project 1
:PROPERTIES:
:ID:       1c3f1261-e626-4ee2-b038-6ca77bf04e75
:Create:   [2023-06-19 18:05]
:zone:     west
:END:
** project 2023
:PROPERTIES:
:ID:       ae643fae-3c29-4cd9-9786-acb1ffb600eb
:Create:   [2023-06-19 18:06]
:zone:     east
:END:

(org-ql-query :select '(org-get-heading :no-tags)
              :from (current-buffer)
              :where '(or (tags "personAlice")
                          (property "zone" "east")))

Error Message: org-ql--byte-compile-warning: Invalid Org QL query: "Invalid Org QL query: \"‘CATEGORY’ is a malformed function\", :warning", :error screenshot: org-ql version:

[alphapapa]

I can't reproduce this problem. You likely have a corrupted installation of the package.

[beast-pro]

I uninstalled and reinstalled the package got same error，could this issue be because I am using emacs v29?

[alphapapa]

I don't know. It seems unlikely, but not impossible. I recommend:

1. Use https://github.com/alphapapa/with-emacs.sh to install org-ql into a clean Emacs configuration and try to reproduce the problem.
2. If it still happens, try using Emacs 28.2 to reproduce the problem.

[pcompassion]

i have this problem as well, it's hard to use edebug,

;; (setq org-use-property-inheritance '("notify"))

toggling this produces error or not.

it seems to happen near

      (items (let (orig-fns)
               (unwind-protect
                   (progn
                     (--each org-ql-predicates
                       ;; Set predicate functions.
                       (-let (((&plist :name :fn) (cdr it)))
                         ;; Save original function.
                         (push (list :name name :fn (symbol-function name)) orig-fns)
                         ;; Temporarily set new function definition.
                         (fset name fn)))
                     ;; Run query on buffers.
                     (->> buffers
                          (--map (with-current-buffer it
                                   (unless (derived-mode-p 'org-mode)
                                     (display-warning 'org-ql-select (format  "Not an Org buffer: %s" (buffer-name)) :error))
                                   (org-ql--select-cached :query query :preamble preamble :preamble-case-fold preamble-case-fold
                                                          :predicate predicate :action action :narrow narrow)))
                          (-flatten-n 1)))
                 (--each orig-fns
                   ;; Restore original function mappings.
                   (-let (((&plist :name :fn) it))
                     (fset name fn)))))))

where you apply predicates function, it is somehow treating inherited tag as function

[pcompassion]

I think I found where it comes from

so org-use-property-inheritance can be list of strings, and org-ql assumes it's t or nil.

(org-ql-defpred property (property &optional value &key inherit)
  "Return non-nil if current entry has PROPERTY, and optionally VALUE.
If INHERIT is nil, only match entries with PROPERTY set on the
entry; if t, also match entries with inheritance.  If INHERIT is
not specified, use the Boolean value of
`org-use-property-inheritance', which see (i.e. it is only
interpreted as nil or non-nil)."
  :normalizers ((`(,predicate-names)
                 ;; HACK: This clause protects against the case in
                 ;; which the arguments are nil, which would cause an
                 ;; error in `rx-to-string' in other clauses.  This
                 ;; can happen with `org-ql-completing-read',
                 ;; e.g. when the input is "property:" while the user
                 ;; is typing.
                 ;; FIXME: Instead of this being moot, make this
                 ;; predicate test for whether an entry has local
                 ;; properties when no arguments are given.
                 (list 'property ""))
                (`(,predicate-names ,property ,value . ,plist)
                 ;; Convert keyword property arguments to strings.  Non-sexp
                 ;; queries result in keyword property arguments (because to do
                 ;; otherwise would require ugly special-casing in the parsing).
                 (when (keywordp property)
                   (setf property (substring (symbol-name property) 1)))
                 (list 'property property value
                       :inherit (if (plist-member plist :inherit)
                                    (plist-get plist :inherit)
                                  org-use-property-inheritance))))

(defcustom org-use-property-inheritance nil
  "Non-nil means properties apply also for sublevels.

This setting is chiefly used during property searches.  Turning it on can
cause significant overhead when doing a search, which is why it is not
on by default.

When nil, only the properties directly given in the current entry count.
When t, every property is inherited.  The value may also be a list of
properties that should have inheritance, or a regular expression matching
properties that should be inherited.

However, note that some special properties use inheritance under special
circumstances (not in searches).  Examples are CATEGORY, ARCHIVE, COLUMNS,
and the properties ending in \"_ALL\" when they are used as descriptor
for valid values of a property.

Note for programmers:
When querying an entry with `org-entry-get', you can control if inheritance
should be used.  By default, `org-entry-get' looks only at the local
properties.  You can request inheritance by setting the inherit argument
to t (to force inheritance) or to `selective' (to respect the setting
in this variable)."
  :group 'org-properties
  :type '(choice
      (const :tag "Not" nil)
      (const :tag "Always" t)
      (repeat :tag "Specific properties" (string :tag "Property"))
      (regexp :tag "Properties matched by regexp")))

[alphapapa]

@pcompassion Thanks for digging into that. I can confirm the bug. This code seems to fix the problem. Please let me know if it does for you:

(org-ql-defpred property (property &optional value &key inherit)
  "Return non-nil if current entry has PROPERTY, and optionally VALUE.
If INHERIT is nil, only match entries with PROPERTY set on the
entry; if t, also match entries with inheritance.  If INHERIT is
not specified, use the Boolean value of
`org-use-property-inheritance', which see (i.e. it is only
interpreted as nil or non-nil)."
  :normalizers ((`(,predicate-names)
                 ;; HACK: This clause protects against the case in
                 ;; which the arguments are nil, which would cause an
                 ;; error in `rx-to-string' in other clauses.  This
                 ;; can happen with `org-ql-completing-read',
                 ;; e.g. when the input is "property:" while the user
                 ;; is typing.
                 ;; FIXME: Instead of this being moot, make this
                 ;; predicate test for whether an entry has local
                 ;; properties when no arguments are given.
                 (list 'property ""))
                (`(,predicate-names ,property ,value . ,plist)
                 ;; Convert keyword property arguments to strings.  Non-sexp
                 ;; queries result in keyword property arguments (because to do
                 ;; otherwise would require ugly special-casing in the parsing).
                 (when (keywordp property)
                   (setf property (substring (symbol-name property) 1)))
                 `(property ,property ,value
                            :inherit ,(if (plist-member plist :inherit)
                                          (plist-get plist :inherit)
                                        (pcase org-use-property-inheritance
                                          (`nil nil) (`t t)
                                          ((pred stringp) org-use-property-inheritance)
                                          ((pred listp) `(quote ,org-use-property-inheritance)))))))
  ;; MAYBE: Should case folding be disabled for properties?  What about values?
  ;; MAYBE: Support (property) without args.

  ;; NOTE: When inheritance is enabled, the preamble can't be used,
  ;; which will make the search slower.
  :preambles ((`(,predicate-names ,property ,value . ,(map :inherit))
               ;; We do NOT return nil, because the predicate still needs to be tested,
               ;; because the regexp could match a string not inside a property drawer.
               (list :regexp (unless inherit
                               (rx-to-string `(seq bol (0+ space) ":" ,property ":"
                                                   (1+ space) ,value (0+ space) eol)))
                     :query query))
              (`(,predicate-names ,property . ,(map :inherit))
               ;; We do NOT return nil, because the predicate still needs to be tested,
               ;; because the regexp could match a string not inside a property drawer.
               ;; NOTE: The preamble only matches if there appears to be a value.
               ;; A line like ":ID: " without any other text does not match.
               (list :regexp (unless inherit
                               (rx-to-string `(seq bol (0+ space) ":" ,property ":" (1+ space)
                                                   (minimal-match (1+ not-newline)) eol)))
                     :query query)))
  :body
  (pcase property
    ('nil (user-error "Property matcher requires a PROPERTY argument"))
    (_ (pcase value
         ('nil
          ;; Check that PROPERTY exists
          (org-ql--value-at
           (point) (lambda ()
                     (org-entry-get (point) property))))
         (_
          ;; Check that PROPERTY has VALUE.

          ;; TODO: Since --value-at doesn't account for inheritance,
          ;; we should generalize --tags-at to also work for property
          ;; inheritance and use it here, which should be much faster.
          (string-equal value (org-ql--value-at
                               (point) (lambda ()
                                         (org-entry-get (point) property inherit)))))))))

[pcompassion]

@alphapapa I confirm it works for my case. Thank you for library and support.

As newcomer to elisp, I felt it was way hard to debug org-ql.el because I couldn't find the definition of the macro. I found the code in edebug session. (I thought it was macro magic, thought I was seeing the expanded macro or something ..)

Now you had requested to test that code, I couldn't find the definition using my search tool (rg) and it seems ^@ at 41836 pos, is causing rg to skip org-ql.el file from searching.

just to let you know. it could help outside people to diagnose the problem.

[alphapapa]

@pcompassion Which macro did you have trouble finding the definition of? C-h f should take you to the definition of any macro or function.

[pcompassion]

@alphapapa org-ql.el was completely hidden from me because my project search use "rg" and rg ignores the org-ql.el file because it has ^@ , you can see it by doing manually rg "org-ql-defpred" org-ql.el

well c-h f could do it, but i was using my project search. under the repos directory and of course i could use c-h f to try to find definition of something sometimes.

but, at the time of debugging, i didn't know the function name, because i didn't know which function i should debug. so i started from org-ql-search , then from there I used project-search such as rg or xref

[alphapapa]

The issue is that this regular expression has a null byte: https://github.com/alphapapa/org-ql/blob/131407814ebfd8d409f23bc5cceeeb2b5da1a8d9/org-ql.el#L854 See the rg man page's documentation on the --binary option.

and of course i could use c-h f to try to find definition of something sometimes.

For libraries that are loaded into Emacs, C-h f should generally be your first move.

I still don't know which macro you were looking for the definition of, though. :)

[alphapapa]

Thanks to all for reporting and investigating this bug. I'll need to add some more tests before merging, so since this isn't a new bug, I'm retargeting for v0.8. Any users who are affected by this bug are welcome to add the code in this comment to their Emacs config temporarily: https://github.com/alphapapa/org-ql/issues/356#issuecomment-1723501188

[pcompassion]

aight, it's not so important , but not everything has to be important..

Here's what happend to me, and i thought it might happen to others, but I guess chance is slim.

Hit a bug, set a breakpoint on entry function (which was org-ql-select for me) it's dying somewhere, but can't use debug-on-error (because it was user-error) I see error mesage in message buffer, I search for the error message, using my search (rg) and on.. then I look for the reference of that error variable. and nothing comes up. strange.

only thing I could think of was , it was macro magic. Something has to be there but I couldn't find. No trail, org-ql-defred was also not searchable

I was debugging org-ql-select, and couldn't find definitions of functions that were used in org-ql-select.

I didn't know until it was because rg was not scanning the file ..

When I filed the bug, I was gonna try to fix the bug myself, but couldn't find the org-ql-defred i thought it was macro magic.. duh

When you asked me to test it, what? I couldn't find it, how am i gonna edit that function to test it. By this time, I thought maybe something was wrong with my system and I found the reason

So if someone like me who's using rg and who's not familiar with elisp, might have hard time debugging the org-ql code. anyway. Thanks for the fix

Today I had hard time, with org-ql again... where having (message ..) inside (org-ql-select :action , somehow message call is affecting cursors, I coulnd't find why.. elisp is hard..

[alphapapa]

Thanks for the explanation. The only possible misunderstanding I can see is this:

can't use debug-on-error (because it was user-error)

FYI, you can modify the variable debug-ignored-errors to enable backtraces for user-error (this is mentioned in the user-error docstring).

With regard to ripgrep: shrug It's not inconceivable that a source code file might contain a null byte, so maybe you should configure ripgrep to use --binary by default, either in its own config or in the way Emacs calls it. Or just consider trying other search tools if you can't seem to get useful results from one. To put it another way, Elisp development doesn't depend on external tools like ripgrep.

Today I had hard time, with org-ql again... where having (message ..) inside (org-ql-select :action , somehow message call is affecting cursors, I coulnd't find why..

I don't know what you mean by this. If you're asking for help, you'll need to be specific. But please do so in an appropriate place, not this bug report.