Request: create a "core" org-ql package

This directory currently includes code that depends on helm and org-super-agenda.

I would like to use org-ql as a dependency for one of my packages, but I can't make all my users download helm and all of org-super-agenda.

[edit: relevant issue on package - Trevoke/org-gtd.el/issues/104 ]

There may be a misunderstanding: org-ql does not depend on helm. That is provided by a separate package, helm-org-ql, which happens to also live in this repository.

It is true that org-ql currently depends on org-super-agenda, because that is used in org-ql-view to provide grouping support. A future version of org-ql will use taxy for grouping and may drop that dependency, making it optional.

Until then, org-super-agenda will remain a dependency of org-ql. But, not to worry, merely having it installed does no harm; users must choose to activate org-super-agenda-mode and configure it before it has any effect. And its functionality remains available to org-ql-view even when the mode is disabled.

Ah! My apologies, I don't know what melpa recipe I looked at, but indeed, the melpa recipe for org-ql does not include helm-org-ql. So let's consider that chunk of the conversation done, thanks for the correction.

Re: org-super-agenda, I'm not suggesting the package being installed would be harmful, just that ... From a dependency resolution perspective, this may lead to odd update behaviors, because org-super-agenda has its own reasons for changing beyond org-ql. So, maybe this is really a conversation about how org-super-agenda started off having one purpose, but it turns out it actually is more useful generically than just as an org-agenda supercharger, so its name is technically obsolete?

From a dependency resolution perspective, this may lead to odd update behaviors, because org-super-agenda has its own reasons for changing beyond org-ql.

Not to worry, since I'm the author of both packages, I've no intentions of letting any "odd update behaviors" develop between them. ;)

So, maybe this is really a conversation about how org-super-agenda started off having one purpose, but it turns out it actually is more useful generically than just as an org-agenda supercharger, so its name is technically obsolete?

I don't think so. I've no intention of org-super-agenda taking on a different purpose. There may be a misunderstanding here: org-ql-view imitates org-agenda buffers so that things that work with org-agenda buffers (like org-super-agenda, as well as built-in tools) can work with org-ql-view buffers as well. To some extent, org-ql-view buffers are also org-agenda buffers (though not all features are implemented).

The more generically useful library is taxy, which is based on my experience writing org-super-agenda, bufler, etc. As I said, I have a WIP branch of org-ql that uses taxy-magit-section to implement grouping for org-ql-view buffers; if the user so chooses, that can replace the use of org-super-agenda for grouping org-ql-view buffers.

But I don't intend for org-super-agenda to go anywhere; it remains useful for grouping org-agenda buffers. (Perhaps someday, far into the future, it might be refactored to be based on taxy, but we'll see.)

Having said all of that: strictly speaking, org-ql need not depend on org-super-agenda. If the user doesn't make use of :super-groups grouping, it's not used. But providing support for optional dependencies in Emacs packages is very awkward, usually leading to errors, byte-compiler warnings, installation problems, complaints from users who install packages in unorthodox ways, etc. In my judgment, given how useful the two packages are together, and the problems that would result from trying to remove the required dependency, it wouldn't be worth the trouble. And as long as org-super-agenda behaves itself (i.e. it does nothing unless so-configured)--and as long as I am its maintainer, it will--there's no reason to not have it installed, just like any number of other Emacs libraries that are installed but dormant, both built-in and third-party.

Does that explanation help? Let me know if it doesn't answer your concerns. :)

It makes sense, thank you for taking the time to explain all this. Based on what you said, my thoughts now take the direction of suggesting that org-ql-view be a separate tool as well (and maybe org-ql-view should be named org-ql, and the base logic should be org-ql-core.

I think you are likely correct in saying that in standard pragmatic terms, "it should be fine".

My predicament, I suppose, is that if org-gtd depends on org-ql then it also depends on org-super-agenda, and this means that there's two reasons for my dependencies to be updated - and since technically they are both end-user-facing packages, I would be violating a principle of depending on code that changes less often than my own. So by asking to put the "core org-ql" logic in a separate package, I'm really asking if there's a way to create a kernel of logic that is likely to change less often than code that depends on it.

What do you think?

I don't think there would be any benefit in putting org-ql-view in a separate package. Both libraries are in relatively early stages of development, and changes in one may mean changes in another. Splitting them into separate packages would mean extra work and complexity for no benefit.

The base logic is already in org-ql. I don't think there would be any benefit in renaming it to org-ql-core.

org-super-agenda is already at 1.0+ status, meaning any changes in 1.x versions are intended to be backward compatible. Any 2.x release would be merely hypothetical at this time and years away, if at all.

org-ql is at pre-1.0 status, meaning incompatible changes could be made. However, I do not make such changes lightly, given its user base, and I attempt to provide compatibility between 0.x releases.

a principle of depending on code that changes less often than my own

That's a great ideal to aspire to, but I don't think that can ever be guaranteed. I think the best you can do is to look at a project's history and future plans, and to talk with its developers about their intentions. As I hope I've made clear, I intend for org-ql and org-super-agenda to be useful foundations to build applications on, so I wouldn't be shaking those foundations without good reason, and not often--I know the pain of updating downstream libraries that depend on upstream changes which were made needlessly.

So, while it would be ideal to have org-super-agenda not be a dependency of org-ql, and that will hopefully be the case in the future, as of now it's a low-priority goal. But I don't think you need to worry about using org-ql in your project in the meantime.

Oops, I never wrote my reply to this!

As long as you consider org-ql and org-ql-view to be in early stages of development then you're right, it would be too much of a pain to split them, because you know neither what changes need to be made easy and which ones can be kept underneath an API . This surprises me a bit because I thought only fairly stable packages made it all the way to MELPA (my packages certainly went under rather intense scrutiny when I submitted them). Still - no matter, I think, for this particular conversation.

Re: the great ideal to aspire to, of course it can never be guaranteed, but that's not a valid argument, because that's literally the only way for everything to work. We depend on programming languages to have a stable syntax so we can write code that works across versions. We depend on that syntax to write code that highlights/indents properly in editors. We depend on org-mode (and dash, and f, and s, and transient) to have a stable API so we can write code that depends on it. We depend on emacs to have stable libraries so we can write code that even survives across emacs versions. We rely on very foundational protocols like TCP/IP (remember when VoIP came out, never mind IPv6 ?), DNS, SMTP, to build systems that can interact. Heck, I don't know how much of the web is using HTTP 2.x yet, I expect a very high percentage is still on HTTP 1.1..

Anyway - "talk to the dev about their intentions" -- The "kik" package that broke the entire modern Javascript world because the dev removed "leftpad" is a hilariously extreme example of what can go wrong (and in that case, talking to the dev wouldn't have helped anyway, since they got pissed at npm itself), but ... My current main package relies incredibly heavily on org-mode and I think I'd be going nuts if I had to talk to org-mode about what direction they want to take org-capture, org-refile, and org-agenda, in order to know whether I can build something on top of it, and if I had to do this with every package I need to depend on, I would just copy the code into my own project in order to save myself hours of pain. This being said, does this mean I ignore the CHANGELOGs for the new versions of the packages I depend on? No. I read those. I do have a dependency on them and I try to be a responsible member of the Free Software world and a responsible author of a package that, uh, I dunno, at least two or three people use (I honestly don't know how many people do use my package, but it violates certain emacs philosophies because of how opinionated it currently is, so I suspect most emacs hackers look at it and are disgusted by it, which is .. a work in progress).

Anyway -- all of this being said, about the highly abstract principles of code organization and dependency management, it sounds like you're serious about treating org-ql, albeit in a pre-1.0 status, as though it is 1.0 already, and I appreciate that.

It occurs to me that I have never actually told you (in re-reading this thread) why I'm being such a pain here. Sorry, if I'd written that, I think it would have made more sense to you.

What I actually intend to do is surface the org-ql syntax to my end users to help them create and compose their own custom GTD views - because the org-ql lisp syntax is, I think, easier to understand for non-coders (and my users are often non-coders) than the string match syntax provided by org-mode -- and it is CERTAINLY more regular. So I am very much trying to make sure there's not going to be changes that surface to my end users. And you have certainly gone through enough pain telling me you're going to avoid those changes as much as you can, so.. Thanks.

I'll start looking into how I can make org-ql work. Thanks.

As long as you consider org-ql and org-ql-view to be in early stages of development then you're right, it would be too much of a pain to split them, because you know neither what changes need to be made easy and which ones can be kept underneath an API.

I plan for these packages to be long-lived, and I have long-term plans that are far from complete, so in that regard, yes, they're at an early stage. That doesn't mean that they are "unstable" in terms of bugs or API changes (though certainly not bug-free, either).

This surprises me a bit because I thought only fairly stable packages made it all the way to MELPA (my packages certainly went under rather intense scrutiny when I submitted them). Still - no matter, I think, for this particular conversation.

MELPA has over 5,000 packages, and they are not continually reviewed--only once, upon submission. There are no policies enforced regarding how diligently an author develops a package afterward. It's up to each author to use his best judgment.

That also begs the question of what stability is: lack of bugs, or unchanging API, or...? And again, development methodology is up to each author.

Re: the great ideal to aspire to, of course it can never be guaranteed, but that's not a valid argument, because that's literally the only way for everything to work. We depend on programming languages to have a stable syntax so we can write code that works across versions. We depend on that syntax to write code that highlights/indents properly in editors. We depend on org-mode (and dash, and f, and s, and transient) to have a stable API so we can write code that depends on it. We depend on emacs to have stable libraries so we can write code that even survives across emacs versions. We rely on very foundational protocols like TCP/IP (remember when VoIP came out, never mind IPv6 ?), DNS, SMTP, to build systems that can interact. Heck, I don't know how much of the web is using HTTP 2.x yet, I expect a very high percentage is still on HTTP 1.1..

Well, I was responding to this that you said: "a principle of depending on code that changes less often than my own." We're not talking about IETF protocols and ANSI language specifications here. We're talking about an informal hierarchy of Emacs Lisp packages that have evolved over many years and continue to do so as people work on them in their spare time. Some of them are developed purposefully to enable other packages to depend on them, while others may grow into that intention over time, while others are simply used as other authors see fit, without any regard for the dependency's author's intentions.

Anyway - "talk to the dev about their intentions" -- The "kik" package that broke the entire modern Javascript world because the dev removed "leftpad" is a hilariously extreme example of what can go wrong (and in that case, talking to the dev wouldn't have helped anyway, since they got pissed at npm itself),

You're preaching to the choir here. :)

but ... My current main package relies incredibly heavily on org-mode and I think I'd be going nuts if I had to talk to org-mode about what direction they want to take org-capture, org-refile, and org-agenda, in order to know whether I can build something on top of it

As would I. Yet Org offers no guarantees about those internal libraries and their APIs, and so I regularly have to deal with changes to them in my projects, like this one. I have at times asked them to be more conservative and slow in changes to them, but the Org developers aren't usually receptive to that request; while they are aware of the "ecosystem" of Org-related packages on MELPA and elsewhere, they have little interest in officially supporting the Org APIs outside of Org itself. Instead, they prefer to be free to do what's best for Org itself. This is not to say that they disregard the downstream effects of their changes--they're usually well-documented--but they don't usually let those effects stop them from making such changes.

Anyway, I think you'll find that my Org-related packages' APIs tend to be more stable than those of Org itself. (Of course, the size of my packages and APIs are dwarfed by Org itself, so it's apples-to-oranges.)

But in general, if you're going to be writing and publishing Org-related packages, occasional breakage due to upstream changes is inevitable.

Anyway -- all of this being said, about the highly abstract principles of code organization and dependency management, it sounds like you're serious about treating org-ql, albeit in a pre-1.0 status, as though it is 1.0 already, and I appreciate that.

What's in a version number? Would 1.0 by any other name be as stable? Or something like that. Many software projects are never called "1.0", for various reasons, yet they end up being "stable" and widely used. There is no universal way of numbering versions, SemVer notwithstanding.

It occurs to me that I have never actually told you (in re-reading this thread) why I'm being such a pain here. Sorry, if I'd written that, I think it would have made more sense to you.

Introductions usually do help, yes. :)

What I actually intend to do is surface the org-ql syntax to my end users to help them create and compose their own custom GTD views - because the org-ql lisp syntax is, I think, easier to understand for non-coders (and my users are often non-coders) than the string match syntax provided by org-mode -- and it is CERTAINLY more regular. So I am very much trying to make sure there's not going to be changes that surface to my end users. And you have certainly gone through enough pain telling me you're going to avoid those changes as much as you can, so.. Thanks.

Yes, I don't plan to make any fundamental changes to org-ql's query syntax. I use org-ql myself, and it's hard enough to find time to make changes to it, let alone deal with downstream effects in my own Emacs config that breaking changes to org-ql would require. I'm not just a developer--I'm also a user. :)

Regarding your GTD views: it might be good to discuss your ideas in more detail. I use org-ql for GTD-like systems already (e.g. project-specific views to help me plan projects, next actions, avoid stuck/neglected projects, etc), so I'm curious what gaps you have found and plan to fill with your own project.

If you would, please open a new issue to discuss those ideas (it's too easy for me to overlook notifications on closed issues).

alphapapa / org-ql

Request: create a "core" org-ql package #327