Transclude content over HTTP

[josephmturner]

This PR adds the following features:

;; This file contains functionality related to transcluding content over HTTP
;; using plz.el.  Features include:
;;
;; - Transclude plain text
;;   + Transclude only Org headings matching search options
;; - Transclude HTML converted to Org using Pandoc, a lá `org-web-tools'
;;   + Transclude only HTML headings matching link anchor
;; - TODO: Support :lines

This PR uses @alphapapa's HTTP library plz.el, which is on GNU ELPA, and therefore considered part of Emacs. I'm open to considering core HTTP libraries if adding another dependency is a hard no.

This PR also adds a hack to guess what sections of text in an HTML file should be considered part of the link target. Please see test/org-transclusion-http.org for example usage.

[nobiot]

@josephmturner , Thank you for the PR. Sorry that I don't have a headspace to focus on reviewing code at the moment (a bit too tied up at the other parts of my life). Just one thing regarding the HTTP libraries part.

I'm open to considering core HTTP libraries if adding another dependency is a hard no.

My gut feel is that I would prefer not to adding another dependency. It is not a hard no, but I have been avoiding additional dependencies. I'd prefer to continue with this. For the HTTP, to me, it would depend on two conditions:

1. (Cost) How time-consuming for you to do this
2. (Benefit) How (un)legible the code will be if we avoid plz

Do you have any comments on these two? Or should we consider other factors?

[alphapapa]

@nobiot If I may, since I've been working closely with Joseph on these projects: I would not recommend trying to use the built-in url library for this functionality. It would likely lead to various problems which plz has been developed to avoid, due to various issues known to exist in url (ones which are hard to describe and harder to solve). I developed various workarounds for them myself before developing plz, which I used in some of my packages, but none of them were complete solutions. In the end, I developed plz because it was necessary. plz is now a mature, reliable library which I can recommend without reservation.

I do empathize with wanting to avoid adding dependencies, in general. So maybe the network functionality could be in a separate package which could optionally integrate with org-transclusion when installed.

[nobiot]

@alphapapa Thank you for the detail. I can see that my aversion to dependencies may be rather a psychological thing with no real foundation -- what would be the reasons why users and developers should avoid many dependencies? I actually do not know for certain.

For developers, I think what you say about the built-in url library makes sense and we should include plz.

For users...?

Here is my line of thinking... Not sure if it goes anywhere, but anyway:

(1) If transclusion via http is attractive as a feature, then it makes sense to include plz so that users do not have to think about additional installation of a separate package.

(2) If transclusion via http is a marginal feature that only a fraction of transclusion users care, then it makes sense to have it a separate package.

I am leaning toward (1) above from my gut feeling.

I personally cannot think of a use case at the moment, but transclusion has been used by so many people with interests and workflows that do not overlap with mine at all (I think it's a great thing :smile: ).

I wonder if some others could jump into this conversation and add different perspectives... Not sure if Reddit is a good forum for such a question...

[alphapapa]

Avoiding dependencies is generally a good idea for any software, because it makes it more complicated, and it means adding code that is outside your control. But every situation is unique; in this one, everything involved is Free Software, the dependency is "part of Emacs," and it's reached a state of maturity. So it's a relatively safe dependency to add (speaking as impartially as I can). :)

If you decide to add support for this network stuff in the same package, I'd probably still recommend putting it in a separate library within the package, if it's feasible, and enabling it with an option in the main library. That way users could keep the network functionality disabled and unloaded. (Consider the recent release of Emacs 29.3 due to an issue which could have allowed arbitrary code execution upon loading a file with certain content; similar concerns generally apply to network-related code.)

[josephmturner]

If you decide to add support for this network stuff in the same package, I'd probably still recommend putting it in a separate library within the package, if it's feasible, and enabling it with an option in the main library. That way users could keep the network functionality disabled and unloaded.

@alphapapa With the current patch, the network-related code is in org-transclusion-http.el, and users need to manually add org-transclusion-http to org-transclusion-extensions to enable it. By default, installing and requiring org-transclusion does not automatically evaluate any network-related code.

[josephmturner]

I am in favor of putting the network logic in a separate package. I'll be making progress on that in org-transclusion-http.

In the meantime, please see #229, which I believe supercedes this PR. Otherwise, please feel free to re-open this issue.

[nobiot]

@alphapapa Thank you for the advice for my general question. I think we now have a path forward. Much appreciated!