Fetching dependencies from git

[danlamanna]

I have a fairly simple project that I would like to set up that requires a dependency that isn't hosted on a package manager. It seems this is becoming more common as people use straight and package-vc-install to install directly. Is it possible to declare such a dependency in eldev given that it's not on a package manager or on a local path?

[doublep]

No, it's currently not possible, but I long have thought about adding a plugin (i.e. optional extension) to Eldev that would do just this, probably using straight. If you could specify in more details what you'd need, I could have a look and maybe implement this in 1.5, the next version.

[danlamanna]

I'd like the functionality and bootstrapping that's available to eldev-use-local-dependency but with an external git repo that can be pulled from a host. Whether it uses the spec format from package-vc.el or straight.el doesn't make a big difference to me.

[doublep]

A problem here is that I have never used such functionality. I basically understand what you need and why, and I see how it would be a useful addition to Eldev too. But to design a good and usable interface I'd rather first see some real-world examples. If possible (i.e. not a private project), please provide a link to the outer project and the dependency. If not, at least show some code how you would use it, especially how you autofetch it from Git it in your normal Emacs — I assume you already do that.

[suhail-singh]

... especially how you autofetch it from Git it in your normal Emacs

FWIW, for the package doctest this is what my init.el configuration looks like in Emacs 29.1:

(use-package doctest :ensure nil
  :vc (doctest :url "https://github.com/ag91/doctest.git"
               :branch "master"
               :rev :newest))

The :ensure nil prevents use-package from trying to fetch it from the package-archives.

[doublep]

Hm, I just looked at the docstring of use-package and it doesn't mention :vc, at least in the version I have (2.4.5). Is it new? Is it built-in in use-package, or does it need something else?

[suhail-singh]

My bad! I'm using a not-yet-available-via-upstream-use-package extension called vc-use-package. Earlier in my init.el I have:

(unless (package-installed-p 'vc-use-package)
  (package-vc-install "https://github.com/slotThe/vc-use-package" :newest))

Of course, instead of first installing vc-use-package and then using that extension via the :vc keyword in use-package, you may want to directly use package-vc-install to install.

[doublep]

I see. So, basically, if Eldev was to support this, in project's file Eldev you would somehow specify where (and maybe how) to get certain dependencies. Something like:

(eldev-fetch-package-from 'abc                             ; dependency package name
                          "https://github.com/abc.el/abc"  ; the URL
                          ; possible additional options, e.g. :fetcher 'git
                          )

And then when it needs to install dependency X, it first checks if it has a URL for it. If it does, fetch (via Git or whatever) from that URL, else fall back to the usual PA lookup. (Internally this could be implemented via some existing tool or not, don't know yet.)

Is that the desired mechanism? This means you'd have to enumerate all dependencies not available in package archives (MELPA or something) together with their URL. On the other hand, there is probably no way to look up "canonical repository URL for Elisp package named X", so you do have to specify it manually...

[suhail-singh]

in project's file Eldev you would somehow specify where (and maybe how) to get certain dependencies

The simplest implementation I can imagine would be one where in the Eldev file one would note for each dependency the arguments that would be needed to install said dependency via package-vc-install. And then eldev would take care of invoking package-vc-install for each of those.

A more generalized approach would be one which parameterizes over the installation mechanism (e.g. package-vc-install vs straight.el etc). Having never used straight.el, I'm not sure what advantage having the option to use it would confer over being restricted to package-vc-install.

... you'd have to enumerate all dependencies not available in package archives (MELPA or something) together with their URL.

Yes, I believe that that's unavoidable (and that's okay).

[suhail-singh]

@doublep I encountered this issue today when trying to manage my fork of git-email via eldev (since one of the "soft" dependencies isn't on any archive).

If there were some way to specify git URLs for some of the dependencies (perhaps via a specification syntax as the one used by the :vc keyword in use-package in the upcoming Emacs 30 release), it would help matters.

[doublep]

Let's suppose your project-x declares that dependency libfoo must be fetched from certain Git repository. If libfoo itself depends on libbar, how should Eldev find that package?

Probably an option would be that libbar is to be looked up in archives that project-x itself declares. Or do you have a better idea?

[suhail-singh]

Let's suppose your project-x declares that dependency libfoo must be fetched from certain Git repository. If libfoo itself depends on libbar, how should Eldev find that package?

If libfoo doesn't use Eldev, then looking for libfoo libbar in the archives and the git URLs that project-x declares is the only sensible thing in my opinion.

If libfoo uses Eldev itself, perhaps it may help to use that dependency information. However, even in that case it would help for project-x configuration to have the ability to override things (if so desired).

Probably an option would be that libbar is to be looked up in archives that project-x itself declares.

I believe if that's the only thing that is implemented, that it would be adequate.

[d12frosted]

If libfoo itself depends on libbar, how should Eldev find that package?

I think Eldev should not treat libbar in any special way. It's just a transitive dependency. Try to fetch it based on Eldev file of the project-x and if this transient dependency can not be satisfied - let the user explicitly either add a proper archive or declare from which repository to fetch.

If libfoo uses Eldev itself, perhaps it may help to use that dependency information.

If it's just to get dependency information, I am against this idea as it would complicate things both implementation and usage-wise. Since Eldev is a build tool, the only meaningful solution here is to build differently depending on existence of Eldev file - but do it in an 'isolated' way, i.e. without polluting and messing around with project-x configurations.

I believe if that's the only thing that is implemented, that it would be

I agree, just an ability to pull dependencies from Git would be a HUGE improvement. It would unblock me from using Eldev in all scenarios that I have.

[suhail-singh]

I agree, just an ability to pull dependencies from Git would be a HUGE improvement.

Agreed. Hopefully, we won't have to wait too much longer.

@doublep please let us know in case there are additional points that you would like some discussion / clarification on.