Allow to extend load path to use local packages when compiling

[publicimageltd]

I would like to compile a file which requires packages which are stored in some local directory, w/o using any package manager. (My own local packages, simply). In my init.el, I add the directory to the load path (recursively). Is there any way I can tell makem.sh to also add a specific directory to the load path before compiling the local file? Looking at the code, I see that it finds all dependencies as they are declared via Package-Requires, so extending the load path would handle these local packages well, I think.

[alphapapa]

I guess another option could be added to extend the load-path. I'm not sure if I want to add that additional complexity. Your use case seems rather unorthodox to me, and this script is intended to cover the most common use cases.

A workaround you could use would be to use the sandbox options to set up a sandbox Emacs config and install your special dependencies into it, then use that sandbox directory with this script. Once you set it up, it should continue working.

[publicimageltd]

Thanks for replying. I might try the sandbox option, I'll have to look at how it works.

Interesting, however, that the case of 'local' dependencies seems to be uncommon. I have some dozen local packages in my load path, this is my way of keeping init.el clean and clear. They are not so well crafted that I would publish them on melpa, but they wrap some useful functionalities for different use cases.

I actually wanted to use makem.sh to install a two-level process of working on these files: Edit them in a git directory, with tests and all, then 'publish' the newer version (compiled .elc files) in the local load path. And once the package seems to be useful enough for other users, I might even publish it on github or melpa.

So I'm interested to hear how you, or other, handle this. What is the common practice of developing this kind of 'local' additions, then?

[alphapapa]

Thanks for replying. I might try the sandbox option, I'll have to look at how it works.

The sandbox is a very important feature, because it helps ensure that your package works in a clean configuration, unpolluted by your personal config. If you don't use it to test your package, it's almost inevitable that you will end up getting bug reports that are caused by your only testing your package in your personal config.

Interesting, however, that the case of 'local' dependencies seems to be uncommon. I have some dozen local packages in my load path, this is my way of keeping init.el clean and clear. They are not so well crafted that I would publish them on melpa, but they wrap some useful functionalities for different use cases.

So I'm interested to hear how you, or other, handle this. What is the common practice of developing this kind of 'local' additions, then?

I'm not clear on what you mean by "local packages" or "local additions." As I said, the best way to handle this is to use a sandbox config and install the dependency packages into it. That does not require publishing them on MELPA, but if you intend for the tested package to be installed on MELPA, they should be, otherwise users will have to install them manually.

I actually wanted to use makem.sh to install a two-level process of working on these files: Edit them in a git directory, with tests and all, then 'publish' the newer version (compiled .elc files) in the local load path. And once the package seems to be useful enough for other users, I might even publish it on github or melpa.

That's not generally how Emacs packages are done. It's not like C programs that require compiled binaries to be distributed.

It sounds like you're essentially reimplementing what package-install does. That's not necessary, and it's making your project more complicated.

I recommend simply using the built-in package management tools in Emacs. As well, Quelpa makes it easy to install packages from source code repos: it's a layer on top of package-install, so it handles byte-compilation automatically. As well, this tool makes it easy to upgrade a package that's been installed with Quelpa: https://github.com/alphapapa/unpackaged.el#upgrade-a-quelpa-use-package-forms-package

So I think the bottom line is that you're using a workflow that isn't necessary or optimal for working on and installing Emacs packages. If you use a more standard workflow, these issues will disappear.

[publicimageltd]

Thanks for this long and helpful answer! The workflow it is, indeed. I looked at quelpa, and it seems to be what I want. I now install my own 'local packages' with a file fetcher recipe. Thus, I can develop the 'local package' in a separate directory and upgrade it explicitly with quelpa-upgrade once a new feature is implemented. The only thing missing is to be able to fetch the package from a local git repository (and ideally even from a certain branch). It looks like the git fetcher presupposes the repository to be hosted on some server, or could I just use a path-to-git-repo? I will check it out.

If you have any ideas on the git fetcher (since you also contribute to quelpa), I'd love some hints. But I'll close the issue, thanks again!

[alphapapa]

It works for local repos as well. IIRC I merged a patch related to that effect myself.

[publicimageltd]

Great!!

But what's the recipe format? I tried the following and it did not work:

(quelpa '(packagename :fetcher git  :url "path/to/directory")

I also tried :path instead of :url.

[akirak]

@publicimageltd See this document: https://github.com/melpa/melpa/#recipe-format

[publicimageltd]

That's exactly what I did. Giving it a second try today, I finally found out that the problem was the URL: the file path seems to be added onto user-emacs-directory, so this part has to be left out in the declaration. I strongly recommend (a) to document the possibility to also use URLs with the file:// protocol; and that this prefix has to be added to the path; it was not obvious at all to me; and (b) to document that the file path will be expanded automatically. I only tried it again and again because you guys were acting as if I'd been missing the obvious. :-)

Thanks anyways for your comments. The possibility to clone local git repositories is something I'd been looking for.

[contrapunctus-1]

I'm running into the same issue - I have a local package, which I haven't yet submitted to MELPA, as a dependency. I tried copying the local package directory to .sandbox/27.1/elpa/, and also added (add-to-list 'load-path "/path/to/foo/") (require 'foo) to the sandbox init.el. But when I run ./makem.sh -s.sandbox -C test I still get Cannot open load file: No such file or directory, foo

I haven't tried Quelpa yet.

[akirak]

@contrapunctus-1 You'll have to patch package-lint to work around the issue. It's not officially supported yet: https://github.com/purcell/package-lint/issues/172