Packages with subdirectories and non-elisp files are not handled properly

alem0lars commented 9 years ago

If a specifies in its recipe the files to be installed, some of them seems to be ignored. For example the SLIME package from melpa specifies:

(slime :repo "slime/slime"
       :fetcher github
       :files ("*.el"
               ("lib" "lib/hyperspec.el")
               "swank"
               "*.lisp"
               "*.asd"
               ("contrib" "contrib/*" (:exclude "contrib/test"))
               "doc/slime.texi"
               "doc/slime.info"
               "doc/dir"
               "ChangeLog"))

Which lists the files that should be included.

However when I install the app-emacs/slime package I only get these files:

slime-autoloads.el
slime.el
slime.elc
slime-tests.el
slime-tests.elc

In fact, starting emacs with slime now gives me the error: Could not open load file: lib/hyperspec.

jauhien commented 9 years ago

Yes, this is a big issue. gs-elpa does not handle additional files at the moment. Do you know, how does the package-el itself handle this kind of things? If so, could you provide me links to the relevant information? If no, I will do a research and fix it, but when I will have enough free time (and this issue is in my todo for quite a long time already).

jauhien commented 9 years ago

@alem0lars: which version of SLIME do you need? Try 2.0_p20130930 or even 9999 from the lisp overlay.

jauhien commented 9 years ago

@alem0lars: see also https://bugs.gentoo.org/show_bug.cgi?id=530148 for SLIME version bump.

alem0lars commented 9 years ago

I couldn't find a reference to the :files entry in package.el.

Tell me if I'm wrong, but that information is in the recipe not in the repository that hosts the package itself. This is because it's not a package specific thing but a package archiver specific thing. It just tells to MELPA what to keep when generating the package from the sources.

So it shouldn't have nothing to do with the package manager too, which takes an already packaged thing and loads it.

So, let's sum up:

The informations that we aren't considering aren't package specific
They are defined inside the recipes in the MELPA repository, e.g. https://github.com/milkypostman/melpa/blob/master/recipes/slime
We can and should consider the MELPA way (not package.el)

Considering only the :files thing for a moment (in my opinion it's the most urgent requirement, because it prevents packages from working).

In the MELPA source code, one interesting file is package-build.el:

:files is read using: (plist-get config :files).
I think this is the function where builds the paths:

(defun pb/expand-source-file-list (dir config)
  "Shorthand way to expand paths in DIR for source files listed in CONFIG."
  (mapcar 'car (package-build-expand-file-specs dir (pb/config-file-list config))))

It clones the repository (according to :fetcher) For example, the github fetcher is implemented with the following functions:

(defun pb/checkout-github (name config dir)
  "Check package NAME with config CONFIG out of github into DIR."
  (let* ((url (format "git://github.com/%s.git" (plist-get config :repo))))
    (pb/checkout-git name (plist-put (copy-sequence config) :url url) dir)))

(defun pb/checkout-git (name config dir)
  ; .... omitted code ...
  (pb/run-process nil "git" "clone" repo dir)))
  ; .... omitted code ...
  (apply 'pb/run-process dir "git" "log" "--first-parent" "-n1" "--pretty=format:'\%ci'"
               (pb/expand-source-file-list dir config))

What if we just clone the entire repository as it does?

jauhien commented 9 years ago

MELPA is used by package.el package manager then. And it somehow handles these issues. Or am I wrong?

The other way to proceed: we can create other ebuild generator, gs-melpa or whatever that will take the recipes, not the ELPA format files. Or add new kind of repos to gs-elpa. This possibility seems interesting for me. The issue here is that usage of recipes can significantly slow down ebuilds generation, but it can be handled by server-side automatic generation of ebuilds or some information for them. Some kind of MELPA, but for ebuilds. The similar way to roverlay or gs-pypi.

Note, that at the moment gs-elpa works with the info provided by MELPA format files, so it has no access to recipes.

in my opinion it's the most urgent requirement, because it prevents packages from working

The majority of packages work, as they are pure elisp packages. At least, all that I needed so far worked without any problems. )

jauhien commented 9 years ago

Ok, it seems package.el just copies all the additional files, so it should be quite easy to fix. I have just tried to install slime with it. I will read package.el code and then do the relevant changes in gs-elpa. May be on this weekend if I have enough free time.

alem0lars commented 9 years ago

Ok, thanks :)

jauhien commented 9 years ago

@alem0lars: please, check if my last commit to gs-elpa fixes the problem. To do so, unmask live gs-elpa, emerge it and resync the repo:

sudo flaggie app-portage/gs-elpa '+**'
sudo emerge -va -1 app-portage/gs-elpa
sudo layman -s melpa-stable

After it, please, try to emerge SLIME or whatever package with multiple files you need from melpa-stable (or change melpa-stable to melpa if you are using it).

jauhien commented 9 years ago

It seems I am doing the same as package.el now. As I compile all the elisp files in the package archive and then install all the subdirectories.

jauhien commented 9 years ago

Changes commited to the tree as 0.1.3 release. Please, test if they fix your problem.

alem0lars commented 9 years ago

I've just installed SLIME. Now the files under contrib are installed too, but swank and lib aren't copied, which cause errors.

jauhien commented 9 years ago

It copies all the files:

jauhien@zcj ~ % equery f app-emacs/slime
 * Searching for slime in app-emacs ...
 * Contents of app-emacs/slime-2.11:
/usr
/usr/share
/usr/share/emacs
/usr/share/emacs/site-lisp
/usr/share/emacs/site-lisp/site-gentoo.d
/usr/share/emacs/site-lisp/site-gentoo.d/50slime-gentoo.el
/usr/share/emacs/site-lisp/slime
/usr/share/emacs/site-lisp/slime/ChangeLog
/usr/share/emacs/site-lisp/slime/contrib
/usr/share/emacs/site-lisp/slime/contrib/ChangeLog
/usr/share/emacs/site-lisp/slime/contrib/Makefile
/usr/share/emacs/site-lisp/slime/contrib/README.md
/usr/share/emacs/site-lisp/slime/contrib/bridge.el
/usr/share/emacs/site-lisp/slime/contrib/bridge.elc
/usr/share/emacs/site-lisp/slime/contrib/inferior-slime.el
/usr/share/emacs/site-lisp/slime/contrib/inferior-slime.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-asdf.el
/usr/share/emacs/site-lisp/slime/contrib/slime-asdf.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-autodoc.el
/usr/share/emacs/site-lisp/slime/contrib/slime-autodoc.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-banner.el
/usr/share/emacs/site-lisp/slime/contrib/slime-banner.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-c-p-c.el
/usr/share/emacs/site-lisp/slime/contrib/slime-c-p-c.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-cl-indent.el
/usr/share/emacs/site-lisp/slime/contrib/slime-cl-indent.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-clipboard.el
/usr/share/emacs/site-lisp/slime/contrib/slime-clipboard.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-compiler-notes-tree.el
/usr/share/emacs/site-lisp/slime/contrib/slime-compiler-notes-tree.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-editing-commands.el
/usr/share/emacs/site-lisp/slime/contrib/slime-editing-commands.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-enclosing-context.el
/usr/share/emacs/site-lisp/slime/contrib/slime-enclosing-context.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-fancy-inspector.el
/usr/share/emacs/site-lisp/slime/contrib/slime-fancy-inspector.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-fancy-trace.el
/usr/share/emacs/site-lisp/slime/contrib/slime-fancy-trace.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-fancy.el
/usr/share/emacs/site-lisp/slime/contrib/slime-fancy.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-fontifying-fu.el
/usr/share/emacs/site-lisp/slime/contrib/slime-fontifying-fu.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-fuzzy.el
/usr/share/emacs/site-lisp/slime/contrib/slime-fuzzy.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-highlight-edits.el
/usr/share/emacs/site-lisp/slime/contrib/slime-highlight-edits.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-hyperdoc.el
/usr/share/emacs/site-lisp/slime/contrib/slime-hyperdoc.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-indentation.el
/usr/share/emacs/site-lisp/slime/contrib/slime-indentation.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-mdot-fu.el
/usr/share/emacs/site-lisp/slime/contrib/slime-mdot-fu.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-media.el
/usr/share/emacs/site-lisp/slime/contrib/slime-media.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-motd.el
/usr/share/emacs/site-lisp/slime/contrib/slime-motd.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-mrepl.el
/usr/share/emacs/site-lisp/slime/contrib/slime-mrepl.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-package-fu.el
/usr/share/emacs/site-lisp/slime/contrib/slime-package-fu.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-parse.el
/usr/share/emacs/site-lisp/slime/contrib/slime-parse.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-presentation-streams.el
/usr/share/emacs/site-lisp/slime/contrib/slime-presentation-streams.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-presentations.el
/usr/share/emacs/site-lisp/slime/contrib/slime-presentations.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-references.el
/usr/share/emacs/site-lisp/slime/contrib/slime-references.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-repl.el
/usr/share/emacs/site-lisp/slime/contrib/slime-repl.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-sbcl-exts.el
/usr/share/emacs/site-lisp/slime/contrib/slime-sbcl-exts.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-scheme.el
/usr/share/emacs/site-lisp/slime/contrib/slime-scheme.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-scratch.el
/usr/share/emacs/site-lisp/slime/contrib/slime-scratch.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-snapshot.el
/usr/share/emacs/site-lisp/slime/contrib/slime-snapshot.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-sprof.el
/usr/share/emacs/site-lisp/slime/contrib/slime-sprof.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-trace-dialog.el
/usr/share/emacs/site-lisp/slime/contrib/slime-trace-dialog.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-tramp.el
/usr/share/emacs/site-lisp/slime/contrib/slime-tramp.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-typeout-frame.el
/usr/share/emacs/site-lisp/slime/contrib/slime-typeout-frame.elc
/usr/share/emacs/site-lisp/slime/contrib/slime-xref-browser.el
/usr/share/emacs/site-lisp/slime/contrib/slime-xref-browser.elc
/usr/share/emacs/site-lisp/slime/contrib/swank-arglists.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-asdf.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-c-p-c.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-clipboard.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-fancy-inspector.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-fuzzy.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-goo.goo
/usr/share/emacs/site-lisp/slime/contrib/swank-hyperdoc.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-ikarus.ss
/usr/share/emacs/site-lisp/slime/contrib/swank-indentation.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-jolt.k
/usr/share/emacs/site-lisp/slime/contrib/swank-kawa.scm
/usr/share/emacs/site-lisp/slime/contrib/swank-larceny.scm
/usr/share/emacs/site-lisp/slime/contrib/swank-listener-hooks.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-media.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-mit-scheme.scm
/usr/share/emacs/site-lisp/slime/contrib/swank-mlworks.sml
/usr/share/emacs/site-lisp/slime/contrib/swank-motd.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-mrepl.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-package-fu.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-presentation-streams.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-presentations.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-r6rs.scm
/usr/share/emacs/site-lisp/slime/contrib/swank-repl.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-sbcl-exts.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-snapshot.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-sprof.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-trace-dialog.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank-util.lisp
/usr/share/emacs/site-lisp/slime/contrib/swank.rb
/usr/share/emacs/site-lisp/slime/dir
/usr/share/emacs/site-lisp/slime/lib
/usr/share/emacs/site-lisp/slime/lib/hyperspec.el
/usr/share/emacs/site-lisp/slime/lib/hyperspec.elc
/usr/share/emacs/site-lisp/slime/metering.lisp
/usr/share/emacs/site-lisp/slime/nregex.lisp
/usr/share/emacs/site-lisp/slime/sbcl-pprint-patch.lisp
/usr/share/emacs/site-lisp/slime/slime-autoloads.el
/usr/share/emacs/site-lisp/slime/slime-tests.el
/usr/share/emacs/site-lisp/slime/slime-tests.elc
/usr/share/emacs/site-lisp/slime/slime.el
/usr/share/emacs/site-lisp/slime/slime.elc
/usr/share/emacs/site-lisp/slime/slime.info
/usr/share/emacs/site-lisp/slime/start-swank.lisp
/usr/share/emacs/site-lisp/slime/swank
/usr/share/emacs/site-lisp/slime/swank-loader.lisp
/usr/share/emacs/site-lisp/slime/swank.asd
/usr/share/emacs/site-lisp/slime/swank.lisp
/usr/share/emacs/site-lisp/slime/swank/abcl.lisp
/usr/share/emacs/site-lisp/slime/swank/allegro.lisp
/usr/share/emacs/site-lisp/slime/swank/backend.lisp
/usr/share/emacs/site-lisp/slime/swank/ccl.lisp
/usr/share/emacs/site-lisp/slime/swank/clisp.lisp
/usr/share/emacs/site-lisp/slime/swank/cmucl.lisp
/usr/share/emacs/site-lisp/slime/swank/corman.lisp
/usr/share/emacs/site-lisp/slime/swank/ecl.lisp
/usr/share/emacs/site-lisp/slime/swank/gray.lisp
/usr/share/emacs/site-lisp/slime/swank/lispworks.lisp
/usr/share/emacs/site-lisp/slime/swank/match.lisp
/usr/share/emacs/site-lisp/slime/swank/mkcl.lisp
/usr/share/emacs/site-lisp/slime/swank/rpc.lisp
/usr/share/emacs/site-lisp/slime/swank/sbcl.lisp
/usr/share/emacs/site-lisp/slime/swank/scl.lisp
/usr/share/emacs/site-lisp/slime/swank/source-file-cache.lisp
/usr/share/emacs/site-lisp/slime/swank/source-path-parser.lisp
/usr/share/emacs/site-lisp/slime/xref.lisp

Are you using 0.1.3 version from the tree? Have you resynced your overlay with layman? Please, show me the build log.

alem0lars commented 9 years ago

Oh sorry you're right :smile: Re-synced, re-emerged and now it works fine!

Thanks a lot for the support :+1:

jauhien commented 8 years ago

Ok, seems to work now. Closing.

jauhien / gs-elpa

Packages with subdirectories and non-elisp files are not handled properly #4