Including additional components from a package

[bmaia]

Operating system, Python and Exodus version: Ubuntu 17.04, Python 3.5.3, Exodus 1.1.8 Target OS: CentOS 7.4

Some packages like nmap, for example, include additional components inside the /usr/share/[package] folder. Those components (static files, scripts etc) are bundled with the .deb/.rpm, but they're not listed by ldd. One easy way to list those dependencies is using rpm -ql nmap (RedHat/CentOS/Fedora) or dpkg-query -L nmap (Debian/Ubuntu):

$ dpkg-query -L nmap
/.
/usr
/usr/bin
/usr/bin/ncat
/usr/bin/nmap
(...)
/usr/share/nmap
/usr/share/nmap/nmap-mac-prefixes
/usr/share/nmap/nmap-os-db
/usr/share/nmap/nmap-payloads
(...)

Here's what you have to do manually now:

bernardomr@ubuntu:~$ exodus nmap | ssh bernardomr@172.16.136.133
Pseudo-terminal will not be allocated because stdin is not a terminal.
bernardomr@172.16.136.133's password: 
Installing executable bundle in "/home/bernardomr/.exodus"...
Successfully installed, be sure to add /home/bernardomr/.exodus/bin to your $PATH.

Trying to run nmap with scripts:

[bernardomr@centos ~]$ nmap -sV localhost

Starting Nmap 7.40 ( https://nmap.org ) at 2018-02-06 05:12 EST
Unable to find nmap-services!  Resorting to /etc/services
NSE: failed to initialize the script engine:
could not locate nse_main.lua
stack traceback:
    [C]: in ?

QUITTING!

strace output:

[bernardomr@centos ~]$ cat strace.log | grep \.nse
read(4, "erver\nserverview-as   3169/tcp  "..., 4096) = 4096
stat("/home/bernardomr/.nmap/updates/7.30/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/home/bernardomr/.exodus/lib/updates/7.30/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/home/bernardomr/.exodus/lib/../share/nmap/updates/7.30/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/usr/share/nmap/updates/7.30/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/home/bernardomr/.nmap/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/home/bernardomr/.exodus/lib/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/home/bernardomr/.exodus/lib/../share/nmap/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
stat("/usr/share/nmap/nse_main.lua", 0x7fffffff9cd0) = -1 ENOENT (No such file or directory)
could not locate nse_main.lua

We need to manually copy the /usr/share/nmap folder to ~/.exodus/share/nmap]:

bernardomr@ubuntu:~$ cd /usr/share/ ; tar cpf - nmap | ssh bernardomr@172.16.136.133 "mkdir ~/.exodus/share ; tar xpf - -C ~/.exodus/share"

And now we can properly run the nmap scan with scripts:

[bernardomr@centos ~]$ nmap -sV localhost

Starting Nmap 7.40 ( https://nmap.org ) at 2018-02-06 05:23 EST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00013s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 998 closed ports
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 7.4 (protocol 2.0)
25/tcp open  smtp    Postfix smtpd
Service Info: Host:  localhost.localdomain

I wonder if it would be useful to add a command line switch to bundle additional folders with those components automatically. I'm not sure if there's any better way to list the dependencies other than using rpm and dpkg (maybe disassembling the binary?), but it would be interesting to run exodus like this:

$ exodus --tarball nmap --additional-files /usr/share/nmap --output nmap.tgz

[sangaline]

Thanks for the clear use case. I had already been planning on adding an --include flag for additional files and dlopened libraries, but I really like the idea of optionally using the package listing to specify all of the direct dependencies for an application. Currently, the format of the installation script changes when the output file is - or when stdout isn't a tty (so that it can be streamed over ssh). Something similar could potentially be done here for stdin. Maybe you could run

dpkg-query -L nmap | exodus nmap | ssh ...

which would be equivalent to

dpkg-query -L nmap | exodus --include - --output - nmap | ssh ...

I'm generally in favor of something like this, but there are a few implications that I would like think about more first. For example, the nmap package that you listed includes a secondary ncat binary. This binary will (potentially) be useless unless it has a launcher which can invoke the linker for it. You could just package each one separately, but you would end up with a lot of file duplication between the extracted bundles. This raises the question of whether file deduplication should occur across all dependencies or just the libraries.

This gets even more complicated when you move to packages where different binaries within a package exec one another based on relative paths (e.g. Google Chrome). Should you automatically detect all of the ELF executables in a bundle and include their indirect dependencies automatically? There are some implications there as to where the launchers and the original binaries are actually located on the system.

Overall, I think that adding this would probably be a good time to try to rethink the directory structure so that it's better suited for packages with multiple binaries. I know that's not strictly necessary for what you're proposing here, but I would like to limit the number of significant changes to the bundle structure to whatever extent possible. I'll give this some thought and come back later with some more ideas about how to proceed.

[sangaline]

@bmaia Exodus 2.0 was released today, and there are now several ways to add additional files!

# Pipe the list of files into the Exodus command.
find /usr/share/nmap | exodus nmap

# Use the --add/--additional-file command-line argument.
exodus --add /usr/share/nmap nmap

# Attempt to automatically detect the system package and all of the files it includes.
exodus --detect nmap

I'm going to close this for now, but please let me know if you run into any further issues.