Implement simple packager

[mpenet]

Since we cannot statically compile (yet) libs + the runtime, it could be nice to have a way to bundle pixie-vm with a lib and it's dependencies and have some kind of startup script for it to make it "transparent".

This could be something similar to makeself or UPX From a quick test we could shrink pixie-vm to 3.1 mo with compression. That command would resolve all depedencies of a project, trigger compilation (pxic), create a platform specific script and bundle all the compiled files with pixie-vm. Upon execution the first time it would uncompress all of it in a temp/local directory and run a startup command.

This is nothing too fancy but that would make distribution a bit easier.

[heyLu]

Yeah, that would be really nice!

We can basically just pack up pixie-vm + dust + the library, but we also need to specify what program to run after unpacking. (Or we read it from a :main key in project.edn.)

On Linux, we could unpack to ~/.cache/<some-unique-id> and use that if present. However, we'd really have to use some unique key, maybe an MD5 hash of the whole package.

[mpenet]

Kinda, I was thinking not to bundle dust with it, try to make it slim and use the project.edn to generate a thin startup script with things like the entry point derived from project.edn and proper load path, and have all the dependencies prefetched on the dev machine and bundled in the archive. Then we don't need to bundle dust, worry about curl presence or whatnot, it makes deploys to other OSes even ok'ish in the long run.

For the unpacking yes, something of that sort this could be as simple as ~/.cache/<package-name>-<version> even (it's easier to get to it if you want to clean things up), but an md5 would be fine.

[heyLu]

Ah yes, baking-in the load path makes sense. To make things start up more quickly we can (and should) byte-compile everything as well.

[mpenet]

Yes that what I mean with "trigger compilation (pxic)", sorry I am not super clear, just throwing ideas :] The goal is clojure like uberjar + the runtime + startup script all in one

[thomasmulvaney]

I have written a tarfile reader in pixie so it would be possible to make a tar.gz of a byte compiled library with all its dependencies and read them as though they were extracted.

[mpenet]

@thomasmulvaney the problem is that pixie would have to be installed first in that case. I think it could be better to have all of it (runtime, deps, compiled (or not) pxi files) in a single self extracting archive. This way we just have a startup penality the first time it's run and no need for prerequisites on a fresh machine. makeself is a good example to follow and get inspiration from imho.

[mpenet]

Unfortunately my bash-foo is very weak, I could probably get something working but this will be poorly written. Feel free to take over this task.

[heyLu]

I've got an example up and running. It's not automated, but it works™.

By dark magic, you'll have to assemble the pixie stdlib, your dependencies and your own code into a directory. It will look something like the following:

$ tree packaged-base
packaged-base
├── deps
│   └── heyLu
│       └── hiccup.pxi
│           ├── LICENSE.html
│           ├── project.edn
│           ├── README.md
│           ├── src
│           │   └── hiccup
│           │       ├── compiler.pxi
│           │       ├── compiler.pxic
│           │       ├── core.pxi
│           │       ├── core.pxic
│           │       ├── def.pxic
│           │       ├── element.pxic
│           │       ├── form.pxic
│           │       ├── middleware.pxi
│           │       ├── middleware.pxic
│           │       ├── page.pxic
│           │       ├── util.pxi
│           │       └── util.pxic
│           └── test
│               └── hiccup
│                   └── test
│                       ├── def.clj
│                       ├── element.clj
│                       ├── form.clj
│                       ├── middleware.clj
│                       ├── page.clj
│                       ├── test-core.pxi
│                       ├── test-core.pxic
│                       └── util.clj
├── pixie
│   ├── async.pxi
│   ├── async.pxic
│   ├── buffers.pxi
│   ├── buffers.pxic
│   ├── channels.pxi
│   ├── channels.pxic
│   ├── csp.pxi
│   ├── csp.pxic
│   ├── ffi-infer.pxi
│   ├── ffi-infer.pxic
│   ├── fs.pxi
│   ├── fs.pxic
│   ├── io
│   │   ├── common.pxi
│   │   ├── common.pxic
│   │   ├── tcp.pxi
│   │   ├── tcp.pxic
│   │   ├── tty.pxi
│   │   ├── tty.pxic
│   │   ├── uv-common.pxi
│   │   └── uv-common.pxic
│   ├── io-blocking.pxi
│   ├── io-blocking.pxic
│   ├── io.pxi
│   ├── io.pxic
│   ├── math.pxi
│   ├── math.pxic
│   ├── parser
│   │   ├── json.pxi
│   │   └── json.pxic
│   ├── parser.pxi
│   ├── parser.pxic
│   ├── PixieChecker.hpp
│   ├── repl.pxi
│   ├── repl.pxic
│   ├── set.pxi
│   ├── set.pxic
│   ├── stacklets.pxi
│   ├── stacklets.pxic
│   ├── stdlib.pxi
│   ├── stdlib.pxic
│   ├── streams
│   │   ├── utf8.pxi
│   │   ├── utf8.pxic
│   │   ├── zlib
│   │   │   ├── ffi.pxi
│   │   │   └── ffi.pxic
│   │   ├── zlib.pxi
│   │   └── zlib.pxic
│   ├── streams.pxi
│   ├── streams.pxic
│   ├── string.pxi
│   ├── string.pxic
│   ├── system.pxi
│   ├── system.pxic
│   ├── test.pxi
│   ├── test.pxic
│   ├── time.pxi
│   ├── time.pxic
│   ├── uv.pxi
│   └── uv.pxic
├── pixie-vm
├── run.pxi
├── run.sh
└── src
    └── packaged
        ├── core.pxi
        └── core.pxic

15 directories, 85 files

In addition, I added a top-level run.pxi, which simply calls a function called main in a namespace:

$ cat packaged-base/run.pxi
(require packaged.core)

(apply packaged.core/main program-arguments)

Now you can use makeself to generate a self-extracting archive, that runs that script:

$ makeself packaged-base packaged.run "An example packaged pixie app" ./pixie-vm `cat .load-path` run.pxi
...
$ ./packaged.run 'hello' 'world?!'
...
<h1>HELLO WORLD?!</h1>

And then you have to automate this... Maybe I'll try that tomorrow, but if anyone wants to take over, feel free to do so!

[mpenet]

Cool stuff. We don't need to include the .pxi files in the package right? just the pxic should be enough (given we compiled everything first). There are other "improvements" we'll have to think about later, like how to handle "resources" (prolly via project.edn).