Implement tebako packaging mechanism

[ronaldtse]

The goal is to package a language interpreter with its dependencies and packages into a single executable.

We will implement something similar to https://github.com/metanorma/ruby-packer in this manner:

1. Use vanilla Ruby, redirect file calls to the internal packaged file system
2. Package the Ruby runtime, its dependencies and packages (i.e. gems) in a DwarFS system.
3. Adopt code from ruby-packer that allows detecting gems and consolidate necessary files.

The flow goes:

1. Package the filesystem. Write all files (Ruby binary, gems, compiled gems) into dwarFS, store the dwarFS file somewhere, then compress the dwarFS file.
2. Embed this compressed dwarFS file into the executable. We can directly link it using mechanisms described here:
   • https://codeplea.com/embedding-files-in-c-programs
   • https://developers.redhat.com/blog/2019/07/05/how-to-store-large-amounts-of-data-in-a-program
   • The incbin macro can be used here for simplicity: https://github.com/graphitemaster/incbin
3. On execution, the code runs dwarFS and opens the compressed file with memory allocation of the full size (in read-only mode).
4. Then execute the Ruby from the dwarFS package.

Originally from: https://github.com/metanorma/ruby-packer/issues/16

[maxirmx]

Ruby-packer uses libsquashfs which is an SDK specially designed to provide for build-in SquashFS support in the application. DwarFS does not include any SDK. There is a FUSE driver that can be adjusted and linked into an application. It is not something envisioned or supported by the author and doing it basically means very hard fork with limited options to incorporated further versions of DwarFS

[ronaldtse]

@maxirmx libsquashfs was written by the author of ruby-packer -- just to implement ruby-packer. The method he used was to simplify the existing SquashFS FUSE driver into a library. I believe we can do the same for DwarFS using the existing FUSE driver as a basis.

The author of DwarFS has indicated this usage can be supported and I believe he is open to such extension.

i.e. the work is to come up with a "libdwarfs" (by modifying the FUSE driver) and use it for tebako.

[maxirmx]

We need to split dwarfs.cpp and dwarfs applications into: 1) "libdwarfsfuse" that includes _runfuse function and filesystem operations -- _op_init ... opgetxattr This new libary will be embeddable dwarfs fuse driver libdwarfsfuse shall staically link existing libdwarfs and other libraries (?? if possible) 2) Smaller dwarfs application that does argument parsing (main and _rundwarfs functions)

[ronaldtse]

Do you mean libdwarfsfuse or libdwarfs?

Why do we need a "smaller dwarfs application"? We can do image builds directly using the existing dwarfs?

[maxirmx]

Task list as of Sept 6th, 2021

• [x] 1. Adopt custom ruby build from ruby-packager with version upgrade to 2.7 (1 week)
• [x] 2. #14 (1 week)
• [x] 2.1 #15 (Added on Sept 4th, +1 week)
• [x] 2.2 #13 Now #24 (Added on Sept 6th, +1 week)

At this point we shall be able to create packaged ruby executable manually. It will have dependency on fuse

• [x] 3. https://github.com/tamatebako/libdwarfs/issues/1, this item also covers #10 (1 week)
• [x] 4. https://github.com/tamatebako/libdwarfs/issues/2 (1 week)
• [x] 5. #18 (1 week)
• [x] 6. #19 (1 week)

At this point we shall be able to create packaged ruby executable manually without dependencies on fuse

[ronaldtse]

@maxirmx can you help split the tasks into individual issues and link them back here? It's easier to manage that way. Thanks.

[maxirmx]

@maxirmx can you help split the tasks into individual issues and link them back here? It's easier to manage that way. Thanks.

Done

[ronaldtse]

Thanks! I've also cleaned up the list for presentation.