Shipping formats repository with a compiler distribution

GreyCat commented 7 years ago

Given that we're almost ready to do 0.7 release and we've got imports, I'd want to discuss if we'd want to ship our formats repo along with the ksc. Currently, user has to download it manually and, what's even more challenging, to use it properly with absolute imports, one has to issue some kind of command like

ksc -I /path/to/formats/repo -t java some_file_that_uses_imports.ksy

Shipping formats repo along with ksc will probably make these much easier for end-user. For that, the following must be done:

Decide where formats repo should go

For Debian (and Linux in general), I propose /usr/share/kaitai-struct/.

For universal package, probably it's a good idea to derive formats repo path relatively to launcher script, i.e.:

kaitai-struct-compiler-0.7/bin
kaitai-struct-compiler-0.7/bin/kaitai-struct-compiler
kaitai-struct-compiler-0.7/bin/kaitai-struct-compiler.bat
kaitai-struct-compiler-0.7/lib 
kaitai-struct-compiler-0.7/formats # <= here
kaitai-struct-compiler-0.7/formats/archive
kaitai-struct-compiler-0.7/formats/game
kaitai-struct-compiler-0.7/formats/...

For Windows, probably the same trick as with universal package will work too. We'll just deploy formats somewhere along with bin and lib, and the same path search code will probably work.

For POM build, probably we can skip all this and just start with empty default search path.

Decide what to ship

Probably whole formats repo, but without /_build directory.

Package the files

Probably it's not hard to change sbt configuration to package additional files for universal and Windows builds. For Debian, I'd probably opt to:

package empty /usr/share/kaitai-struct directory in kaitai-struct-compiler*.deb
package /usr/share/kaitai-struct directory contents in distinct kaitai-struct-formats*.deb

Fix compiler build to make sure that default search path contains wanted directories

There are 2 fixes to be done:

For universal and Windows — probably we need to fix launcher script to pass a "kaitai struct compiler home" directory somehow to a JVM; may be that var already exists.
For Linux — probably build.sbt should set up some constant that will preset search path to a hard-coded directory

Comments, opinions? Cc @koczkatamas @tschoening

ams-tschoening commented 7 years ago

Agreeing with everything read and for Windows the var seems to already exist:

if "%KAITAI_STRUCT_COMPILER_HOME%"=="" set "KAITAI_STRUCT_COMPILER_HOME=%~dp0\.." set "APP_LIB_DIR=%KAITAI_STRUCT_COMPILER_HOME%\lib\"

Adding an APP_FMT_DIR in the same way seems like a reasonable choice.

GreyCat commented 7 years ago

@tschoening Technically, the less the have to modify, the better. If we'll run a modifier copy of launcher script, then we'll have to maintain it later, and that I'd like to avoid ;) So, from practical point of view, it's easier to process these from the Scala code.

KAITAI_STRUCT_COMPILER_HOME seems to exist in bat launcher, but there's nothing like that in bash launcher. There is app_home in there, but it's not exported to the environment, nor passed through -D to JVM. I'll try to think of something.

GreyCat commented 7 years ago

Confirmed that KAITAI_STRUCT_COMPILER_HOME is available on Windows, but nothing like that seems to be passed in Unix environment → probably have to fix the launcher :(

KOLANICH commented 7 years ago

I vote aganst of filesystem-based shipment in favour of network-based described in the issue about modularization.

GreyCat commented 7 years ago

Ugh, many hours of ugly debugging later, it finally seems to work. At the very least, everything compiler-related now builds properly.

kaitai-io / kaitai_struct