unpack/pack_nested_esf robustness and flexibility

[burnce]

Multiple changes to improve robustness and flexibility of nested packing and unpacking:

• Weaken assumption from "esf2xml"/"xml2est" present in current working directory to present in same directory as "pack_nested_esf"/"unpack_nested_esf"
• Fixed issue with mac detected as windows in pack_nested_esf ('Darwin' contains 'win')
• Improved detection of OS to distinguish between Mac and linux
• Allow specific versions of lzma to be specified using environment variables.
  • LZMA to designate one executable for both compression and decompression
  • LZMA_COMPRESSION to designate executable specifically for compression
  • LZMA_DECOMPRESSION to designate executable specifically for decompression
• Print error message to STDERR on failure of lzma system call

[taw]

I don't like this solution, since it will require everyone to setup some environment variables to do things they don't understand.

I tried it on a Mac and I have this lzma installed and it works for me already:

$ ls -l `which lzma`
lrwxr-xr-x taw admin 27 B Thu Mar 19 18:20:36 2020  /usr/local/bin/lzma ⇒ ../Cellar/xz/5.2.5/bin/lzma

Is there some other lzma from other source that doesn't work?

xz package also has xz program, maybe we should use this instead on a Mac?

Either that, or we could bundle statically linked binary.

That OS detection code is probably better than what I had, so I can take that part.

[burnce]

I take your point about ease of use. In my opinion the environment flags are useful as an advanced feature. If they are not set and the path happens to contain the correct lzma everything should work.

I have xz-5.2.5 installed, but I cannot get the compression to create a file compatible with esf encoding/decoding.

Did you test unpacking a file packed with xz-5.2.5 ? savefile -> unpack_nested_esf -> pack_nested_esf -> unpack_nested_esf fails with message about corrupted file.

Also, be sure that the pack_nested_esf is actually running the compression. In my case, timestamps showed that compressed_data.esf.xz had not been regenerated from compressed_data.esf during packing. (This in turn led me to find the issue with OS detection and add the error message on unsuccessful execution of lzma)

My suspicion is that the corruption issue is with "streamed" vs. "non-streamed format"

# compressed_data.esf.xz  created using LZMA Utils (lzma-4.32.7)
# compressed_data2.esf.xz created using XZ (xz-5.2.5)

$ file *.xz
compressed_data.esf.xz:  LZMA compressed data, non-streamed, size 26539708
compressed_data2.esf.xz: LZMA compressed data, streamed

$ lzmainfo compressed_data*.esf.xz

compressed_data.esf.xz
Uncompressed size:             25 MB (26539708 bytes)
Dictionary size:               8 MB (2^23 bytes)
Literal context bits (lc):     3
Literal pos bits (lp):         0
Number of pos bits (pb):       2

compressed_data2.esf.xz
Uncompressed size:             Unknown
Dictionary size:               8 MB (2^23 bytes)
Literal context bits (lc):     3
Literal pos bits (lp):         0
Number of pos bits (pb):       2

From the xz documentation it is clear that xz cannot create non-streamed files

xz supports decompressing .lzma files with or without end-of-payload marker, but all .lzma files created by xz
will use end-of-payload marker and have uncompressed size marked as unknown in the .lzma header.  This may  be
a  problem  in  some  uncommon situations.  For example, a .lzma decompressor in an embedded device might work
only with files that have known uncompressed size.  If you hit this problem, you need to  use  LZMA  Utils  or
LZMA SDK to create .lzma files with known uncompressed size.

[burnce]

I've bundled an lzma version for mac. It is statically linked but still depends on:

• /usr/lib/libc++.1.dylib
• /usr/lib/libSystem.B.dylib

[taw]

Manually merged due to some minor merge conflicts.

I verified that I can run lzma.mac on my macbook, but didn't do any special checking beyond that.

(I'm also pretty sure that running ./foo is much simpler than trying to figure out current path)