Xz compression support

[wanam]

Hi @rovo89

I'm trying to add xz support for xposed, there are many Samsung Apps that are still using it on the newest TW Roms.

I implemented it the same way as you did for gz, i use an updated libxml2 https://github.com/wanam/libxml2 , where xz files were moved out of xmlIO, my main issue here is related to how makefile works, i need to link this library to dex2oat, including "libxml2" on "art_static_libraries" does not seem to be enough, as i get after a successful compilation a linking errors saying:

art/dex2oat/dex2oat.cc:655: error: undefined reference to 'libxml2xzopen(char const, char const_)' art/dex2oat/dex2oat.cc:666: error: undefined reference to '__libxml2xzread(void, void_, unsigned int)' art/dex2oat/dex2oat.cc:682: error: undefined reference to 'libxml2xzclose(void)' art/dex2oat/dex2oat.cc:678: error: undefined reference to '__libxml2xzclose(void)'

What i'm doing wrong?

[rovo89]

Hard to say, looks like the library isn't added to the linker.

Have you had a look at using external/lzma instead? https://android.googlesource.com/platform/external/lzma/+/master/xz-embedded/Android.mk https://android.googlesource.com/platform/external/lzma/+/master/C/Android.mk That's what I would try to use, probably the first one because it seems to be pretty small. I didn't find any projects using them in AOSP, but it's already in the source tree and updated from time to time, so I assume it's used (and therefore tested) internally.

[wanam]

Thanks for your reply,

I just tried xz-embedded and it builds with no issues, i have no errors while inflating the zx file, but i get later this error on a file i comressed my self with 7zip just for tests:

09-30 15:02:02.930 I/dex2oat ( 6968): /system/bin/dex2oat --zip-fd=11 --zip-location=/system/app/SPlanner_Material/SPlanner_Material.apk --oat-fd=12 --art-fd=-1 --compress-image --oat-location=/data/dalvik-cache/arm64/system@app@SPlanner_Material@SPlanner_Material.apk@classes.dex --instruction-set=arm64 --instruction-set-features=default --runtime-arg -Xms64m --runtime-arg -Xmx512m --swap-fd=14 09-30 15:02:02.940 I/dex2oat ( 6968): Using '/system/app/SPlanner_Material/arm64/SPlanner_Material.odex.xz.xposed' instead of file descriptor 09-30 15:02:02.950 W/dex2oat ( 6968): Failed to open ELF file from '/system/app/SPlanner_Material/arm64/SPlanner_Material.odex.xz.xposed': Failed to map ELF file: Failed to find e_shoff value 10969216 less than 1024 in 09-30 15:02:02.950 E/dex2oat ( 6968): Failed to open some dex files: 1 09-30 15:02:03.160 E/installd( 2971): DexInv: --- END '/system/app/SPlanner_Material/SPlanner_Material.apk' --- status=0x0100, process failed

Here is my decompression method, i must be doing something wrong reading/writing files with my lack of knowledge on c++:

static uint8_t xz_in[BUFSIZ];
static uint8_t xz_out[BUFSIZ];
static File* InflateXZ(const std::string& filename, int out_fd, std::string* err) {

  if (out_fd == -1) {
  *err = "No swap file available";
  return nullptr;
  }

  std::unique_ptr<File> out_file(new File(out_fd, false));
  struct xz_buf b;
  struct xz_dec *dec;
  enum xz_ret r;

  FILE* pFile;
  pFile = fopen(filename.c_str(), "rb");
  fread(xz_in, BUFSIZ, 1, pFile);

  b.in = xz_in;
  b.in_pos = 0;
  b.in_size = sizeof(xz_in);
  b.out = xz_out;
  b.out_pos = 0;  // out_pos contains the resulting decompressed size after xz_dec_run() returns
  b.out_size = BUFSIZ;

  // init CRC lookup table
  xz_crc32_init();

  // allocate/initialize decoder
  dec = xz_dec_init(XZ_SINGLE, 0);
  if(dec == NULL) {
     LOG(WARNING) << "xz_dec_init(XZ_SINGLE, 0); FAILED!";
     return nullptr;
  }

  // run decoder
  r = xz_dec_run(dec, &b);
  if (r != XZ_STREAM_END) {
    *err = StringPrintf("xz_dec_run FAILED returned %d", r);
  }

  // deinitialize/deallocate decoder
  xz_dec_end(dec);

  if (!out_file->WriteFully(b.out, b.out_size)) {
    *err = StringPrintf("Could not write to fd=%d: %s", out_fd, out_file->GetPath().c_str());
    return nullptr;
  }

  if (out_file->Flush() != 0) {
  *err = StringPrintf("Could not flush swap file fd=%d", out_fd);
  return nullptr;
  }

  out_file->DisableAutoClose();
  return out_file.release();
}

[wanam]

NVM i fixed it, it seems also 7zip does not use a standard xz compression, i was getting a corrupt file all the time (XZ_DATA_ERROR || XZ_BUF_ERROR), but it works now after compressing an example odex file with the linux xz command.

I will post a test version for Samsung users and update my repo soon.

[wanam]

Hi @rovo89 ,

I think we have a limitation issue with xz embedded, it seems to support only the files compressed with -check=crc32, it fails with "XZ_OPTIONS_ERROR" on crc64 files.

I will try this https://android.googlesource.com/platform/external/lzma/+/master/xz-embedded/xz_config.h#14 and see what happens.

[wanam]

@rovo89 I added some required files to xz-embedded (https://github.com/wanam/xz-embedded) to enable arm and crc64 modes and it seems to work great now, at least with my manual xz compressed files.

I pushed also my changes to my repos, i think my xz inflating method still need some improvements/optimizations from your side :)

[wanam]

Closing this issue. Here are the sources for xz support: https://github.com/wanam/android_art/commit/14705913c552cb41344af3807a55b0240c6aab87 https://github.com/wanam/xz-embedded

Thank you again for your help.