OnePlus 6 Android 9: Invalid Vdex header

[mwilky]

Getting an error when trying to deodex OxygenOS Android 9.0 on OnePlus 6.

Please see full debug log attached: http://renovate-ice.com/log.txt

app, framework and priv-app files I have uploaded here: http://renovate-ice.com/deodex_error.zip

Anything else you need let me know

[anestisb]

You're setting the output path to a directory that does not exist (./test). Create the directory ($mkdir test) before running the tool and you should be ok.

I might consider adding some directory existence checking logic in the tool to ensure that output directory exists before continuing with the file processing.

[mwilky]

So after creating the output path, i get a bunch (not all) of .cdex files. Still getting the invalid header in log though

http://renovate-ice.com/log2.txt

[anestisb]

The reason you get the "Invalid Vdex header" message is because you're providing an input directory that does not have only vdex files. From your log I see for example that is has oat, apk & jar files. The tool is configured to ignore all non-vdex file from the input directory. The way it identifies valid vdex files is by processing the header section (thus the invalid header message).

Now the reason that some vdex containers might not contain an actual dex file (no DexSection present), is because it's not mandatory. Since the vdex containers have other purposes too (e.g. verifier dependencies) it is not mandatory to contain actual dex files, so there is nothing to export. You can verify that by running strings against the vdex file and grep against cdex (or inspect via a hex editor and search for cdex).

If you have a specific case/file that you think is not processed properly please be explicit and mention the reproduction steps. Its very hard to trace something in such a large log.

[mwilky]

Managed to get it all working after fresh eyes looking at it, thanks for your guidance.

@anestisb are you aware of any dexopt flags (or anything else) we need to set ROM side to improve the performance when deodexed? Noticing some real delays stutters compared to leaving everything as vdex, and have never succesfully managed to re-convert back to vdex.

[anestisb]

@mwilky I don't really understand what you're trying to achieve. After you've deodexed the Dex files from the Vdex containers, they're fully functional and can preoptimized again from the AOSP build environment. I'm already doing this as part of the android-prepare-vendor project. You simply have to repackage the extracted Dex files back to their matching Apks/Jars and recompile your ROM with the required Makefile adjustments to include the precompiled binaries.

[mwilky]

@anestisb I understand that, and when having the dex file bundled inside the apk/jar it works fine. The only issue is you get a lot more noticeable lag when running a deodexed system (dex inside jar), compared to running a odexed system (vdex outside jar).

[anestisb]

@mwilky then there must be something wrong with your ROM ART optimization/building options. Are you preoptimizing all system apps/frameworks at build time? Or have you modified the AOSP default ART build settings?

You shouldn't have any noticeable performance penalties. I've already tested against a few sample system apps and everything appears to work as expected.

[mwilky]

@anestisb yeah that's where I'm thinking the issue is, I'm on an oem based ROM so they may have tinkered with them. Could you send me a list of system props (grep dex) from a system you have tested which has no performance decrease? The issue only seems to occur when framework.jar has the dex file deodexed bundled in the jar. When I run a build fully odexed and say just services.jar deodexed, no lag is present.

[anestisb]

Check that the WITH_DEXPREOPT flag is true and not overwritten from the makefiles. The props from a Pixel device are:

sailfish:/ $ getprop | grep -E 'dex|vm|art'
[dalvik.vm.appimageformat]: [lz4]
[dalvik.vm.dex2oat-Xms]: [64m]
[dalvik.vm.dex2oat-Xmx]: [512m]
[dalvik.vm.dex2oat-minidebuginfo]: [true]
[dalvik.vm.dexopt.secondary]: [true]
[dalvik.vm.heapgrowthlimit]: [256m]
[dalvik.vm.heapmaxfree]: [8m]
[dalvik.vm.heapminfree]: [512k]
[dalvik.vm.heapsize]: [512m]
[dalvik.vm.heapstartsize]: [8m]
[dalvik.vm.heaptargetutilization]: [0.75]
[dalvik.vm.image-dex2oat-Xms]: [64m]
[dalvik.vm.image-dex2oat-Xmx]: [64m]
[dalvik.vm.isa.arm.features]: [default]
[dalvik.vm.isa.arm.variant]: [kryo]
[dalvik.vm.isa.arm64.features]: [default]
[dalvik.vm.isa.arm64.variant]: [kryo]
[dalvik.vm.lockprof.threshold]: [500]
[dalvik.vm.stack-trace-dir]: [/data/anr]
[dalvik.vm.usejit]: [true]
[dalvik.vm.usejitprofiles]: [true]
[init.svc.vendor.devstart_sh]: [stopped]
[partition.system.verified]: [2]
[partition.vendor.verified]: [2]
[persist.sys.dalvik.vm.lib.2]: [libart.so]
[persist.sys.ssr.restart_level]: [venus,AR6320,slpi,modem,adsp]
[persist.sys.webview.vmsize]: [147551200]
[pm.dexopt.ab-ota]: [speed-profile]
[pm.dexopt.bg-dexopt]: [speed-profile]
[pm.dexopt.boot]: [verify]
[pm.dexopt.first-boot]: [quicken]
[pm.dexopt.inactive]: [verify]
[pm.dexopt.install]: [speed-profile]
[pm.dexopt.priv-apps-oob]: [false]
[pm.dexopt.priv-apps-oob-list]: [ALL]
[pm.dexopt.shared]: [speed]
[ro.art.hiddenapi.warning]: [1]
[ro.cp_system_other_odex]: [1]
[ro.dalvik.vm.native.bridge]: [0]
[sys.wifitracing.started]: [1]

[mwilky]

Much appreciated pal, thanks for your time

[mwilky]

@anestisb still seeing the lag even with the props set to the same as yours. Most noticeable when pressing recent apps button/home button, and the time taken to actually do the action. All the vdex files are getting created in data/dalvik-cache/arm64.

For an example say services.jar. Should the vdex file in data/dalvik-cache/arm64, which gets created from a stock deodexed services.jar be identical to a pre-deodexed services vdex?

[baadnewz]

hey there, Im working with @mwilky on the ROM and the lag is noticeable. The issue is: we're dealing with an OEM ROM (Oxygen OS BETA Android Pie for OnePlus 6) and even if we match Pixel props regarding dex and there is no change in terms of lag/no lag. We do not have access to OOS source so we can't tune ROM build parameters, we're dealing with what we are given, prebuild ODEXed ROM, which we deodex and work our mods through smali. Also we dont have access to what params they use when they generate the files

Any idea or suggestion you might have we appreciate it. Thanks

[anestisb]

@mwilky @baadnewz it's hard to help without having an actual device to replicate the issue and investigate. The only reason I can think of is that the when the runtime is initialized, it's not happy with the preoptimized boot framework code you modify, and since Pie ART enforces symlinks to system for boot frameworks, the runtime fails-over to interpreter mode which is very slow. Try obtaining a logcat output of the zygote process after a device reboot and check for warnings/errors during the runtime initialization process.