Closed wagnerch closed 1 year ago
LibreELEC uses /usr/lib/kodi/kodi.sh to call/starts kodi.bin, it loads /etc/profile (RO Partition)who reads all in /etc/profile.d/ (RO Partition too) that has 00-addons.conf that have an interesting code for us....
addons profile.d/*.profile
for config in /storage/.kodi/addons/ /profile.d/ .profile; do if [ -f "$config" ] ; then . $config fi done
That could be a solution to set enviromental variables on the same userspace that kodi.bin runs and that mount point is RW, so addding a addon profile.d/something.profile file with the LD_PRELOAD setting will do what we need....I guess.
Yes, profile could also work. However, the file I am using already does this though. /storage/.config/kodi.conf is merged with /run/librelec/kodi.conf by the prestart script /usr/lib/kodi/kodi-config:
# systemctl show kodi |egrep '^ExecStartPre='
ExecStartPre={ path=/usr/lib/kodi/kodi-config ; argv[]=/usr/lib/kodi/kodi-config ; ignore_errors=yes ; start_time=[Wed 2021-05-12 06:39:47 EDT] ; stop_time=[Wed 2021-05-12 06:39:47 EDT] ; pid=3131 ; code=exited ; status=0 }
In this script:
if [ -f /storage/.config/kodi.conf ] ; then
cat /storage/.config/kodi.conf >>/run/libreelec/kodi.conf
fi
And back in the kodi systemd unit file this environment file is sourced in (prior to ExecStart) via:
# systemctl show kodi |egrep '^EnvironmentFiles='
EnvironmentFiles=/usr/lib/kodi/kodi.conf (ignore_errors=no)
EnvironmentFiles=/run/libreelec/kodi.conf (ignore_errors=yes)
EnvironmentFiles=/run/libreelec/debug/kodi.conf (ignore_errors=yes)
But I think to your point, the inputstream.adaptive could be delivered with the script to do the preload. It basically goes into the addon's folder with a directory of profile.d, and any files in that folder will be loaded by LibreELEC's scripts. What we are talking about only covers LibreELEC, doesn't cover Raspberry Pi OS, Xbian, OSMC, etc. They would all have to make similar changes.
I personally think dynamic linker changes is pretty hard pill to swallow, even when glibc/binutils mainline hasn't adopted it yet.
Yup, doing it by profile.d will made the addon responaible of this instead of the distro
if ld.so.preload file exists could be possible run with it instead of LD_PRELOAD
sudo echo "/storage/.kodi/cdm/libwidevinecdm.so" > /etc/ld.so.preload
LibreELEC /etc mount point belongs to a RO Squash File System...but it is an alternative where user can write on /etc , if it is LE can use profiles, if not....kodi.config or something else
I have a fairly brute routine to unpack .relr.dyn section, but libbfd is sparsely documented and I am not sure if it can create relocation entries in .rel.dyn. From what I can tell we would have to:
Not really an expert on the binutils library (bfd), as I mentioned before it is sparsely documented.
uint32_t *relative = (uint32_t *)section_data;
uint32_t *end = relative + (sz/4); // should be DT_RELRSZ / DT_RELRENT
uint32_t base = 0;
printf("sz=%ld base=%08X start=%p end=%p\n", sz, *relative, relative, end);
for (; relative < end; ++relative) {
uint32_t entry = *relative;
if ((entry & 1) == 0) {
printf("%08X *\n", entry);
base = entry + 4;
continue;
}
uint32_t offset = base;
while (entry != 0) {
entry >>= 1;
if ((entry & 1) != 0) {
printf("%08X\n", offset);
}
offset += 4;
}
base += (8*4-1) * 4; // 31-bits * 4
}
The way the RELR relocation entries work is same for 32-bit & 64-bit (except the word size is different). Basically the way the encoding works is: address bitmap... address bitmap...
Each bitmap is either 31 or 63 relocation entries (this is due to a 4-byte vs 8-byte word size), plus the address is a relocation entry. The low bit 0 is a like flag indicating whether the word is a address or bitmap, bitmaps will always have this bit set to 1 and is discarded, if the bit is 0 then it is an address. This means the address can only be even.
This code fragment assumes little endian elf binary and little endian host, too lazy to do it proper for a hack. Anyways, I believe it is feasible, but far from an expert on relocation entries, so just taking a guess that they need to be R_ARM_RELATIVE just because of the sheer number of entries that have vanished between 1679 and 2252.
Just a little bit extra info that may help. I found the LLVM suite seems to support RELR. Eg. Strip wouldn't error when reading the .so etc
@matthuisman I haven't checked it out yet, probably would be a good idea to look. Latest Android NDK mentions RELR as well, and there was some blurb about them moving to LLVM a while ago from Binutils.
The push for this from Google is to save storage on Android devices, and expectation/assumption is it will improve performance.
looks like Google wants to make their own libc https://www.phoronix.com/scan.php?page=news_item&px=Google-New-LLVM-Libc-Thoughts
so possibly they won't bother upstreaming RELR support to glibc
Also, it is possible for IA to somehow call LD_PRELOAD before dlopen() the widevine blob?
and I think https://llvm.org/docs/CommandGuide/llvm-objcopy.html would be what we could try to use to edit the binary
On Android, bionic is their own libc. I guess they are kind of waffling with ChromeOS. :) I was looking at objcopy for Binutils, and likewise it is limited in it's capability.
I think they already tried to upstream this patch to glibc, I believe those posts were on gnu-gabi. It's a bit unclear what happened there. On the Google group the last poke was from a Google employee asking for an update.
@wagnerch wondering if you have seen/looked into the following regarding the static tls size
https://github.com/bminor/glibc/commit/ffb17e7ba3a5ba9632cee97330b325072fbe41dd
which introduced this tunable - glibc.rtld.optional_static_tls
https://www.gnu.org/software/libc/manual/html_node/Dynamic-Linking-Tunables.html
Would require glibc 2.32+ (which as you noted, libreelec does use already). The default behaviour is 512 bytes, so maybe slowly increment it, see if it gives enough headroom maybe.
Just throwing out ideas to be honest, hoping something is useful.
edit: should note, found this eventually due to this https://bugzilla.redhat.com/show_bug.cgi?id=1722181 of which https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c15 may give a command you might be able to use to calculate an appropriate size.
@fuzzard I haven't looked into it, but that is a great tip. Appears you would set it via an env variable: https://www.gnu.org/software/libc/manual/html_node/Tunables.html https://www.gnu.org/software/libc/manual/html_node/Dynamic-Linking-Tunables.html
Something like: export GLIBC_TUNABLES=glibc.rtld.optional_static_tls=1024
I'll give this a shot tomorrow, to see if this eliminates the need for the LD_PRELOAD. Spent half of the day putzing around with libbfd only to decide it's too cumbersome for the job, so now just parsing the ELF headers, section headers, program headers, and so forth with my own code. It's actually pretty simple, understanding what all of the elements are is another matter. And still working on an assumption with no real actual knowledge of whether it would work. LOL.
If anyone is interested: https://paste.ubuntu.com/p/yXJ53wGH9B/
It slurps in the whole file (argv[1]) into memory, and rolls right into dumping data, doing some assertions. Like the last snippet, it assumes it is parsing a little endian ELF binary and is on a little endian platform. I think the little endians won the war, if you ask me. :)
Basically, would need to figure out the right order to re-write headers/sections, etc, and rebuild the relocation table, delete the relr sections & dynamic table entries (related to relr), fix sizes, and write out a binary only to see if my assumption is right. If I am wrong, then well I guess I learned something new and how not to make a light bulb.
GLIBC_TUNABLES didn't seem to make any difference. I tried 1K, and in my test program I tried up to 500K and still nothing. I don't know if the test program needs to declare thread local variables, but I know 1679 works fine with it and 2252 works fine preloaded.
Basically what we need to do (until there is ARM so for common Linux distros available - probably not going to happen soon):
If the glibc patch is only adding support for ELFs using SHT_RELR, it should not change the way every other system component work, correct?
@kszaq Correct. The glibc patch is only adding support for SHT_RELR sections for ELF executables that use them. The patch from what I can see only modifies the dynamic linker code. So binutils still woudln't understand SHT_RELR and any dynamic table entries related to them, but don't see this as a big problem for the casual user.
I am running this patch on LibreELEC, haven't seen an issue.
So widevine blob is optional and so is IA itself.
So I guess we would need to check the .so exists before preloading. Or does trying to preload non existent .so have no issues? So a restart after installing blob would probably be needed?
Yeah preload is a problem, because nothing will be preloaded if the library doesn't exist. So first time users and updates would need a restart to pick it up. Not really ideal.
btw, was able to get something sort of working with Android's relocation_packer tool, but it's not quite designed for what we need. It's designed for a predecessor to RELR relocations, where Android was doing some sort of RLE compression on the existing relocations. What the program does is actually replace SHT_REL section & data with a compressed & encoded version that the Android dynamic linker understood. It doesn't add entries to the dynamic section, or create/delete sections, and the original version messes up the virtual addresses (which I commented out).
This does unpack SHT_RELR into SHT_REL, but it doesn't clean up dynamic section, or remove the relr section. And the binary doesn't quite work, maybe alignment, maybe something else, maybe a bad theory, etc. https://github.com/wagnerch/relr-unpack/tree/master/src
@samnazarko You may want to be following this issue if not already. Affects all ARM Widevine platforms
I’m following it.
Cheers
Sam
On 13 May 2021, at 23:04, Matt Huisman @.***> wrote:
@samnazarkohttps://github.com/samnazarko You may want to be following this issue if not already. Affects all ARM Widevine platforms
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/xbmc/inputstream.adaptive/issues/678#issuecomment-840861693, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAHDWLABF4IDGUQT6MZFZDTTNREIJANCNFSM44Q62UKA.
I think it might be worth trying to approach this from a different angle if the old dlopen() method with stock glibc doesn't work anymore.
eg sandbox libwidevine handling out of kodi into a separate process/jail/container/... and use IPC within the IA addon to communicate with it.
This gives you a lot more freedom (you can use whatever libc/linker/... or workarounds you like) and doesn't need changes to the base system (glibc, preload workarounds etc).
And it might even allow us to finally switch LibreELEC armv8 (RPi4, ...) userspace to aarch64 - currently 32bit-only widevine is stopping us to do this. With a clear separation between kodi and widevine we would just need to care about getting that single arm32 program running on LE.
BTW: A major drawback of the LD_PRELOAD approach is that things will go really south when preloading fails - eg if libwidevine starts to dynamically link to some non-existent lib, switches to a different arch or is otherwise corrupted. Then kodi won't start anymore and users will be even more unhappy compared to just having a non-working netflix.
I can't speak for the whole LE team but I think it's highly unlikely we would be adding out-of-tree glibc patches (this will become a maintenance nightmare on each glibc upgrade) or general LD_PRELOAD hooks which have the potential to break kodi startup (this will become a support nightmare)
@wagnerch I have patched glibc in my LE build and I checked
-Wl,--no-as-needed libwidevine.so
and
patchelf --add-needed /storage/.kodi/cdm/libwidevinecdm.so libssd_wv.so
but unfortunately the TLS error is still there.
These work for the standalone app that simply dlopen("libwidevinecdm.so", RTLD_LAZY);
- my guess is that because we need to add this requirement for the main app (Kodi) and not libssd_wv.so
because when the main app is loaded static TLS is already used up to the point when we cannot dlopen a lib that declares dependency on libwidevine. In this scenario patching Kodi executable would be equal to using LD_PRELOAD
.
I'm wondering how ChromeOS deals with this TLS exhaustion - or maybe Chrome simply uses so little static TLS that allows it to dlopen Widevine?
We only have 2 weeks to find a solution. After that the working Widevine CDM releases may get revoked... Not sure whether Google will enforce this as announced. (Did they update all ARM devices they support? Are they willing to break recently EOL devices?)
Our expectation was that LaCrOS would have happened by now, but we were as surprised by the breakage of the Widevine CDM before LaCrOS.
@kszaq You would probably need to set LD_LIBRARY_PATH to /storage/.kodi/cdm before starting Kodi. Wasn't aware of patchelf, but did this:
patchelf --add-needed libwidevinecdm.so wvdl
# cp /storage/.kodi/userdata/addon_data/script.module.inputstreamhelper/backup/13816.64.0/libwidevinecdm.so .
# LD_LIBRARY_PATH=/storage ./wvdl
CDM Version: 4.10.2252.0
Let me know if that works.
@wagnerch Yes, this works: if an executable declares dependency on libwidevinecdm.so
, it runs correctly.
But if I patch libssd_wv.so
to depend on libwidevinecdm.so
with pathelf (or -Wl,--no-as-needed
during compile), Kodi cannot dlopen libssd_wv.so
because of static TLS error.
@kszaq OK, I see. Well that sucks. I don't see anything special in Chromium, but perhaps I am missing it.
https://github.com/chromium/chromium/blob/d7da0240cae77824d1eda25745c4022757499131/base/native_library_posix.cc#L23 https://github.com/chromium/chromium/blob/d7da0240cae77824d1eda25745c4022757499131/base/native_library.cc#L9 https://github.com/chromium/chromium/blob/d7da0240cae77824d1eda25745c4022757499131/base/scoped_native_library.h#L30 https://github.com/chromium/chromium/blob/d7da0240cae77824d1eda25745c4022757499131/base/scoped_native_library.cc#L24 https://github.com/chromium/chromium/blob/d7da0240cae77824d1eda25745c4022757499131/media/cdm/cdm_module.cc#L136
Could be something higher up, perhaps a patched dynamic linker, etc. Maybe the public repositories haven't hit any changes for it yet. Or it might be because of tcmalloc is statically linked with the main Chromium binary. Doing an LD_PRELOAD=libtcmalloc.so.4 also works on my test program.
LD_PRELOAD=libtcmalloc.so.4 also works
Well that be easier to do than preload widevine blob that can't be included in kodi?
CC @tanio99 @graham8
@wagnerch I can confirm that LD_PRELOAD=libtcmalloc.so
makes the latest widevine blob work in Kodi (patched glibc).
To be precise, I tried this with libtcmalloc_minimal.so.4.5.9
I also tried to patchelf
libtcmalloc dependency to libwidevine but this did not work.
What about patchelf the wv decrypter so?
@kszaq Only catch is tcmalloc replaces malloc & friends. I don't think this is a big deal. It was always in libwidevinecdm.so from what I can see based on strings, so who knows what kind of behavior that would have. I don't know if they statically link tcmalloc on desktop browser CDM's.
I read somewhere that this TLS error may be caused by dlopen'ing a malloc replacement library that also uses TLS. No one really explained it, and reading a few of the whitepapers on TLS was a bit too abstract for someone no prior exposure to the concept.
@matthuisman Unfortunately this does not work: Unable to load /storage/.kodi/cdm/libtcmalloc_minimal.so.4.5.9: cannot allocate memory in static TLS block
I start to think that using tcmalloc prevents the TLS error from happening.
patchelf --add-needed libtcmalloc_minimal.so.4.5.9 ssd_wv.so ?
is what you did?
@matthuisman Yes and the result was as above.
Now my conclusion is: to use the latest widevine blob, we need to patch glibc and use tcmalloc.
I thought stripping DT_INIT & DT_FINI might solve the TLS issue, but no such luck. These are the constructor & destructor and are called at dynamic load & unload of the library.
I spent quite a while trying to figure out how to grow the ELF binary file for the unpacked relocations, and I think what would need to happen is moving the .rel.dyn section to the end of the file (before the segment headers), and add a new program header to load the segment. The problem is .rel.dyn is so early in the file that growing it would require pushing down all of the virtual memory addresses. I am guessing the original utility didn't actually close the hole, so it didn't have to deal with this. It basically just adjusted offsets in the file and used a compress relocations (SHT_ANDROID_REL/A), so the unpacker reversed the process and still didn't have to adjust VMAs.
But because this done by a proper linker, it adjusts the VMAs based on page alignment rules. That's the other pain, each section has an alignment, and then sections belong to a program header load segment which has an alignment as well. The most important alignment is the program header load segments, because mmap will not work if the alignment isn't right. So to avoid that whole mess, my thought is to move the Offset & VMA of .rel.dyn. The other problem is this program uses libelf (from elfutils) and it actually doesn't have an interface to delete a section. Ugh...
If you move VMAs, then you have to update a ton of pointers, not even sure if the code would still work, but all of the entries in relocation tables are VMAs, which is a real pain. Plus you have other relocation tables at play as well. So definitely don't want to mess with VMAs.
Best solution so far is patching the dynamic linker, or seeing if there is a browser based version for ARM. Which seems unlikely, because the thread on RPi suggests they are pulling the same sources, so I don't think Google distributes one.
Could Kodi itself just be linked against tcmalloc? From what I gather it's fairly drop in capable (ie just add a linker flag). You won't get the tls issue, as the tcmalloc lib is loaded by the Kodi binary, and not via dlopen. I guess that puts a burden on the elec build systems to build tcmalloc, but tcmalloc is available for normal Linux distros via their package manager of choice it looks like. Also reads like it provides benefits in general anyway, and can just use tcmalloc_minimal so the perf stuff isnt used.
If that works, that just leaves your glibc patch for the relo stuff, I'll leave that for the elec maintainers to discuss the pros/cons of that
A patchset for *ELEC:
https://github.com/LibreELEC/LibreELEC.tv/compare/master...kszaq:widevine_support https://github.com/CoreELEC/CoreELEC/compare/coreelec-19...kszaq:widevine_support
All credit for finding the solution goes to @wagnerch
The only upside of 4.10.2252.0 is that it seems to offer better performance than older versions: on my S905X board a 15Mbps stream is decrypted fast enough to keep buffer filled and a 1080p Netflix video looks smoother - both tested on the same build with tcmalloc preloaded.
@kszaq Yeah, I think the cleaner approach would be to link Kodi against tcmalloc_minimal, apparently you just need to add -ltcmalloc_minimal to the link.
To use TCMalloc, just link TCMalloc into your application via the "-ltcmalloc" linker flag. ... If you'd rather link in a version of TCMalloc that does not include the heap profiler and checker (perhaps to reduce binary size for a static binary), you can link in libtcmalloc_minimal instead.
@wagnerch In my proposed patchset tcmalloc is preloaded for Kodi - this way you can "turn it off" in case there are any issues from using it.
@kszaq Ahh, I see. By the way, it occurred to me that some of the performance bump may be from tcmalloc itself on the whole Kodi binary, since we are preloading a malloc replacement. Not sure if it is noticeable anywhere else? Maybe, try with the preload and the old 4.10.1679.0 on the same 15Mbps stream?
I tested both 4.10.1679.0 and 4.10.2252.0 with the same Kodi build and tcmalloc preloaded. 2252 definitely works better.
I have not noticed any difference in performance anywhere else but I admit that I haven't tested it for too long.
@kszaq Here is the slight variation on your patches to LibreELEC, which is adding TCMalloc as a link dependency. Also moved the dependency on gperftools from glibc to kodi, since this approach is not relying on LD_PRELOAD. I guess I am all in. :)
https://github.com/LibreELEC/LibreELEC.tv/compare/master...wagnerch:widevine_support
# ldd /usr/lib/kodi/kodi.bin |grep tcmalloc
libtcmalloc_minimal.so.4 => /usr/lib/libtcmalloc_minimal.so.4 (0xb6778000)
Nice work. I just did the lazy thing and put the patch in glibc/patches to test it out. As a side note, Debian buster has additional issues, there is some other version dependency that @XECDesign ran into, his findings were glibc-2.28 didn't work (maybe some dependency in widevine), but glibc-2.31 (bullseye) was working. I can appreciate some of the advantages of the JeOS approach now. :)
For Debian Buster builds, the issue with glibc-2.28 are these Debian specific patches: arm/unsubmitted-ldconfig-cache-abi.diff arm/unsubmitted-ldso-abi-check.diff
If it is reverted, and the SHT_RELR patch is applied, then the current solution should work.
Interesting: When I first load Widevine from 13505.111.0 ChromeOS image, start a movie, stop it, then let InputStream Helper update the lib to the one from 13816.64.0, then start another movie, the new Widevine Lib works. When I then restart Kodi, it no longer works (fails to load). Needs further investigation what's causing this. Can someone confirm this behavior? Maybe that helps to find another fix for the problem. _(note that I don't have libtcmallocminimal at all in the system)
@stsichler It means that the new library is loaded after restarting Kodi. InputStream Helper just copies the widevine lib from the ChromeOS image to the Kodi cdm directory.
@mediaminister 'pmap' states that InputStream Helper unloads the lib from the kodi process meanwhile updating, so, no, I'm pretty sure that the updated lib is used.
EDIT: hm, okay. it seems that I misinterpreted the output of 'pmap'. So you're right. It's still the old lib that is loaded until a restart. :-/ sorry for bugging...
Did anyone tried to preload library with LD_PRELOAD? Then cannot allocate memory in static TLS block
doesn't happen.
And of course dlopen() is not needed.
@vpeter4 Yes: https://github.com/xbmc/inputstream.adaptive/issues/678#issuecomment-839287040
Preloading/linking Kodi with tcmalloc is probably a safer approach than to preload libwidevine.
And you still need to patch glibc to support SHT_RELR.
@kszaq Just quick question.. with the changes, older blob can still be loaded? I assume you but just wanted to check. Thinking if they suddenly swap back or something like that. We can support both old and new
Yes, both old and new work. I checked old vs new performance with a patched LE build and there are no issues when using "old" widevine.
It appears Google may have changed the latest Widevine libraries, the size has changed by about 1M, and it is not linking all of the same libraries as it used to (perhaps some stuff is statically linked now). In any case these are the errors, but there isn't enough logging to know the issue. Also the dlerror() message is squashed and never bubbles up.
2021-05-09 20:29:53.661 T:4807 ERROR: AddOnLog: inputstream.adaptive: Unable to load widevine shared library (/storage/.kodi/cdm/libwidevinecdm.so)
2021-05-09 20:29:53.661 T:4807 ERROR : AddOnLog: inputstream.adaptive: OpenDRMSystem failed
Related issues: https://github.com/CastagnaIT/plugin.video.netflix/issues/1154
Working CDM:
13505.111.0/libwidevinecdm.so: linux-vdso.so.1 (0xbefbd000) /usr/lib/libarmmem-v7l.so (0xb6808000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0xb67df000) libm.so.6 => /usr/lib/libm.so.6 (0xb6776000) libdl.so.2 => /usr/lib/libdl.so.2 (0xb6763000) librt.so.1 => /usr/lib/librt.so.1 (0xb674c000) libnss3.so => /usr/lib/libnss3.so (0xb664f000) libnssutil3.so => /usr/lib/libnssutil3.so (0xb661c000) libnspr4.so => /usr/lib/libnspr4.so (0xb65e5000) libc.so.6 => /usr/lib/libc.so.6 (0xb64a6000) /usr/lib/ld-linux-armhf.so.3 (0xb6f66000) libplc4.so => /usr/lib/libplc4.so (0xb6f8f000) libplds4.so => /usr/lib/libplds4.so (0xb6f8b000)
Non-working CDM:
13816.64.0/libwidevinecdm.so: linux-vdso.so.1 (0xbedb9000) /usr/lib/libarmmem-v7l.so (0xb5dd7000) libdl.so.2 => /usr/lib/libdl.so.2 (0xb5dc4000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0xb5d9b000) librt.so.1 => /usr/lib/librt.so.1 (0xb5d84000) libm.so.6 => /usr/lib/libm.so.6 (0xb5d1b000) libc.so.6 => /usr/lib/libc.so.6 (0xb5bdc000) /usr/lib/ld-linux-armhf.so.3 (0xb6f78000)