Firmware unpacking fails

[arun-gurung]

I`m on CentOS 7.3 and I get this error when downloading the firmware. Please help.

[root@macbook-air firmware]# make

Checking dependencies for driver download... /bin/curl /bin/xzcat /bin/cpio

Downloading the driver, please wait... mv: cannot stat ‘System/Library/Extensions/AppleCameraInterface.kext/Contents/MacOS/AppleCameraInterface’: No such file or directory make: *** [AppleCameraInterface] Error 1

[WhyNotHugo]

Looks like the failing line the the one that's piped to /dev/null.

@(curl -s -L -r "$(RANGE)" "$(URL)" | xzcat -q | cpio --format odc -i -d "./$(FILE)") &> /dev/null || true

I'm not sure what the reason for that is -- can you try running that bit manually?

[arun-gurung]

Thanks so much for your reply, how do I run it manually?

[arun-gurung]

Any help, please?

[WhyNotHugo]

Check the Makefile, all three variables are defined there.

[arun-gurung]

This is what I get, if I did it correctly:

[root@macbook-air firmware]# curl -s -L -r "187085540-191012220" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" | xzcat -q | cpio --format odc -i -d "./System/Library/Extensions/AppleCameraInterface.kext/Contents/MacOS/AppleCameraInterface" &> /dev/null || true xzcat: (stdin): Unexpected end of input [root@macbook-air firmware]#

[wvengen]

Interesting, I have that as well. Using -t on cpio instead, I see that AppleCameraInterface is not fully downloaded ... perhaps the new 10.11.3 offset is not correct. Does it work for you if you use "187085540-191012300" for the offset in curl?

[arun-gurung]

With the new range, I get:

[root@macbook-air firmware]# curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" | xzcat -q | cpio --format odc -i -d "./System/Library/Extensions/AppleCameraInterface.kext/Contents/MacOS/AppleCameraInterface" &> /dev/null || true xzcat: (stdin): Unexpected end of input

[wvengen]

@arun-gurung the message is not really a problem (partial downloading means that the archive is incomplete), the question is if you got the extracted file or not (look for a relative System directory).

[arun-gurung]

I don`t get any extracted file, nor is there any relative "System" directory. It just fails downloading.

[wvengen]

That's weird. What does this give for you?

$ curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" | xzcat -q | cpio --format odc -i -t 2>&1 | tee files.txt

I'm getting files.txt

[arun-gurung]

[root@macbook-air firmware]# curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" | xzcat -q | cpio --format odc -i -t 2>&1 | tee files.txt xzcat: (stdin): Unexpected end of input cpio: premature end of file [root@macbook-air firmware]#

[wvengen]

@arun-gurung without files.txt, there isn't much to say here ... oh wait, you say it's empty? That's odd. It looks like curl isn't really working for you, is it?

[WhyNotHugo]

Can you run in intermediate steps and see which one fails?

curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" > partial_dmg
xz --decompress --stdout partial_dmg > decompressed_dmg
cpio --format odc -i -t 2 --file decompressed_dmg > files.txt

This should not swallow up any errors.

[arun-gurung]

curl seems to do fine, but xz gives an error:

[root@macbook-air firmware]# curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" > partial_dmg [root@macbook-air firmware]# xz --decompress --stdout partial_dmg > decompressed_dmg xz: partial_dmg: Unexpected end of input

[UnaiM]

Hi! Newbie here (and also newbie to Linux─ only 2 days since I wiped out macOS out of my machine).

I had the same issue and ended up brute-forcing it until it worked. You will need:

• osxupd10.11.3.dmg
• dmg2img
• xar
• parse_pbzx2.py
• xz (from anaconda repo)
• cpio (from anaconda repo)
• kmod-hfsplus (from ELRepo) probably, haven’t tested without it

Here are the steps I took:

1. dmg2img path/to/osxupd10.11.3.dmg
2. Mount osxupd10.11.3.img─ the only way I was able to was from the Disks app choosing Attach Disk Image… from the application menu
3. Retrieve OSXUpd10.11.3.pkg from the mounted drive
4. cd /path/to/empty/folder, otherwise xar will refuse to extract certain files
5. xar -xf path/to/OSXUpd10.11.3.pkg
6. cd OSXUpd10.11.3.pkg
7. python /path/to/parse_pbzx2.py Payload
8. xz -d Payload.part00.cpio.xz
9. cd path/to/bcwc_pcie/firmware
10. cpio -di < path/to/Payload.part00.cpio
11. In path/to/bcwc_pcie/firmware/Makefile, add # after the @ in the line containing the curl command.
12. Happy days! You should be able to run make

Hope it’s useful :)

[WhyNotHugo]

These are the steps that the script is supposed to take; I wonder why it's failing (maybe the upstream DMG was repackaged?).

[UnaiM]

Can it be because it’s assuming gzip compression, instead of pbzx? According to this answer in StackOverflow:

Since OSX 10.10, the Payload in the .pkg files is encoded as pbzx (which is in turn lzma compressed). It can no longer be extracted using gunzip.

This is what led me towards using parse_pbzx2.py.

[patjak]

I took the liberty to change the title of the issue to something more descriptive. Yes, seems they repackaged the dmg. Urgh.

[arun-gurung]

If they repackaged the .dmg, will you be able to change the script so we all can compile the driver automatically?

-----Original Message----- From: Patrik Jakobsson notifications@github.com To: patjak/bcwc_pcie bcwc_pcie@noreply.github.com Cc: arun-gurung arun-g@runbox.no, Mention mention@noreply.github.com Sent: man., 22 mai 2017 15:24 Subject: Re: [patjak/bcwc_pcie] Firmware unpacking fails (#115)

I took the liberty to change the title of the issue to something more descriptive. Yes, seems they repackaged the dmg. Urgh.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/patjak/bcwc_pcie/issues/115#issuecomment-303098604

[wvengen]

For me the command still works (on Linux) ... how could that be?

[WhyNotHugo]

I wonder if apple is serving different content based on IP/location/UA/etc. Can you confirm you get these same results?

$ curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" -o myfile
$ md5sum myfile
c948c722c9e2eb9660dfb3296620a0b5  myfile

[wvengen]

$ curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" -o myfile
$ md5sum myfile
c948c722c9e2eb9660dfb3296620a0b5  myfile

Interesting.

$ uname -a
Linux localhost 4.10.0-21-generic #23-Ubuntu SMP Fri Apr 28 16:14:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2g zlib/1.2.11 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy PSL 
$ xz --version
xz (XZ Utils) 5.2.2
liblzma 5.2.2
$ cpio --version
cpio (GNU cpio) 2.11
Copyright © 2010 Free Software Foundation, Inc.
Licence GPLv3+: GNU GPL version 3 or later <https://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Phil Nelson, David MacKenzie, John Oleynick
and Sergey Poznyakoff.

[WhyNotHugo]

$ curl --version
curl 7.54.0 (x86_64-pc-linux-gnu) libcurl/7.54.0 OpenSSL/1.1.0e zlib/1.2.11 libpsl/0.17.0 (+libicu/59.1) libssh2/1.8.0 nghttp2/1.22.0
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 

$ xz --version
xz (XZ Utils) 5.2.3
liblzma 5.2.3

$ cpio --version
cpio (GNU cpio) 2.12
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Phil Nelson, David MacKenzie, John Oleynick,
and Sergey Poznyakoff.

I wonder if there's a regression in cpio. Can you upgrade and check?

[wvengen]

Ok, got cpio from Debian exp and xz-utils from source:

$ bin/cpio --version | head -n 1
cpio (GNU cpio) 2.12
$ bin/xz --version
xz (XZ Utils) 5.2.3
liblzma 5.2.3
$ cat myfile | bin/xz -d | bin/cpio --format odc -t | grep 'AppleCameraInterface$'
bin/cpio: warning: skipped 448149 bytes of junk
/tmp/xz-5.2.3/src/xz/.libs/xz: (stdin): Unexpected end of input
bin/cpio: premature end of file
./System/Library/Extensions/AppleCameraInterface.kext/Contents/MacOS/AppleCameraInterface

$ cpio --version | head -n 1
cpio (GNU cpio) 2.11
$ xz --version
xz (XZ Utils) 5.2.2
liblzma 5.2.2
$ cat myfile | xz -d | cpio --format odc -t | grep 'AppleCameraInterface$'
cpio: warning: skipped 448149 bytes of junk
xz: (stdin): Unexpected end of input
cpio: premature end of file
./System/Library/Extensions/AppleCameraInterface.kext/Contents/MacOS/AppleCameraInterface

It seems that cpio 2.12 and xz 5.2.3 on my system handle it well. What else could be different ... ?

[WhyNotHugo]

I've no idea. I'm getting a premature EOF too, but we've the same versions, and the hash matches. If you think of anything else I might try, let me know.

[UnaiM]

$ curl -s -L -r "187085540-191012300" "https://support.apple.com/downloads/DL1858/en_US/osxupd10.11.3.dmg" -o myfile

$ md5sum myfile
c948c722c9e2eb9660dfb3296620a0b5  myfile

$ uname -a
Linux localhost.localdomain 4.4.68-1.el7.elrepo.x86_64 #1 SMP Sun May 14 10:15:33 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 

$ xz --version
xz (XZ Utils) 5.2.2
liblzma 5.2.2

$ cpio --version
cpio (GNU cpio) 2.11
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Phil Nelson, David MacKenzie, John Oleynick,
and Sergey Poznyakoff.

Hm. So the checksum is the same, and I have the same cpio and xz as @wvengen … Could it be curl?

[wvengen]

Since the myfile hash is equal, I think curl is ok. Let's find out if it is xz or not:

$ xz -d <myfile >myfile.unxz
$ md5sum myfile.unxz 
aef16de05eb462e352e71e28ec588021  myfile.unxz

How's that for you?

[UnaiM]

$ xz -d <myfile >myfile.unxz
xz: (stdin): Unexpected end of input
$ md5sum myfile.unxz
aef16de05eb462e352e71e28ec588021  myfile.unxz

Same…

[wpoely86]

Make sure to use cpio 2.12: https://github.com/patjak/bcwc_pcie/pull/67#issuecomment-196722175 It doesn't work on CentOS 7.

[WhyNotHugo]

Nope, fails with cpio 2.12. No distro-specific patches.

$ md5sum myfile
c948c722c9e2eb9660dfb3296620a0b5  myfile
$ xz -d <myfile >myfile.unxz
xz: (stdin): Unexpected end of input
$ md5sum myfile.unxz
aef16de05eb462e352e71e28ec588021  myfile.unxz
$ cpio --version
cpio (GNU cpio) 2.12
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Phil Nelson, David MacKenzie, John Oleynick,
and Sergey Poznyakoff.

[patjak]

Reverting 1aa456e works on my end. The mainline branch actually doesn't contain this commit so that's why I never saw it until now. Not sure what's actually going on though.

I'm going to add a separate project just for the firmware so that we have a central repo that can be packaged by distros and more easily updated (more people with commit access perhaps?).

[WhyNotHugo]

Sounds like a good idea. On Arch/AUR, we actually have a separate package for the firmware so that driver/firmware updates aren't necessarily tied together (especially since building the former is infrequent but slow).

[patjak]

Huh, nevermind, now the master branch is working again. Starting to think there's something wrong with the download. Could we be getting corrupted data or randomly getting redirected to a mirror which have a different copy?

[WhyNotHugo]

Could we be getting corrupted data or randomly getting redirected to a mirror which have a different copy?

Nope, because we were all comparing the md5 of the file we grabbed.

[UnaiM]

I’m trying to update cpio to 2.12 but there seems to be no package for CentOS 7.3. Would you recommend trying to build it, or should I assume that if it’s not out there it’s because it doesn’t work yet? Sorry for slightly OT.

[WhyNotHugo]

cpio 2.12 is definitely not "new" or "experimental", so you should really take it up to your distro's maintainers.

My system reports I installed 2.12 on 2016-05-16T12:54:27 -03, so it's slightly over a year old.

[WhyNotHugo]

I think it's safe to close this now (after the update to master)?

[UnaiM]

I completely forgot to say: as predicted, it all works well now with 2.12 :)

[arun-gurung]

I reinstalled and used kernel 4.4 longterm from elrepo and now everything works without the great workaround which was posted by UnaiM.