sed error in VM - InstallInfo.plist is a binary file instead of plaintext

[kathywh]

After running the command in VM terminal: sed -i.bak -e "s/InstallESDDmg\.pkg/InstallESD.dmg/" -e "s/pkg\.InstallESDDMg/dmg.InstallESD/" "${install_path}InstallInfo.plist" && sed -i.bak2 -e "/InstallESD\.dmg/{n;N;N;N;d;}" "${install_path}InstallInfo.plist" it reported an error: sed: RE error: illegal byte sequence

Script version: 0.73.4 (with modification of: vmname="Mojave") VirtualBox version: 6.0.10 r132072 Host OS: Ubuntu 18.04.3 LTS Guest macOS: Mojave

Screenshots

VM Terminal:

Host Terminal:

[myspaghetti]

If you still have the temporary files and configured VM, would you mind running the following:

./macos_guest_virtualbox.sh prepare_the_installer_app start_the_installer_app place_efi_apfs_drivers detach_installer_vdi_and_viso boot_macos_and_clean_up

Otherwise if you have deleted the virtual machine or temporary files please start the script from the beginning. It appears sed had an illegal key in its regex, which happens because the script is not robust to errant keystrokes. If the issue persists I'll try to reproduce the error.

[kathywh]

I have run 7 clean installs these days, i.e. removing all temp files before running install script, 'sed' command failed 3 times, same RE errors as the above. It seems that the error happens randomly. I have all temp files and VM files after the last failing install. Hope it is helpful for debug.

[myspaghetti]

That's a lot of testing! Thank you for all the testing and reporting.

You can shut down the virtual machine after the sed stage and just run the prepare_the_installer_app stage multiple times, if you want to do any further testing. In order to inspect the byte sequence I would save the output from a failed run to a text file and cat -v it:

[up-arrow key][CTRL-a]echo -n '[CTRL-e]' > /Volumes/Mojave/regex.txt
cat -v /Volumes/Mojave/regex.txt

This should show some non-printing characters that made their way into the regex. You could copy the output outside the virtual machine and inspect it with xxd or some other binary file parser, that would definitely show any non-printing characters that

I couldn't reproduce your issue exactly, but by hitting the shift key while the script was printing I could alter the letter case in the regex, which breaks the script a little differently.

[kathywh]

You're welcome. I ran the prepare_the_installer_app stage several times, and the RE error happened every time. I ran the command you suggested in the previous post, as well as the following command, to find non-printing characters, but none was found by both commands: [up-arrow key][CTRL-a]echo -n '[CTRL-e]' | cat -v

[myspaghetti]

the RE error happened every time

So it's not likely errant keystrokes. I still can't reproduce the error. If you don't mind further troubleshooting:

• Is your keyboard language set to English (United States) while running the script?
• What is the value of the LANG variable on the host bash shell? declare -p LANG
• What is the value of the LC_CTYPE variable on the guest bash shell? declare -p LC_CTYPE

[myspaghetti]

Could you attach Mojave_InstallInfo.plist from the host?

[kathywh]

• Yes, in Settings > Region & Language it reads "Language English (United States)", and I'm sure I selected "English, US" keyboard layout when installing Ubuntu.

• Output of declare -p LANG on host terminal is: declare -x LANG="en_US.UTF-8"

• Output of declare -p LC_CTYPE on guest terminal is: declare -x LC_CTYPE="UTF-8"

Mojave_InstallInfo.plist.tar.gz

[myspaghetti]

Thank you. The reason sed fails is because Mojave_InstallInfo.plist is a binary file instead of a plaintext file. I'll look into why that is.

[myspaghetti]

Apparently wget is receiving the file with gzip compression.

[myspaghetti]

@kathywh please check if ~/.wgetrc is configured with header = Accept-Encoding: gzip,deflate or some other configuration related to gzip.

[kathywh]

There is no ~/.wgetrc, and there is /etc/wgetrc. But no header settings related to gzip in /etc/wgetrc. The lines with header settings are:

# header with your request (so that server administrators can contact
#header = From: Your Name <username@site.domain>
# You can set up other headers, like Accept-Language.  Accept-Language
#header = Accept-Language: en

I think they have no connection with compression.

I attach my /etc/wgetrc here, and you can check if some incorrect settings in it: wgetrc.tar.gz

By the way, my wget version is 1.19.4, and the version info and compilation flags in wget --version output is:

GNU Wget 1.19.4 built on linux-gnu.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls 
+ntlm +opie +psl +ssl/openssl 

Wgetrc: 
    /etc/wgetrc (system)
Locale: 
    /usr/share/locale 
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc" 
    -DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib 
    -I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -DHAVE_LIBSSL -DNDEBUG 
    -g -O2 -fdebug-prefix-map=/build/wget-Xb5Z7Y/wget-1.19.4=. 
    -fstack-protector-strong -Wformat -Werror=format-security 
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall 
Link: 
    gcc -DHAVE_LIBSSL -DNDEBUG -g -O2 
    -fdebug-prefix-map=/build/wget-Xb5Z7Y/wget-1.19.4=. 
    -fstack-protector-strong -Wformat -Werror=format-security 
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall -Wl,-Bsymbolic-functions 
    -Wl,-z,relro -Wl,-z,now -lpcre -luuid -lidn2 -lssl -lcrypto -lpsl 
    ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a 

[myspaghetti]

@kathywh Thank you for your detailed bug report. Please download the following file using different software:

http://swcdn.apple.com/content/downloads/01/36/061-06468/agwdmzh99wgrg3ow1d1xy6gwra6qm7u1s0/InstallInfo.plist

For example, open the URL in your browser, download it with wget, curl, or any other downloader you have available. The file should be an xml/plaintext file. If the file is gzipped the issue is not with the script.

[kathywh]

I downloaded the file using browser, wget and curl many times, and I got plain text file some times while compressed file some times. It's confusing.

[myspaghetti]

If the file is intermittently gzipped on other devices on your network, it may be a firewall or router configuration issue. If it only occurs on one device, it may be an issue with the device's network configuration.

Closing as not likely an issue with the script; however this can be solved with checksum verification for the downloads which I have not implemented yet.