Not extracting files for Surface Laptop 1st gen

bekanui commented 9 months ago

The script is not extracting the files in the fwupdates folder, but generate a tmp.* folder with a subdirectory SurfaceUpdate containing this list of files and directory:

8897bt                SerialIOSPI                                SurfacePen                    graphicsih
8897wlan              SerialIOUART                               SurfacePen217FWUpdate         hdxsstm
AudioDetectionDriver  SurfaceACPIBattery                         SurfacePen217Integration      itouch
AudioSST              SurfaceAccessoryBluetoothPairing           SurfacePenClickFilter         managementengine
AudioSSTBus           SurfaceAccessoryFwUpdate                   SurfacePenIntegration         managementengineext
Chipset               SurfaceAcpiNotify                          SurfacePenPairing             realtekext
DPTF                  SurfaceButton                              SurfacePowerMeter             realtekhsa
DPTFCPU               SurfaceCIF                                 SurfaceSerialHub              serialioi2c
DPTFEsifMgr           SurfaceDigitizerIntegration                SurfaceStorageFwUpdate        surfaceals290xx
DPTFPCH               SurfaceDisplay                             SurfaceSystemTelemetry        surfacecfuoverhid
HIDPCIMiniPort        SurfaceDisplayColor                        SurfaceTouch                  surfacedock2fwupdate
ISHBus                SurfaceDockFwUpdate                        SurfaceTouchServicingML       surfaceinstalleula.rtf
ISHHECI               SurfaceDockIntegration                     camera                        surfacekip
KBLChipset            SurfaceHidMini                             cameracsi2host                surfaceme
ReadMe.txt            SurfaceIntegration                         cameractllogic                surfacesam
SPChipset             SurfaceKeyboardBluetoothFingerprintSensor  camerasensor7251              surfaceservice
SPChipsetHost         SurfaceKeyboardFingerprintIntegration      camerasensor9734              surfacetouchservicingmlextn
SPChipsetThermal      SurfaceKeyboardFingerprintSensor           camerasupport                 surfacetrackpad
SerialIOGPIO          SurfaceNVMExpressController                capabilitylicensingsvcclient  surfaceuefi

The file I'm trying to extract is SurfaceLaptop_Win10_18362_21.101.11476.0.msi and the output of the command bash -x repack.sh ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi -o fwupdates is the following:

+ set -euo pipefail
+ declare -g FILE=
+ declare -g OUTPUT=fwupdates
+ CAB_ARRAY=()
+ declare -g CAB_ARRAY
++ getopt -o hf:o: --long help,input:,output: -- ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi
+ eval set -- ' -- '\''../SurfaceLaptop_Win10_18362_21.101.11476.0.msi'\'''
++ set -- -- ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi
+ true
+ case "${1}" in
+ shift
+ break
+ '[' '' = '' ']'
+ '[' 1 -gt 0 ']'
+ FILE=../SurfaceLaptop_Win10_18362_21.101.11476.0.msi
+ shift
+ '[' 0 -gt 0 ']'
+ '[' 0 -gt 0 ']'
+ '[' ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi = '' ']'
+ '[' fwupdates = '' ']'
+ for c in msiextract gcab dos2unix
+ command -v msiextract
+ for c in msiextract gcab dos2unix
+ command -v gcab
+ for c in msiextract gcab dos2unix
+ command -v dos2unix
+ main
+ mkdir -p fwupdates
+ case "${FILE}" in
+ repackmsi ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi fwupdates
+ local MSI=../SurfaceLaptop_Win10_18362_21.101.11476.0.msi
+ local OUT=fwupdates
+ echo '==> Extracting ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi'
==> Extracting ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi
++ mktemp -p . -d
+ TEMP=./tmp.n55WVlTNAS
+ msiextract -C ./tmp.n55WVlTNAS ../SurfaceLaptop_Win10_18362_21.101.11476.0.msi
+ repackdir ./tmp.n55WVlTNAS fwupdates
+ local DIR=./tmp.n55WVlTNAS
+ local OUT=fwupdates
+ find . -iname '*.inf' -execdir dos2unix --quiet '{}' +

StollD commented 9 months ago

For me the script works as intended with the file you named, so I don't really know what could be done.

You could try extracting the MSI file yourself with msiextract and then pass that directory to the repack script instead of the MSI file.

bekanui commented 9 months ago

Thanks for the reply, after some tests I managed to generate the .cab files by running the script two times. The first time as is, the second one with this line:

196.    find . -iname '*.inf' -execdir dos2unix --quiet {} +

commented. It seems that the script was exiting after that command without repacking the files.

However, I'm not sure if all the .cab files are created, since by comparing the list in the README from what I get it looks like some are missing:

SurfaceTouch_189.5.7688_e359e1e1-f6ce-457a-a8c7-b296f2a8e061.cab
surfacesam_145.2.23435_53d1ef63-84c2-4fa2-bb99-b5872bde6d36.cab
surfacekip_33.226.3328_597868df-2e20-4c37-9549-228077d2dcb8.cab
surfacetrackpad_0.39.43643_38c67e06-e8ee-4acd-b031-35af2c95fef6.cab
surfaceme_184.86.3877_9d637e9a-86cc-400a-ad6c-c99f4518af87.cab
surfaceuefi_34.186.21248_a1575cb8-5184-49ab-a932-9c2146b68ded.cab

I know the two firmwares aren't the same (mine is for the Surface Laptop, while the one in the example is for the Surface Book), but I also tried this with the Surface Book 2 one and I still can't get the full list. Which is unfortunate, since the update I'm more interested in is the keyboard one.

If it helps, this is the output of the command find . -iname '*.inf' -execdir dos2unix '{}' + (without the --quiet flag):

dos2unix: converting UTF-16LE file ./ov9734.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./iacamera64.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./ov7251.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./iaisp64.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./iactrllogic64.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./CSI2HostControllerDriver.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceDock2FwUpdate.inf to Unix format...
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./IntcOED.inf, an UTF-16 conversion error occurred on line 637.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./IntcAudioBus.inf, an UTF-16 conversion error occurred on line 132.
dos2unix: converting file ./SurfaceTouchServicingML.inf to Unix format...
dos2unix: converting UTF-16LE file ./SurfaceTrackpad.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceSerialHubDriver.inf to Unix format...
dos2unix: converting file ./HDXSSTM.inf to Unix format...
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./IntcDAud.inf, an UTF-16 conversion error occurred on line 3733.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./IntcDAud.inf, an UTF-16 conversion error occurred on line 3735.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./64jp8682.inf, an UTF-16 conversion error occurred on line 2302.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./64ih8682.inf, an UTF-16 conversion error occurred on line 2301.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./64on8682.inf, an UTF-16 conversion error occurred on line 2300.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./64kb8682.inf, an UTF-16 conversion error occurred on line 2303.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./64gh8682.inf, an UTF-16 conversion error occurred on line 2302.
dos2unix: Invalid or incomplete multibyte or wide character
dos2unix: Skipping UTF-16 file ./64zh8682.inf, an UTF-16 conversion error occurred on line 2300.
dos2unix: converting UTF-16LE file ./HdBusExt.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./dptf_acpi.inf to Unix format...
dos2unix: converting file ./dptf_cpu.inf to Unix format...
dos2unix: converting file ./SurfaceTouchServicingMLExtn.inf to Unix format...
dos2unix: converting file ./SunrisePoint-LPSystemThermal.inf to Unix format...
dos2unix: converting file ./SunrisePoint-LPSystem.inf to Unix format...
dos2unix: converting file ./SurfaceIntegrationDriver.inf to Unix format...
dos2unix: converting file ./SurfaceME.inf to Unix format...
dos2unix: converting file ./iclsClient.inf to Unix format...
dos2unix: converting file ./SurfaceAcpiNotifyDriver.inf to Unix format...
dos2unix: converting file ./SurfaceUEFI.inf to Unix format...
dos2unix: converting file ./dptf_pch.inf to Unix format...
dos2unix: converting file ./surfacePenClickFilter.inf to Unix format...
dos2unix: converting file ./SurfaceDisplayColor.inf to Unix format...
dos2unix: converting file ./SurfacePen217FwUpdate.inf to Unix format...
dos2unix: converting file ./mrvlpcie8897.inf to Unix format...
dos2unix: converting UTF-16LE file ./iaLPSS2_I2C_SKL.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./iaLPSS2_GPIO2_SKL.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./iaLPSS2_UART2_SKL.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./ISH.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceServiceNullDriver.inf to Unix format...
dos2unix: converting file ./esif_manager.inf to Unix format...
dos2unix: converting file ./SurfacePen217Integration.inf to Unix format...
dos2unix: converting file ./heci.inf to Unix format...
dos2unix: converting UTF-16LE file ./DetectionVerificationDrv.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceCIF.inf to Unix format...
dos2unix: converting UTF-16LE file ./SurfaceAls290XXDriver.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./SurfaceKIP.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./KeyboardBluetoothFingerprintSensor.inf to Unix format...
dos2unix: converting file ./SurfaceStorageFwUpdate.inf to Unix format...
dos2unix: converting file ./keyboardfingerprintsensor.inf to Unix format...
dos2unix: converting UTF-16LE file ./mbtr8897w81x64.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceNVMExpressController.inf to Unix format...
dos2unix: converting file ./AccessoryBluetoothPairing.inf to Unix format...
dos2unix: converting file ./SurfaceAccessoryFwUpdate.inf to Unix format...
dos2unix: converting file ./SurfaceDockIntegration.inf to Unix format...
dos2unix: converting file ./SurfacePenPairing.inf to Unix format...
dos2unix: converting UTF-16LE file ./iaLPSS2_SPI_SKL.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./OemExtension.inf to Unix format...
dos2unix: converting file ./SurfaceHidMiniDriver.inf to Unix format...
dos2unix: converting file ./KabylakeSystem.inf to Unix format...
dos2unix: converting file ./RealtekExt.inf to Unix format...
dos2unix: converting UTF-16LE file ./ISH_BusDriver.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceDisplay.inf to Unix format...
dos2unix: converting UTF-16LE file ./RealtekHSA.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfaceDigitizerIntegration.inf to Unix format...
dos2unix: converting file ./SurfacePenDriver.inf to Unix format...
dos2unix: converting UTF-16LE file ./HID_PCI.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting UTF-16LE file ./SurfaceTouch.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./sunrisepoint-lpSDHost.inf to Unix format...
dos2unix: converting UTF-16LE file ./SurfaceCFUOverHid.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./keyboardfingerprintintegration.inf to Unix format...
dos2unix: converting file ./SurfaceDockFwUpdate.inf to Unix format...
dos2unix: converting file ./SurfaceButton.inf to Unix format...
dos2unix: converting file ./SurfaceSystemTelemetryDriver.inf to Unix format...
dos2unix: converting file ./SurfacePenIntegration.inf to Unix format...
dos2unix: converting UTF-16LE file ./SurfaceSAM.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./SurfacePowerMeter.inf to Unix format...
dos2unix: converting file ./SurfaceACPIBattery.inf to Unix format...
dos2unix: converting UTF-16LE file ./iaPreciseTouch.inf to ANSI_X3.4-1968 Unix format...
dos2unix: converting file ./commonsystem.inf to Unix format...

ryankenney-dev commented 8 months ago

I'm having the same issue with the latest firmware for the Surface Go 3 (SurfaceGo3_Win11_22000_23.110.4352.0.msi_from-cosmos).

The issue appears to be that:

set -e at the top of the script terminates everything on any command failure (non-zero exit).. which is good.
find ... + fails if any child exec fails, and it's failing on certain files, such as: ./64gh8682.inf

The end of the story is that certain .inf files are failing dos2unix using default options:

dos2unix: Skipping UTF-16 file ./64gh8682.inf, an UTF-16 conversion error occurred on line 2302.

I have no idea how to figure out what the proper code-page/encoding arguments to dos2unix might be, but I'm a little terrified that if I guess wrong, and repack.sh succeeds, I'll have a bricked device.

ryankenney-dev commented 8 months ago

Side note: an echo "SUCCESS!" message at the end of the script would help the user know it didn't terminate with an error.

ryankenney-dev commented 8 months ago

I loaded a few files in vim and the toggled the encoding used to view the file (via :e ++enc=...). Here's what I observed:

Some of the .inf files that did not fail the conversion are viewable as utf-8 (but not utf-16 or utf-16le)
Some of the .inf files that did fail the conversion are viewable as utf-16le (but not utf-8 or utf-16)

It makes intuitive sense that utf-8, utf-16, utf-16le would be mutually exclusive.

As an experiment, I looked at the output of file --mime-encoding <inf-file> and just applied --assume-utf16le to dos2unix when it reported utf-16le. This ultimate failed on at least one file. So much for trivial automatic detection.

StollD commented 8 months ago

I pushed a change that makes it ignore errors from dos2unix, since most of the .inf files inside the MSI are just normal Windows drivers and not firmware.

I am not sure if there is a better way to detect the folders with firmware than by grepping their .inf files. Maybe we can detect the capsule updates themselves somehow? And then only process the .inf files from that directory.

Side note: an echo "SUCCESS!" message at the end of the script would help the user know it didn't terminate with an error.

This already exists: https://github.com/linux-surface/surface-uefi-firmware/blob/master/repack.sh#L105-L112

I am looking into ways to make it print an error statement when it fails instead of just exiting.

linux-surface / surface-uefi-firmware

Not extracting files for Surface Laptop 1st gen #35