Stuck trying to build WinPE environment

erikng commented 5 years ago

I've done a tremendous amount of debugging and reading the google groups/reddit and various other places to try and piecemeal how to get glazier working, but I am now stuck and can't figure out why things aren't working:

At this point I have python able to run, but when it tries to run autobuild.py I get a python trace complaining about a DLL load failed

  File "x:\python\lib\lib-tk\FixTk.py", line 68, in <module>
    import _tinker
ImportError: DLL load failed: The specified module could not be found.

I am running 32-bit WinPE v10.0.14393.0, Python 2.7.16 32-bit with VC++ Redistributables.

Here are my current notes as to what I've done to get to this point (I think the most complete public information on how to do this). Any help would be immensely appreciated.

How to debug potential python issues in WinPE:
start cmd.exe
sxstrace trace -logfile:trace.etl

run python.exe in original window and produce error

stop the sxstrace process
sxstrace parse -logfile:trace.etl -outfile:trace.txt
more trace.txt

32-Bit instructions:

Step 1: download Windows ADK: https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install

Step 2: Install WinPE addon (if running Win 10 1809+) https://go.microsoft.com/fwlink/?linkid=2022233

Step 3: Run Deployment and Imaging Tools Environment (as administrator)

Step 4: Create a 32-bit PE folder:
copype x86 C:\WinPE_x86

Step 5: Mount the wim file:
Dism /Mount-Image /ImageFile:"C:\WinPE_x86\media\sources\boot.wim" /Index:1 /MountDir:"C:\WinPE_x86\mount"

Step 6: Get "shutdown.exe" from C:\Windows\System32 and copy it to C:\WinPE_x86\mount\Windows\System32

Step 7: Add the WMI package functionality with DISM
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-WMI.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-WMI_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-NetFX.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-NetFX_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-Scripting.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-Scripting_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-PowerShell.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-PowerShell_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-StorageWMI.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-StorageWMI_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-DismCmdlets.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-DismCmdlets_en-us.cab"

Step 8: Create a "src" folder and copy the glazier folder from the glazier code repo
C:\WinPE_x86\mount\src
C:\WinPE_x86\mount\src\glazier

Step 9: Download official python 2.7 MSI and install it, pointing installer to your PE environment
C:\WinPE_x86\mount\python

Step 10: using pip and install required packages
pip install absl-py
pip install PyYAML
pip install pyfakefs
pip install mock

Step 11: Download gwimpy from https://github.com/google/winops and install it manually into glazier folder
C:\WinPE_x86\mount\src\glazier\gwimpy

Step 12: Extract the Visual Studio C++ 2008 Redistributable package: http://download.microsoft.com/download/1/1/1/1116b75a-9ec3-481a-a3c8-1777b5381140/vcredist_x86.exe
vcredist_x86.exe /extract (GUI will popup to specify a folder)
msiexec /a C:\Users\someone\Downloads\vcred\vc_red.msi /qb TARGETDIR="C:\Users\someone\Downloads\vcred\msiextract"

Step 13: Copy the respective dll and manifest file to the python directory
C:\Users\someone\Downloads\vcred\msiextract\Windows\winsxs\ulCRTx86\x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_x-ww_d08d0375.manifest
C:\Users\someone\Downloads\vcred\msiextract\Windows\winsxs\ulCRTx86\msvcr90.dll

Step 14: Rename both of these files to "Microsoft.VC90.CRT"
C:\WinPE_x86\mount\python\Microsoft.VC90.CRT.dll
C:\WinPE_x86\mount\python\MSVCR90.dll (clone of the above file)
C:\WinPE_x86\mount\python\Microsoft.VC90.CRT.manifest

Step 15: Copy  the VC90 manifest/dll to C:\WinPE_x86\mount\python\DLLs for tkinker to work
Unknown why it is looking for the DLLs here as well....
C:\WinPE_x86\mount\python\DLLs\Microsoft.VC90.CRT.dll
C:\WinPE_x86\mount\python\DLLs\Microsoft.VC90.CRT.manifest

Step 16: Copy the python27.dll into the python folder
32-bit windows: C:\Windows\system32\python27.dll
64-bit windows: C:\Windows\sysWOW64\python27.dll
C:\WinPE_x86\mount\python\python27.dl

Step 17: Create a "resources" folder and install the windowsZones.xml file from https://github.com/unicode-org/cldr/blob/master/common/supplemental/windowsZones.xml and logo.gif
C:\WinPE_x86\mount\resources
C:\WinPE_x86\mount\resources\windowsZones.xml
C:\WinPE_x86\mount\resources\logo.gif (need to provide your own, unknown on size and resolution)

Step 18: Modify the startnet.cmd file with the contents from the Glazier readme (Requires a text editor with administator access)
C:\WinPE_x86\mount\Windows\System32\startnet.cmd

Step 19: Update the constants.py file for your environment
C:\WinPE_x86\mount\src\glazier\lib\constants.py

Step 20: commit the image
Dism /Unmount-Image /MountDir:C:\WinPE_x86\mount /Commit

Step 21: Create ISO to test in a VM
MakeWinPEMedia /ISO C:\WinPE_x86 C:\WinPE_x86\WinPE_x86.iso

erikng commented 5 years ago

Here is the full trace, though I don't think it will help too much

erikng commented 5 years ago

Alright so small update, but I have recreated this issue identically with an amd64 winPE environment, using the latest python 2.7.16 64-bit MSI and the 64-bit VC++ distributable.

Attached is my updated instructions with both 32-bit and 64-bit WinPE/Python/VC++.

I was thinking maybe it had to do with 32-bit environment, but apparently not 😢

How to debug potential python issues in WinPE:
start cmd.exe
sxstrace trace -logfile:trace.etl

run python.exe in original window and produce error

stop the sxstrace process
sxstrace parse -logfile:trace.etl -outfile:trace.txt
more trace.txt

32-Bit/64-Bit instructions:

Step 1: download Windows ADK: https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install

Step 2: Install WinPE addon (if running Win 10 1809+) https://go.microsoft.com/fwlink/?linkid=2022233

Step 3: Run Deployment and Imaging Tools Environment (as administrator)

Step 4: Create a PE folder:
copype x86 C:\WinPE_x86
copype amd64 C:\WinPE_amd64

Step 5: Mount the wim file:
Dism /Mount-Image /ImageFile:"C:\WinPE_x86\media\sources\boot.wim" /Index:1 /MountDir:"C:\WinPE_x86\mount"
Dism /Mount-Image /ImageFile:"C:\WinPE_amd64\media\sources\boot.wim" /Index:1 /MountDir:"C:\WinPE_amd64\mount"

Step 6: Get "shutdown.exe" from C:\Windows\System32 and copy it to C:\WinPE_x86\mount\Windows\System32

C:\WinPE_amd64\mount\Windows\System32

Step 7: Add the WMI package functionality with DISM
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-WMI.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-WMI_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-NetFX.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-NetFX_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-Scripting.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-Scripting_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-PowerShell.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-PowerShell_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-StorageWMI.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-StorageWMI_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\WinPE-DismCmdlets.cab"
Dism /Add-Package /Image:"C:\WinPE_x86\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\x86\WinPE_OCs\en-us\WinPE-DismCmdlets_en-us.cab"

Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-WMI.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\en-us\WinPE-WMI_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-NetFX.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\en-us\WinPE-NetFX_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-Scripting.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\en-us\WinPE-Scripting_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-PowerShell.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\en-us\WinPE-PowerShell_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-StorageWMI.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\en-us\WinPE-StorageWMI_en-us.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-DismCmdlets.cab"
Dism /Add-Package /Image:"C:\WinPE_amd64\mount" /PackagePath:"C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Windows Preinstallation Environment\amd64\WinPE_OCs\en-us\WinPE-DismCmdlets_en-us.cab"

Step 8: Create a "src" folder and copy the glazier folder from the glazier code repo
C:\WinPE_x86\mount\src
C:\WinPE_x86\mount\src\glazier

C:\WinPE_amd64\mount\src
C:\WinPE_amd64\mount\src\glazier

Step 9: Download official python 2.7 MSI and install it, pointing installer to your PE environment
C:\WinPE_x86\mount\python

C:\WinPE_amd64\mount\python

Step 10: using pip and install required packages
pip install absl-py
pip install PyYAML
pip install pyfakefs
pip install mock

Step 11: Download gwimpy from https://github.com/google/winops and install it manually into glazier folder
C:\WinPE_x86\mount\src\glazier\gwimpy

C:\WinPE_amd64\mount\src\glazier\gwimpy

Step 12: Extract the Visual Studio C++ 2008 Redistributable package: http://download.microsoft.com/download/1/1/1/1116b75a-9ec3-481a-a3c8-1777b5381140/vcredist_x86.exe (32-bit)
https://www.microsoft.com/en-us/download/details.aspx?id=2092 (64-bit)
vcredist_x86.exe /extract (GUI will popup to specify a folder)
msiexec /a C:\Users\someone\Downloads\vcred\vc_red.msi /qb TARGETDIR="C:\Users\someone\Downloads\vcred\msiextract"

vcredist_x64.exe /extract (GUI will popup to specify a folder)
msiexec /a C:\Users\someone\Downloads\vcred_64\vc_red.msi /qb TARGETDIR="C:\Users\someone\Downloads\vcred_64\msiextract"

Step 13: Copy the respective dll and manifest file to the python directory
C:\Users\someone\Downloads\vcred\msiextract\Windows\winsxs\ulCRTx86\x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_x-ww_d08d0375.manifest
C:\Users\someone\Downloads\vcred\msiextract\Windows\winsxs\ulCRTx86\msvcr90.dll

C:\Users\someone\Downloads\vcred_64\msiextract\Windows\winsxs\ulCRTx64\amd64_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_x-ww_0296e955.manifest
C:\Users\someone\Downloads\vcred_64\msiextract\Windows\winsxs\ulCRTx64\msvcr90.dll

Step 14: Rename both of these files to "Microsoft.VC90.CRT"
C:\WinPE_x86\mount\python\Microsoft.VC90.CRT.dll
C:\WinPE_x86\mount\python\MSVCR90.dll (clone of the above file)
C:\WinPE_x86\mount\python\Microsoft.VC90.CRT.manifest

C:\WinPE_amd64\mount\python\Microsoft.VC90.CRT.dll
C:\WinPE_amd64\mount\python\msvcr90.dll (clone of the above file)
C:\WinPE_amd64\mount\python\Microsoft.VC90.CRT.manifest

Step 15: Copy  the VC90 manifest/dll to C:\WinPE_x86\mount\python\DLLs for tkinker to work
Unknown why it is looking for the DLLs here as well....
C:\WinPE_x86\mount\python\DLLs\Microsoft.VC90.CRT.dll
C:\WinPE_x86\mount\python\DLLs\Microsoft.VC90.CRT.manifest

C:\WinPE_amd64\mount\python\DLLs\Microsoft.VC90.CRT.dll
C:\WinPE_amd64\mount\python\DLLs\Microsoft.VC90.CRT.manifest

Step 16: Copy the python27.dll into the python folder
32-bit windows: C:\Windows\system32\python27.dll
64-bit windows: same path but different DLL/size
C:\WinPE_x86\mount\python\python27.dll

C:\WinPE_amd64\mount\python\python27.dll

Step 17: Create a "resources" folder and install the windowsZones.xml file from https://github.com/unicode-org/cldr/blob/master/common/supplemental/windowsZones.xml and logo.gif
C:\WinPE_x86\mount\resources
C:\WinPE_x86\mount\resources\windowsZones.xml
C:\WinPE_x86\mount\resources\logo.gif (need to provide your own, unknown on size and resolution)

C:\WinPE_amd64\mount\resources
C:\WinPE_amd64\mount\resources\windowsZones.xml
C:\WinPE_amd64\mount\resources\logo.gif (need to provide your own, unknown on size and resolution)

Step 18: Modify the startnet.cmd file with the contents from the Glazier readme (Requires a text editor with administator access)
C:\WinPE_x86\mount\Windows\System32\startnet.cmd

C:\WinPE_amd64\mount\Windows\System32\startnet.cmd

Step 19: Update the constants.py file for your environment
C:\WinPE_x86\mount\src\glazier\lib\constants.py

C:\WinPE_amd64\mount\src\glazier\lib\constants.py

Step 20: commit the image
Dism /Unmount-Image /MountDir:C:\WinPE_x86\mount /Commit

Dism /Unmount-Image /MountDir:C:\WinPE_amd64\mount /Commit

Step 21: Create ISO to test in a VM
MakeWinPEMedia /ISO C:\WinPE_x86 C:\WinPE_x86\WinPE_x86.iso

MakeWinPEMedia /ISO C:\WinPE_amd64 C:\WinPE_amd64\WinPE_amd64.iso

TsekNet commented 5 years ago

Acknowledged, we still have some work to do in terms of externalizing all of our modules. This will likely not get worked on in the short term, but we are aware of the issue.

erikng commented 5 years ago

Is there anything I can do to get glazier working then? It seems like this tool isn't ready for public consumption if the WinPE portions don't work.

TsekNet commented 5 years ago

It looks like an issue with FixTk.py according to your screenshot. Something is messed up with the Tkinter installation.

Is that machine 64 bit or 32 bit? Does the command python -m Tkinter print the same error message?

You may be able to un-install Tkinter & re-install it, using the package found here, but you might need to un-install & re-install Python itself, as mentioned in the answers to ImportError DLL load failed importing _tkinter.

As a side note, we're working internally to make getting started with Glazier easier, but as noted in our disclaimer, we're a small team internally and will get to this as soon as we can.

TsekNet commented 4 years ago

Closing this issue as it's been open for months now. Please feel free to reopen this issue or open a new one if the problem persists!

erikng commented 4 years ago

Did you ever update the getting started? I moved onto other projects but could recheck. My gut says it's still an issue.

TsekNet commented 4 years ago

Hi Erik!

We expect to release a Go binary in the next few quarters that will include additional getting started docs alongside that release.

That will likely not fix the FixTk issue though. I'm happy to help you troubleshoot this issue if it's still a blocker for you. Did you have a chance to try https://github.com/google/glazier/issues/4#issuecomment-519510668?

google / glazier

Stuck trying to build WinPE environment #4