Open Jbranadev opened 3 months ago
@ntrappe @rameshr-msft @ @grcusanz Do you have any idea what might be happening?
@Jbranadev Hi! Could you please provide some additional information so we can look into this? We need the following:
[^1]: Note that long term support for Windows Server Core LTSC 2019
has ended.
@ntrappe-msft Hello, sorry for the delay in my response, I was trying other schemes, looking to solve the problem I have, currently I have tried with mcr.microsoft.com/windows:20H2, mcr.microsoft.com/windows:ltsc2019 and mcr.microsoft.com /windows:2004, the latter is the one that gave me the fewest inconveniences and differences in terms of loading the dynamic objects that my executable needs.
The configuration that I share with you is so that you can test it on your Host machine and in a Windows container and thus you can appreciate the inconvenience that I report.
Once you have the repository on your local machine, you can do the following:
Assuming that your Host machine, on which you will test, does not have any previous configuration, there is only a need for you to copy the folders that are inside the DiscoC folder to your C:\ drive
Then run the following scripts in administrator mode, make sure you have winget installed on your computer
winget install --source winget --exact --id Microsoft.WindowsSDK.10.0.22000 --force --accept-package-agreements --accept-source-agreements
cd CrossWalkBase
.\\features.ps1
cd C:\app
.\\initCrosswalk.ps1
Then you can run the XW Client Simulator application, which is located in the Frame Simulator folder, in HOST the IP of your Host machine must be, in Port 1966 and in terminal it would be your IP in hexadecimal, and you enter the following data
xxx.xx.xx.xxx 1966 AC1C0F08 000090009990WR9088493200000SGBSCT00182803MAY2405044 9990884932
In this application you can click on connect, then on disconnect and then on send message and then on disconnect to test the application that interests us. on your host machine will produce a result like this and it is what we hope to be able to achieve in the container.
cd winget
docker build --force-rm --no-cache -f dockerfileWinget2 -t windowswinget:2 .
cd ../installWinget
docker build --force-rm --no-cache -f dockerFileInstallWinget2 -t wingetinstall:2 .
cd ../CrossWalkBase
docker build --force-rm --no-cache -f dockerfileBaseWindows2 -t crosswalkbase:2 .
cd ../Seguridad
docker build --force-rm --no-cache -f dockerfile3 -t seguridadbanca:2 .
docker run -d -p 1967:1967 -p 1966:1966 -p 1968:1968 -p 5600:5600 --name seguridadbanca -e DBI_PORT=5600 seguridadbanca:2
docker exec -it xxxxxxxxxxx powershell
# Ver el contenido del archivo DBILOG260424_5600.txt
cd C:\CrossWalk\Bin\Log
ls
Get-Content "C:\CrossWalk\Bin\Log\20240522_HUB_(0)_Log.txt"
modifies the name of the file, since it creates a different log per day
20240522HUB(0)_Log.txt
assuming that if you try it on May 25 the name would be more or less like this 20240525HUB(0)_Log.txt
So, you can now click on connect, then on disconnect and then on send message and then disconnect, in the application that we use to test and then you run GetConten again, you will see a result, where the characters you are receiving are not recognizable, something very contrary to what happens on the host.
Then you can stop the container and execute the following commands, in order to compare the logs and folders with respect to the host
docker cp xxxxxxxxxxx:C:\CrossWalk C:\dockervolumenseguridad
docker cp xxxxxxxxxxx:C:\app C:\dockervolumenseguridad
docker cp xxxxxxxxxxx:C:\inetpub C:\dockervolumenseguridad
docker cp xxxxxxxxxxx:C:\Windows\SysWOW64 C:\dockervolumenseguridad\SysWOW6420042
On your Host machine, if you want to test the application that we are containerizing, you only need to run
cd C:\app
.\\generate_ini_hub.ps1
Then you can use the client application, to test and click on connect, disconnect and send message and then disconnect, it will produce a result like this, here what is being sent if it is received as it should be received
You can use the Beyond Compare application, which allows you to compare files and directories, as if it were a git diff show, you open the application and you can compare the logs and files of your host machine against those of the container and thus you will see that the only difference between The container and the host are the system files used by the image and those used by the container, which is what prevents some dlls found in SYSWOW64 from being loaded.
Here we compare the SysWow64 folders, the one of the host against that of the container, we see that the kernelbase of the container weighs less than that of the HOST, in this case we are comparing the one in the image mcr.microsoft.com /windows:2004, this is the one that The closer it is to the Host and the one that gives the least errors
Here we see how some modules are loaded in the Host that are not loaded in the container and are those that in a certain way are necessary so that other dlls are loaded correctly when the application starts
Here we can see the DLLs that are loaded on the host, but in the container. No, it is worth mentioning that the dlls that are not loaded do exist in the docker image.
I've already tried many things, including replacing the Syswow64 dlls in the docker image with the ones on my host, but when I do that, even when it only replaces one dll, like kernelbase.dll, it turns out that no application starts anymore. in the container image, then, it already rules out that option, therefore, I only have to compare other images against my host, see which of their dll's have a weight most similar to those of my host, to see if they load them all. . the necessary modules, I have that theory, because with ltsc2019 there were many more kernelbase.dll, ntdll.dll modules that were not loaded, then with 20H2, they decreased a little, with 2004 it is already a very considerable reduction, but all those modules that technically provides the operating system not yet dynamically loaded,
Below is a detail of my research, if you need a meeting or support to replicate my problem, I am available.
Dll to compare that of the host with that of the container
Host C:\Windows\SysWOW64\kernelbase.dll Container "C:\Windows\SysWOW64\KERNELBASE.dll"
Host C:\Windows\SysWOW64\ntdll.dll Container "C:\Windows\SysWOW64\ntdll.dll"
DLLs that only exist in one place and not both, (Container and host)
Container C:\Windows\SysWOW64\SspiCli.dll C:\Windows\SysWOW64\MSASN1.dll C:\Windows\SysWOW64\WDSCORE.dll
Host C:\Windows\SysWOW64\UMPDC.dll
They are not loaded on the host, but in the container
ModLoad: 0000000074890000 00000000
7489a000 C:\Windows\SysWOW64\CRYPTBASE.dll
ModLoad: 0000000074890000 00000000
7489a000 C:\Windows\SysWOW64\CRYPTBASE.dll
ModLoad: 00000000748e0000 00000000
748f9000 C:\Windows\SysWOW64\bcrypt.dll
ModLoad: 00000000748e0000 00000000
748f9000 C:\Windows\SysWOW64\bcrypt.dll
are uploaded to the host, but not to the container
ModLoad: 75910000 75935000 C:\Windows\SysWOW64\IMM32.DLL ModLoad: 75910000 75935000 C:\Windows\SysWOW64\IMM32.DLL ModLoad: 740a0000 74125000 C:\Program Files (x86)\Common Files\McAfee\SystemCore\mfehcinj.dll ModLoad: 740a0000 74125000 C:\Program Files (x86)\Common Files\McAfee\SystemCore\mfehcinj.dll ModLoad: 755c0000 7560c000 C:\Windows\SysWOW64\WINTRUST.dll ModLoad: 755c0000 7560c000 C:\Windows\SysWOW64\WINTRUST.dll
ModLoad: 74090000 7409c000 C:\Program Files (x86)\Common Files\McAfee\SystemCore\mfehcthe.dll ModLoad: 74090000 7409c000 C:\Program Files (x86)\Common Files\McAfee\SystemCore\mfehcthe.dll ModLoad: 74040000 74067000 C:\Program Files\McAfee\Endpoint Security\Adaptive Threat Protection\mfedeeprem32.dll ModLoad: 74040000 74067000 C:\Program Files\McAfee\Endpoint Security\Adaptive Threat Protection\mfedeeprem32.dll ModLoad: 73920000 73978000 C:\Program Files\McAfee\MAR\mvcairo_x86.dll ModLoad: 73920000 73978000 C:\Program Files\McAfee\MAR\mvcairo_x86.dll
ModLoad: 73780000 737f4000 C:\Windows\SysWOW64\uxtheme.dll ModLoad: 73780000 737f4000 C:\Windows\SysWOW64\uxtheme.dll
ModLoad: 77470000 77544000 C:\Windows\SysWOW64\MSCTF.dll ModLoad: 77470000 77544000 C:\Windows\SysWOW64\MSCTF.dll
ModLoad: 6a430000 6a640000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.19041.1110_none_a8625c1886757984\comctl32.DLL ModLoad: 6a430000 6a640000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.19041.1110_none_a8625c1886757984\comctl32.DLL
ModLoad: 773e0000 77467000 C:\Windows\SysWOW64\SHCORE.dll ModLoad: 773e0000 77467000 C:\Windows\SysWOW64\SHCORE.dll
ModLoad: 74850000 7486e000 C:\Windows\SysWOW64\SRVCLI.DLL ModLoad: 74850000 7486e000 C:\Windows\SysWOW64\SRVCLI.DLL
ModLoad: 00000000672d0000 00000000
6738a000 C:\Windows\SysWOW64\textinputframework.dll
ModLoad: 00000000672d0000 00000000
6738a000 C:\Windows\SysWOW64\textinputframework.dll
ModLoad: 0000000066fb0000 00000000
6722e000 C:\Windows\SysWOW64\CoreUIComponents.dll
ModLoad: 0000000066fb0000 00000000
6722e000 C:\Windows\SysWOW64\CoreUIComponents.dll
ModLoad: 0000000067230000 00000000
672cb000 C:\Windows\SysWOW64\CoreMessaging.dll
ModLoad: 0000000067230000 00000000
672cb000 C:\Windows\SysWOW64\CoreMessaging.dll
ModLoad: 0000000074150000 00000000
74179000 C:\Windows\SysWOW64\ntmarta.dll
ModLoad: 0000000074150000 00000000
74179000 C:\Windows\SysWOW64\ntmarta.dll
ModLoad: 0000000066ed0000 00000000
66fac000 C:\Windows\SysWOW64\wintypes.dll
ModLoad: 0000000066ed0000 00000000
66fac000 C:\Windows\SysWOW64\wintypes.dll
ModLoad: 000000006daf0000 00000000
6db84000 C:\Windows\SysWOW64\TextShaping.dll
ModLoad: 000000006daf0000 00000000
6db84000 C:\Windows\SysWOW64\TextShaping.dll
@ntrappe-msft @deverouxmartin-msft @fady-azmy-msft @grcusanz Do you have any idea what might be happening?
@Jbranadev We'll try to get some attention on this Issue.
🔖 ADO 51748636 (Internal)
@Jbranadev - can you grant me access to the repository that you referenced above?
Ready, I have given you access to the repository, you just have to accept the invitation
If you need support, a meeting to better understand the problem, we are happy to be available.
@Jbranadev - thank you for sharing access to your repository. My understanding is that you have made a strong correlation between several missing DLLs in the container image and your application processing messages as ")))))))))))))))))))) ))))))))))))))))))))))))))))) ))))))))))" instead of the expected sequence of valid characters. I reviewed the contents of your repository, and it is a bit too complex to facilitate a focused investigation. Given that your investigation has confirmed that the application is unable to load a set of DLLs, we should create a more simplified test environment where we focus on the module load failures. Ideally, we should try to create a very simple test application that clearly demonstrates the missing DLLs. Although you have performed some comparisons between the host and the container in your description above, I do not see any mention that your application encountered any module failures. Forgive me if I missed anything, but did you observe any such failure?
I appreciate the support @deverouxmartin-msft
Note that I have identified the dlls that should have the api-ms-win-xxxxxxxxxxxx-xx-x-x.dll modules, if they exist, both on my host machine and in the container, the application starts correctly on the host and in the container , but it is not until it has to process the messages that the dissonance between the host and the container is seen and it is due to those modules that are not loaded in the container.
The repository and the comment in which I indicated how to replicate the error have all the instructions and complexity that you can currently view, because I wanted to exhaust all possible options before resorting to requesting support through this means, and with the aim that they can have all the information that I have previously had and not spend unnecessary effort on double research. What I already investigated on my side was to make that configuration that currently uses tools like Dependency.exe so that you can use its results and compare those of the host vs. container, as well as the windows sdk debugger, so that you also have access to that log.
The summary is that the dll's that provide the modules that the application needs exist both within the container and on the host, the only difference is that the ones that I mentioned at the beginning in the first comment of this issue are not loaded in the container, I don't know because it does not load those modules, if technically the container has the same configuration as the host machine.
@deverouxmartin-msft Do you want a simpler configuration in which you can replicate the problem within the container, without the logs that are currently generated?
@Jbranadev - thank you for the additional details. In your comparison between the loaded modules of the application running on the host and the loaded modules of the application running in the container, the key difference that you found is the following DLLs which do not get loaded in the container and you believe that this is related to the application processing the incoming messages as ")))))))))))))))))))) ))))))))))))))))))))))))))))) ))))))))))": 'C:\Windows\SysWOW64\IMM32.DLL', 'C:\Program Files (x86)\Common Files\McAfee\SystemCore\mfehcinj.dll', 'C:\Windows\SysWOW64\WINTRUST.dll', 'C:\Windows\SysWOW64\MSASN1.dll', 'C:\Program Files (x86)\Common Files\McAfee\SystemCore\mfehcthe.dll', 'C:\Program Files\McAfee\MAR\mvcairo_x86.dll', 'C:\Program Files\McAfee\Endpoint Security\Adaptive Threat Protection\mfedeeprem32.dll', 'C:\Windows\SysWOW64\uxtheme.dll', 'C:\Windows\SysWOW64\MSCTF.dll', 'C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.19041.1110_none_a8625c1886757984\comctl32.DLL', 'C:\Windows\SysWOW64\SRVCLI.DLL', 'C:\Windows\SysWOW64\textinputframework.dll', 'C:\Windows\SysWOW64\CoreUIComponents.dll', 'C:\Windows\SysWOW64\SHCORE.dll', 'C:\Windows\SysWOW64\CoreMessaging.dll', 'C:\Windows\SysWOW64\ntmarta.dll', 'C:\Windows\SysWOW64\wintypes.dll', 'C:\Windows\SysWOW64\TextShaping.dll'
Additionally, you have confirmed that these DLLs are present in the container image. Is my understanding correct? If so, you need to debug the scenario at a more granular level to understand whether the container scenario causes an alternative code path to be taken such that the messages are not correctly processed. Since you are the owner of this application, you are in a better position to debug that scenario. My hope is that you can isolate the divergent behavior to a specific API or set of APIs at which point we can remove all the complexities of your application in favor of a simplified test application.
Can you also confirm whether you are running the container in process isolation or HyperV isolation?
@Jbranadev - I wanted to follow-up on my last message. Were you able to make any progress? Additionally, can you confirm whether the issue is observed in process or HyperV isolation?
I have not been able to move forward and continue investigating what the problem is.
Because as I mentioned, it is more an issue of the environment than of application configuration.
Your response did not satisfy me because practically despite all the progress and research I had previously had, your response was to continue on my side to see what could be the cause of this behavior, when in a way I expected to receive support from you.
I am busy with other issues and I have practically reached what was within my reach.
You can close the ticket if that's what you want.
Hi @Jbranadev, it is not my intention to hastily close the ticket but rather to support you as best as I can. However, given the complexity of your end-to-end scenario, this might not be the best channel to provide the support you need unless we isolate the root cause at a more granular level. E.g. a failing function which will facilitate a more actionable investigation on my end. That said, was my description of my understanding of your issue correct?
I have perfectly understood this last comment, give me 2 weeks to clear up and be able to better isolate the problem, I will leave only what is necessary for the application to work.
Only that by doing this we will not have visibility of the logs of the dependencies that are not loaded at runtime, we will only see the problem that the message is not received as expected, since for that it used tools that docker does not provide by default .
Hello, I have had some setbacks at work, my schedule has been busy, I will try to do what you requested this weekend, sorry for the delays.
@deverouxmartin-msft Hi, sorry for the delay in my response, I had a busy schedule.
I am sharing the isolated configuration of the application that is not running inside a container, as it would normally do on a host machine.
The configuration that I share with you is so that you can test it on your Host machine and in a Windows container and thus you can appreciate the inconvenience that I report.
The configuration that I share with you is so that you can test it on your Host machine and in a Windows container and thus you can appreciate the inconvenience that I report.
Windows Host OS Version: The version is Microsoft Windows [Versión 10.0.19045.4412] Windows Container Image Version: The configuration I am sharing with you is for version mcr.microsoft.com/windows:20H2 (I have also already tested with: 2004, 1909, ltsc2019, obtaining the same result in all of them)
A simple setup that can reproduce the missing DLL's: You can download repository, where you will find everything you need to replicate the problem.
Once you have the repository on your local machine, you can do the following:
Assuming that your Host machine, on which you will test, does not have any previous configuration, there is only a need for you to copy the folders that are inside the DiscoC folder to your C:\ drive
Then run the following scripts in administrator mode
assuming it is in the directory where you downloaded the repository
cd HubContainer
.\\features.ps1
.\\initCrosswalk.ps1
Then you can run the XW Client Simulator application, which is located in the Frame Simulator folder, in HOST the IP of your Host machine must be, in Port 1966 and in terminal it would be your IP in hexadecimal, and you enter the following data
xxx.xx.xx.xxx 1966 AC1C0F08 000090009990WR9088493200000SGBSCT00182803MAY2405044 9990884932
In this application you can click on connect, then on disconnect and then on send message and then on disconnect to test the application that interests us. on your host machine will produce a result like this and it is what we hope to be able to achieve in the container.
Open a terminal with administrator permissions and run the following commands to build the docker images
cd HubContainer
docker build --force-rm --no-cache -f dockerFile -t hubcontainer .
#Ejecutar contenedor
docker run -d -p 1967:1967 -p 1966:1966 -p 1968:1968 -p 5600:5600 --name HubContainerTest -e DBI_PORT=5600 hubcontainer
# Ver el contenido del archivo log
docker exec -it 09baf6424da3 powershell
cd C:\CrossWalk\Bin\Log
ls
Get-Content "C:\CrossWalk\Bin\Log\20240731_HUB_(0)_Log.txt"
modifies the name of the file, since it creates a different log per day
20240522HUB(0)_Log.txt
assuming that if you try it on May 25 the name would be more or less like this 20240525HUB(0)_Log.txt
So, you can now click on connect, then on disconnect and then on send message and then disconnect, in the application that we use to test and then you run GetConten again, you will see a result, where the characters you are receiving are not recognizable, something very contrary to what happens on the host.
Then you can stop the container and execute the following commands, in order to compare the logs and folders with respect to the host
docker cp xxxxxxxxxxx:C:\CrossWalk C:\dockervolumenseguridad
docker cp xxxxxxxxxxx:C:\app C:\dockervolumenseguridad
docker cp xxxxxxxxxxx:C:\inetpub C:\dockervolumenseguridad
docker cp xxxxxxxxxxx:C:\Windows\SysWOW64 C:\dockervolumenseguridad\SysWOW6420042
On your Host machine, if you want to test the application that we are containerizing, you only need to run
cd C:\app .\generate_ini_hub.ps1
Then you can use the client application, to test and click on connect, disconnect and send message and then disconnect, it will produce a result like this, here what is being sent if it is received as it should be received
This is the granular configuration you asked for, it only contains the application that is not able to access the modules that Windows would provide as an OS, so we assume that the container image is the one that is not providing those modules, described in the previous comments.
It is worth mentioning that the files do exist, but they have a different weight than those of a host machine.
I look forward to your comments
@Jbranadev - thank you for the follow-up and for taking the time to document the install instructions. I took the time to follow the instructions you outlined, and unfortunately, I encountered a few failures, but I decided to continue with the steps to test the application on the host. The "Conectar" and the "Desconectar" buttons worked as you described above. The "Enviar Msg" button failed with the error "Mensaje NO Puede Ser Nulo". Perhaps the install failures contributed to the last error message. That said, my expectation of isolating the root cause at a more granular level was that you could identify a failing Microsoft function (or set of functions) or at worst to provide a very simple app that reproduces the symptoms that you believe are contributing to the malformed output of special characters. You have made the observation that your application is not able to access modules provided by Windows even though they exist. You have also made the observation that those modules differ in size between the container and the host. There is nothing unusual about the files differing in size between the container and host. However, if the module is present in the container and cannot be accessed by your application, are you stating that those modules fail to load? If so, is your application receiving module load failures or any other error result from any function exposed by a Microsoft API? Your app has many moving parts which complicate debugging any underlying issue that may be impacting your scenario. I do not want to speculate on what might be causing the malformed messages. If there is no other way of debugging your application, then the best course of action is to contact Microsoft Support where they can help to isolate the issue in your end-to-end scenario which falls outside of the scope of the support we can provide via the GitHub community channel.
And containerized an application within a container, said application is configured and started without any problem, it listens and processes the requests that are made, the problem is that the request is not received correctly, for example
If I send: 000090009990WR9088493200000SGBSCT00182803MAY2405044 9990884932 Processes what is received as a set of special characters: )))))))))))))))))))) ))))))))))))))))))))))))))))) ))))))))))
and this is because of the dll's that it needs to work, it cannot load the following ones
dll's that do exist in my container, if they are found in the corresponding directories, and I have already registered them with the following commands
but although they are registered without any problem, they are not loaded dynamically, because they depend on the following dll's
api-ms-win-security-base-l1-2-0.dll api-ms-win-core-version-private-l1-1-0.dll api-ms-win-core-heap-l2-1-0.dll api-ms-win-core-libraryloader-l1-2-0.dll api-ms-win-core-libraryloader-l1-2-1.dll api-ms-win-core-apiquery-l1-1-0.dll api-ms-win-core-delayload-l1-1-1.dll api-ms-win-core-sidebyside-l1-1-0.dll api-ms-win-core-atoms-l1-1-0.dll api-ms-win-core-synch-l1-2-1.dll api-ms-win-core-errorhandling-l1-1-2.dll api-ms-win-core-localization-l1-2-2.dll api-ms-win-core-largeinteger-l1-1-0.dll api-ms-win-core-calendar-l1-1-0.dll api-ms-win-core-path-l1-1-0.dll api-ms-win-core-rtlsupport-l1-2-0.dll api-ms-win-core-com-l1-1-1.dll api-ms-win-rtcore-ntuser-private-l1-1-4.dll api-ms-win-security-accesshlpr-l1-1-0.dll api-ms-win-security-capability-l1-1-0.dll ext-ms-win-security-chambers-l1-1-0.dll api-ms-win-coreui-secruntime-l1-1-0.dll ext-ms-onecore-orientation-l1-1-0.dll ext-ms-win-session-usermgr-l1-1-0.dll ext-ms-win-mininput-inputhost-l1-1-0.dll ext-ms-win-rtcore-minuser-private-ext-l1-1-0.dll ext-ms-win-rtcore-minuser-internal-l1-1-0.dll api-ms-win-rtcore-ntuser-private-l1-1-2.dll api-ms-win-rtcore-ntuser-private-l1-1-7.dll api-ms-win-rtcore-ntuser-private-l1-1-0.dll ext-ms-win-rtcore-gdi-devcaps-l1-1-0.dll ext-ms-win-rtcore-ntuser-dc-access-l1-1-0.dll ext-ms-win-rtcore-ntuser-window-ext-l1-1-0.dll ext-ms-onecore-shellchromeapi-l1-1-0.dll api-ms-win-core-quirks-l1-1-0.dll api-ms-win-appmodel-runtime-l1-1-1.dll api-ms-win-appmodel-runtime-l1-1-0.dll api-ms-win-ntuser-sysparams-l1-1-0.dll api-ms-win-core-winrt-propertysetprivate-l1-1-0.dll api-ms-win-core-winrt-propertysetprivate-l1-1-1.dll ext-ms-win-rtcore-ntuser-integration-l1-1-0.dll ext-ms-win-rtcore-ntuser-synch-ext-l1-1-0.dll ext-ms-win-rtcore-minuser-private-ext-l1-1-1.dll api-ms-win-security-activedirectoryclient-l1-1-0.dll api-ms-win-core-job-l2-1-0.dll
which are technically provided by the OS by default, but they cannot be loaded within the container, so I assume that the base image of my container does not have access to these dlls, and I have tried installing the following packages through winget, which should to install the redistributable c++ packages in the container image, but despite that, these dlls cannot be loaded,
Here I leave an example of my docker images
Image 1
Image 2
Technically with that configuration those dll's should already be available, however they cannot be loaded at runtime