windows ci failed with STATUS_DLL_INIT_FAILED

GreptimeTeam / greptimedb

An open-source, cloud-native, unified time series database for metrics, logs and events with SQL/PromQL supported. Available on GreptimeCloud.

https://greptime.com/

Apache License 2.0

4.18k stars 298 forks source link

windows ci failed with STATUS_DLL_INIT_FAILED #3902

Open evenyag opened 4 months ago

evenyag commented 4 months ago

What type of bug is this?

Unexpected error

What subsystems are affected?

Standalone mode

Minimal reproduce step

Run the nightly CI https://github.com/GreptimeTeam/greptimedb/actions/runs/9009678033/job/24754350747

What did you expect to see?

The CI is passed

What did you see instead?

The CI is failed

What operating system did you use?

Windows Server 2022

What version of GreptimeDB did you use?

latest

Relevant log output and stack trace

Caused by:
  for `benchmarks::bin/nyc-taxi`, command `'C:\a\greptimedb\greptimedb\target\debug\deps\nyc_taxi-1c449a394a7932c1.exe' --list --format terse` exited with code 0xc0000142: A dynamic link library (DLL) initialization routine failed. (os error 1114)

     Running `target\debug\sqlness-runner.exe`
error: process didn't exit successfully: `target\debug\sqlness-runner.exe` (exit code: 0xc0000142, STATUS_DLL_INIT_FAILED)
Error: Process completed with exit code 1.

evenyag commented 4 months ago

On Windows Server 2022 there are some strange DDL issues.

1434:0d10 @ 09019093 - LdrpInitializeNode - ERROR: Init routine 00007FFDA6925700 for DLL "C:\Windows\SYSTEM32\vertdll.dll" failed during DLL_PROCESS_ATTACH
1434:0d10 @ 09019093 - LdrpInitializeProcess - ERROR: Running the init routines of the executable's static imports failed with status 0xc0000142
1434:0d10 @ 09019093 - _LdrpInitialize - ERROR: Process initialization failed with status 0xc0000142

After adding vertdll.dll from Windows 10 to target/debug/, the sqlness.exe could be executed. So it seems that the binary has some issues using the vertdll.dll on Windows Server 2022.

evenyag commented 4 months ago

The dumpbin command shows that we uses WaitOnAddress from the vertdll.dll

 .\dumpbin.exe -imports sqlness-runner.exe

Outputs:

vertdll.dll
            141C09710 Import Address Table
            1423918D8 Import Name Table
                    0 time date stamp
                    0 Index of first forwarder reference

                        7 WaitOnAddress
                        9 WakeByAddressSingle
                        8 WakeByAddressAll

I wonder if it is related to this issue https://github.com/rust-lang/rust/issues/123999 as https://github.com/rust-lang/rust/pull/124019 mentioned WaitOnAddress

evenyag commented 4 months ago

Some tests are failed on Windows Server 2019. We got a permission denied error while deleting a file. But the file is actually deleted.

2024-05-11T12:00:57.565505Z  INFO common_telemetry::logging: logs dir = /tmp/__unittest_logs
2024-05-11T12:00:57.566327Z  INFO mito2::cache::file_cache::tests: temp dir is C:\Users\ADMINI~1\AppData\Local\Temp\2\T8ZIFB
2024-05-11T12:00:57.777286Z DEBUG mito2::cache::file_cache: deleted file files/8589934592000.67b7aad5-1616-491e-9301-55e6dadfd2e3.parquet
2024-05-11T12:00:57.778244Z  WARN mito2::cache::file_cache: Failed to delete a cached file files/8589934592000.67b7aad5-1616-491e-9301-55e6dadfd2e3.parquet err=PermissionDenied (permanent) at delete => permission denied

Context:
   service: fs
   path: files/8589934592000.67b7aad5-1616-491e-9301-55e6dadfd2e3.parquet

Source:
   Access is denied. (os error 5)