search

microsoft / LightGBM

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
https://lightgbm.readthedocs.io/en/latest/
MIT License
16.53k stars 3.82k forks source link

`OSError: [WinError 122] The data area passed to a system call is too small` in Windows bdist CI job #6544

Closed StrikerRUS closed 2 weeks ago

StrikerRUS commented 1 month ago

Seems that Azure Pipelines builds are constantly failing for the master branch.

image

For example, the latest one link: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16597&view=results

One of the failing jobs is Windows bdist

2024-07-12T18:51:57.5021734Z Downloading pydistcheck-0.7.1-py3-none-any.whl (19 kB)
2024-07-12T18:51:58.6643657Z Installing collected packages: pydistcheck
2024-07-12T18:51:58.8103421Z Successfully installed pydistcheck-0.7.1
2024-07-12T18:51:59.8819463Z Traceback (most recent call last):
2024-07-12T18:51:59.8820899Z   File "<frozen runpy>", line 198, in _run_module_as_main
2024-07-12T18:51:59.8821855Z   File "<frozen runpy>", line 88, in _run_code
2024-07-12T18:51:59.8828650Z   File "C:\Miniconda\envs\test-env\Scripts\pydistcheck.exe\__main__.py", line 7, in <module>
2024-07-12T18:51:59.8829712Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1157, in __call__
2024-07-12T18:51:59.8841284Z     return self.main(*args, **kwargs)
2024-07-12T18:51:59.8848274Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8848970Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1078, in main
2024-07-12T18:51:59.8849427Z     rv = self.invoke(ctx)
2024-07-12T18:51:59.8850949Z          ^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8851311Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1434, in invoke
2024-07-12T18:51:59.8857514Z     return ctx.invoke(self.callback, **ctx.params)
2024-07-12T18:51:59.8858902Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8861213Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 783, in invoke
2024-07-12T18:51:59.8863013Z     return __callback(*args, **kwargs)
2024-07-12T18:51:59.8863771Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8865099Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\cli.py", line 256, in check
2024-07-12T18:51:59.8866530Z     errors += this_check(distro_summary=summary)
2024-07-12T18:51:59.8867326Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8868843Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\checks.py", line 64, in __call__
2024-07-12T18:51:59.8870013Z     has_debug_symbols, cmd_str = _file_has_debug_symbols(
2024-07-12T18:51:59.8870823Z                                  ^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8871605Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 73, in _file_has_debug_symbols
2024-07-12T18:51:59.8877657Z     has_debug_symbols, cmd_str = _look_for_debug_symbols(lib_file=file_absolute_path)
2024-07-12T18:51:59.8878095Z                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8879249Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 48, in _look_for_debug_symbols
2024-07-12T18:51:59.8879935Z     stdout = _run_command(args=[*cmd_args, lib_file])
2024-07-12T18:51:59.8880333Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8880723Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 16, in _run_command
2024-07-12T18:51:59.8881941Z     stdout = subprocess.run(args, capture_output=True, check=True).stdout
2024-07-12T18:51:59.8882309Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8883036Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 548, in run
2024-07-12T18:51:59.8896405Z     with Popen(*popenargs, **kwargs) as process:
2024-07-12T18:51:59.8897987Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8904669Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 1026, in __init__
2024-07-12T18:51:59.8913959Z     self._execute_child(args, executable, preexec_fn, close_fds,
2024-07-12T18:51:59.8915143Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 1538, in _execute_child
2024-07-12T18:51:59.8929852Z     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2024-07-12T18:51:59.8930741Z                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8933401Z ==================== running pydistcheck ====================
2024-07-12T18:51:59.8934510Z OSError: [WinError 122] The data area passed to a system call is too small
jameslamb commented 1 month ago

I'm investigating this.

Since this is not happening on PRs but is happening reliably on merges to master, I'm going to try to investigate by trying to answer the question "what is different about Windows bdist jobs triggered by merges?".

One idea I'm considering is that maybe some filepath is slightly different. https://github.com/sarugaku/shellingham/issues/8 suggests that maybe this error can show up when a too-long filepath is passed through to something using the Windows API to interact with the filesystem.

jameslamb commented 1 month ago

I compared the logs between a recent successful PR build (build link) and master build (build link).

The most significant difference I see is that on merges to master, these CodeQL scanning tasks are being injected: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16693&view=logs&j=ea56812e-e7ae-55d0-6abc-4a217857fa9f&t=39805323-ea77-5bd7-9d32-4263a4c166c3.

From the logs there, it looks like those are setting up some kind of tracing.

full logs (click me) ```text 2024-07-20T16:48:19.9155659Z ##[section]Starting: Initialize CodeQL (auto-injected) 2024-07-20T16:48:19.9162511Z ============================================================================== 2024-07-20T16:48:19.9162907Z Task : CodeQL 3000 Init 2024-07-20T16:48:19.9163086Z Description : Initilizes CodeQL before the code build task. 2024-07-20T16:48:19.9163275Z Version : 0.1.332 2024-07-20T16:48:19.9163434Z Author : Microsoft Corporation 2024-07-20T16:48:19.9163582Z Help : [CodeQL 3000 Extension WiKi](https://aka.ms/codeql3000) 2024-07-20T16:48:19.9163816Z ============================================================================== 2024-07-20T16:48:20.8493473Z ##[verbose]Creating folder: D:\a\_temp\codeql3000 2024-07-20T16:48:20.8494654Z ##[verbose]Created folder: D:\a\_temp\codeql3000 2024-07-20T16:48:20.8495262Z ##[verbose]Task environment in: D:\a\_temp\codeql3000 2024-07-20T16:48:20.8495808Z ##[verbose]Creating folder: d 2024-07-20T16:48:20.8496348Z ##[verbose]Created folder: D:\a\_temp\codeql3000\d 2024-07-20T16:48:20.8499115Z ==================================================== 2024-07-20T16:48:20.8499712Z = Session Id: f61febea-f6ac-4665-8ccd-aa439b276cd4 = 2024-07-20T16:48:20.8500231Z = = 2024-07-20T16:48:20.8500717Z = For support, please visit CodeQL Support System = 2024-07-20T16:48:20.8501275Z = https://aka.ms/codeqlsupport  = 2024-07-20T16:48:20.8501791Z = = 2024-07-20T16:48:20.8502282Z ==================================================== 2024-07-20T16:48:20.8502817Z Configuration: 2024-07-20T16:48:20.8503289Z Cadence = 72 2024-07-20T16:48:20.8503761Z CLIVersion = 2.17.6 2024-07-20T16:48:20.8504208Z Enabled = true 2024-07-20T16:48:20.8523550Z LogLevel = 4 2024-07-20T16:48:20.8524438Z QuerySuite = sdl-required 2024-07-20T16:48:20.8524919Z ##[verbose]Environment: 2024-07-20T16:48:20.8525455Z ##[verbose]databaseFolder = "D:\\a\\_temp\\codeql3000\\d" 2024-07-20T16:48:20.8526291Z ##[verbose]sessionId = "f61febea-f6ac-4665-8ccd-aa439b276cd4" 2024-07-20T16:48:20.8526808Z ##[verbose]metadata = {} 2024-07-20T16:48:20.8527301Z ##[verbose]identifierKeys = [] 2024-07-20T16:48:20.8527794Z ##[verbose]Jobs: 2024-07-20T16:48:20.8528306Z ##[verbose]buildSource = "CodeQL3000_AutoInjected" 2024-07-20T16:48:21.4994441Z ##[verbose]C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe exit code: 0 2024-07-20T16:48:21.5002339Z ##[verbose]runningEnvironment= {"Platform":"Windows","Release":"10.0.17763","Architecture":"x64","TotalMemory":7515721728,"FreeMemory":5252374528,"Disk":"D:","DiskSize":"14.00G","DiskFree":"11.79G","WinContainer":"false"} 2024-07-20T16:48:21.5011064Z ##[verbose]Looking for project config file D:\a\1\s\CodeQL.yml 2024-07-20T16:48:21.5012393Z ##[verbose]Looking for project config file D:\a\1\s\CodeQL.yml 2024-07-20T16:48:21.5013073Z ##[verbose]Looking for project config file D:\a\1\s\.CodeQL.yml 2024-07-20T16:48:21.5013713Z ##[verbose]Looking for project config file D:\a\1\s\.CodeQL.yml 2024-07-20T16:48:21.5014351Z ##[verbose]Looking for project config file D:\a\1\s\lgtm.yml 2024-07-20T16:48:21.5020369Z ##[verbose]Looking for project config file D:\a\1\s\lgtm.yml 2024-07-20T16:48:21.5021144Z ##[verbose]Looking for project config file D:\a\1\s\.lgtm.yml 2024-07-20T16:48:21.5029832Z ##[verbose]Looking for project config file D:\a\1\s\.lgtm.yml 2024-07-20T16:48:21.5138623Z Identifying languages in the repository. 2024-07-20T16:48:21.7442849Z ##[verbose]python: Supported. Artifacts: code: 57, ignored: 0 2024-07-20T16:48:21.7444705Z ##[verbose]cpp: Supported. Artifacts: code: 1219, ignored: 0 2024-07-20T16:48:21.7450910Z ##[verbose]javascript: Supported. Artifacts: code: 5, ignored: 0 2024-07-20T16:48:21.7452188Z ##[verbose]powershell: Supported. Artifacts: code: 3, ignored: 0 2024-07-20T16:48:21.7456827Z No language was specified, trying to autodetect 2024-07-20T16:48:21.7457723Z Detected languages: python,cpp,javascript,powershell 2024-07-20T16:48:21.7504909Z ##[verbose]System.JobName: bdist 2024-07-20T16:48:21.7506212Z Looking for recently created databases 2024-07-20T16:48:22.6855109Z ##[verbose]Error: Request failed with status code 403 2024-07-20T16:48:22.6858151Z ##[verbose]No python database was found. 2024-07-20T16:48:22.6858917Z New python database will be built now. 2024-07-20T16:48:22.6859610Z ##[verbose]No cpp database was found. 2024-07-20T16:48:22.6860257Z New cpp database will be built now. 2024-07-20T16:48:22.6860964Z ##[verbose]No javascript database was found. 2024-07-20T16:48:22.6861608Z New javascript database will be built now. 2024-07-20T16:48:22.6862509Z ##[verbose]No powershell database was found. 2024-07-20T16:48:22.6863191Z New powershell database will be built now. 2024-07-20T16:48:22.6863888Z Outdated databases: python,cpp,javascript,powershell 2024-07-20T16:48:22.6864534Z Sending intent to build CodeQL database 2024-07-20T16:48:23.6824562Z ##[verbose]JobId: bc7a7ae5-9eaf-4ef6-832b-bbbd2119e156, Status: Accepted, Token: ****** 2024-07-20T16:48:23.6826258Z ##[verbose]Intent for python sent as job bc7a7ae5-9eaf-4ef6-832b-bbbd2119e156 2024-07-20T16:48:24.6627835Z ##[verbose]JobId: b466e409-9277-473a-8e0d-8cd19cefdbfb, Status: Accepted, Token: ****** 2024-07-20T16:48:24.6629645Z ##[verbose]Intent for cpp sent as job b466e409-9277-473a-8e0d-8cd19cefdbfb 2024-07-20T16:48:25.6703086Z ##[verbose]JobId: acd784b3-c41e-4c6f-82f2-92d1a9766321, Status: Accepted, Token: ****** 2024-07-20T16:48:25.6705129Z ##[verbose]Intent for javascript sent as job acd784b3-c41e-4c6f-82f2-92d1a9766321 2024-07-20T16:48:26.6837510Z ##[verbose]JobId: a256b939-5527-4f46-80ea-817e798add45, Status: Accepted, Token: ****** 2024-07-20T16:48:26.6839176Z ##[verbose]Intent for powershell sent as job a256b939-5527-4f46-80ea-817e798add45 2024-07-20T16:48:26.6839972Z Installing CodeQL CLI 2024-07-20T16:48:26.6840664Z ##[verbose]Creating folder: distribution 2024-07-20T16:48:26.6842186Z ##[verbose]Created folder: D:\a\_temp\codeql3000\distribution 2024-07-20T16:48:26.6910876Z ##[verbose]Downloading microsoft-codeql-endpoint/codeql-cli/2.17.6/win64.zip at 16:48:26 GMT+0000 (Greenwich Mean Time) 2024-07-20T16:48:26.7298382Z ##[verbose]Using cURL at C:\Windows\system32\curl.exe 2024-07-20T16:50:03.8193056Z ##[verbose]Comparing file hashes 2024-07-20T16:50:05.4372260Z ##[verbose]Downloaded to D:\a\_temp\codeql3000\distribution\win64.zip at 16:50:05 GMT+0000 (Greenwich Mean Time) 2024-07-20T16:50:05.4375203Z ##[verbose]Extracting D:\a\_temp\codeql3000\distribution\win64.zip to D:\a\_temp\codeql3000\distribution 2024-07-20T16:50:05.5171849Z ##[verbose]Using 7zip at C:\ProgramData\Chocolatey\bin\7z.exe 2024-07-20T16:50:05.5178949Z [command]C:\ProgramData\Chocolatey\bin\7z.exe x D:\a\_temp\codeql3000\distribution\win64.zip -oD:\a\_temp\codeql3000\distribution -bd -y 2024-07-20T16:50:05.7541360Z 2024-07-20T16:50:05.7543034Z 7-Zip 24.07 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-06-19 2024-07-20T16:50:05.7543444Z 2024-07-20T16:50:05.7543899Z Scanning the drive for archives: 2024-07-20T16:50:05.7544323Z 1 file, 438261024 bytes (418 MiB) 2024-07-20T16:50:05.7544733Z 2024-07-20T16:50:05.7545049Z Extracting archive: D:\a\_temp\codeql3000\distribution\win64.zip 2024-07-20T16:50:14.9097751Z -- 2024-07-20T16:50:14.9099610Z Path = D:\a\_temp\codeql3000\distribution\win64.zip 2024-07-20T16:50:14.9100427Z Type = zip 2024-07-20T16:50:14.9101046Z Physical Size = 438261024 2024-07-20T16:50:14.9101541Z 2024-07-20T16:50:14.9115326Z Everything is Ok 2024-07-20T16:50:14.9115910Z 2024-07-20T16:50:14.9116768Z Folders: 528 2024-07-20T16:50:14.9117434Z Files: 3316 2024-07-20T16:50:14.9117981Z Size: 825376595 2024-07-20T16:50:14.9118697Z Compressed: 438261024 2024-07-20T16:50:14.9200393Z Setting up CodeQL CLI 2024-07-20T16:50:14.9201455Z Initializing database 2024-07-20T16:50:14.9204221Z ##[verbose]Source code folder is: D:\a\1\s 2024-07-20T16:50:14.9205031Z ##[verbose]Database folder is: D:\a\_temp\codeql3000\d 2024-07-20T16:50:14.9211199Z ##[verbose]CLI args: database,init,--source-root,D:\a\1\s,--language,python,cpp,javascript,powershell,--db-cluster,-vvvv,--begin-tracing,D:\a\_temp\codeql3000\d 2024-07-20T16:50:14.9212764Z [command]D:\a\_temp\codeql3000\distribution\codeql\codeql.exe database init --source-root D:\a\1\s --language python,cpp,javascript,powershell --db-cluster -vvvv --begin-tracing D:\a\_temp\codeql3000\d 2024-07-20T16:50:16.0486277Z Writing logs to D:\a\_temp\codeql3000\d\log\database-init-20240720.165016.002.log. 2024-07-20T16:50:16.4006343Z Scanning for [codeql-extractor.yml] from D:\a\_temp\codeql3000\distribution\codeql\.codeqlmanifest.json 2024-07-20T16:50:16.4178165Z Parsing D:\a\_temp\codeql3000\distribution\codeql\cpp\codeql-extractor.yml. 2024-07-20T16:50:16.5473302Z Parsing D:\a\_temp\codeql3000\distribution\codeql\csharp\codeql-extractor.yml. 2024-07-20T16:50:16.5714717Z Parsing D:\a\_temp\codeql3000\distribution\codeql\csv\codeql-extractor.yml. 2024-07-20T16:50:16.5911364Z Parsing D:\a\_temp\codeql3000\distribution\codeql\go\codeql-extractor.yml. 2024-07-20T16:50:16.6170043Z Parsing D:\a\_temp\codeql3000\distribution\codeql\html\codeql-extractor.yml. 2024-07-20T16:50:16.6194153Z Parsing D:\a\_temp\codeql3000\distribution\codeql\java\codeql-extractor.yml. 2024-07-20T16:50:16.6437675Z Parsing D:\a\_temp\codeql3000\distribution\codeql\javascript\codeql-extractor.yml. 2024-07-20T16:50:16.6818175Z Parsing D:\a\_temp\codeql3000\distribution\codeql\powershell\codeql-extractor.yml. 2024-07-20T16:50:16.6959417Z Parsing D:\a\_temp\codeql3000\distribution\codeql\properties\codeql-extractor.yml. 2024-07-20T16:50:16.6963380Z Parsing D:\a\_temp\codeql3000\distribution\codeql\python\codeql-extractor.yml. 2024-07-20T16:50:16.7127724Z Parsing D:\a\_temp\codeql3000\distribution\codeql\ruby\codeql-extractor.yml. 2024-07-20T16:50:16.7131652Z Parsing D:\a\_temp\codeql3000\distribution\codeql\swift\codeql-extractor.yml. 2024-07-20T16:50:16.7143505Z Parsing D:\a\_temp\codeql3000\distribution\codeql\tsql\codeql-extractor.yml. 2024-07-20T16:50:16.7244603Z Parsing D:\a\_temp\codeql3000\distribution\codeql\xml\codeql-extractor.yml. 2024-07-20T16:50:16.7375927Z Parsing D:\a\_temp\codeql3000\distribution\codeql\yaml\codeql-extractor.yml. 2024-07-20T16:50:16.9807926Z Calculating baseline information in D:\a\1\s 2024-07-20T16:50:16.9809976Z Ignoring the following directories when processing baseline information: .git, .hg, .svn. 2024-07-20T16:50:16.9811029Z Running command in D:\a\1\s: D:\a\_temp\codeql3000\distribution\codeql\tools\win64\scc.exe --by-file --exclude-dir .git,.hg,.svn --format json --no-large --no-min . 2024-07-20T16:50:17.5861024Z Running command in D:\a\1\s: [D:\a\_temp\codeql3000\distribution\codeql\javascript\tools\configure-baseline.cmd] 2024-07-20T16:50:17.6852119Z Found 2249 baseline files for cpp. 2024-07-20T16:50:17.6854909Z Found 55 baseline files for python. 2024-07-20T16:50:17.6855773Z Found 2 baseline files for javascript. 2024-07-20T16:50:17.6856555Z Calculated baseline information for languages: cpp, javascript, python (676ms). 2024-07-20T16:50:17.9215815Z Resolving extractor python. 2024-07-20T16:50:17.9270963Z Found candidate extractor root for python: D:\a\_temp\codeql3000\distribution\codeql\python. 2024-07-20T16:50:17.9582879Z Successfully loaded extractor Python (python) from D:\a\_temp\codeql3000\distribution\codeql\python. 2024-07-20T16:50:18.0313808Z Created skeleton CodeQL database at D:\a\_temp\codeql3000\d\python. This in-progress database is ready to be populated by an extractor. 2024-07-20T16:50:18.0315824Z Resolving extractor cpp. 2024-07-20T16:50:18.0326168Z Found candidate extractor root for cpp: D:\a\_temp\codeql3000\distribution\codeql\cpp. 2024-07-20T16:50:18.0329981Z Successfully loaded extractor C/C++ (cpp) from D:\a\_temp\codeql3000\distribution\codeql\cpp. 2024-07-20T16:50:18.0407901Z Created skeleton CodeQL database at D:\a\_temp\codeql3000\d\cpp. This in-progress database is ready to be populated by an extractor. 2024-07-20T16:50:18.0409158Z Resolving extractor javascript. 2024-07-20T16:50:18.0488888Z Found candidate extractor root for javascript: D:\a\_temp\codeql3000\distribution\codeql\javascript. 2024-07-20T16:50:18.0675484Z Successfully loaded extractor JavaScript/TypeScript (javascript) from D:\a\_temp\codeql3000\distribution\codeql\javascript. 2024-07-20T16:50:18.0678014Z Created skeleton CodeQL database at D:\a\_temp\codeql3000\d\javascript. This in-progress database is ready to be populated by an extractor. 2024-07-20T16:50:18.0679033Z Resolving extractor powershell. 2024-07-20T16:50:18.0679816Z Found candidate extractor root for powershell: D:\a\_temp\codeql3000\distribution\codeql\powershell. 2024-07-20T16:50:18.0680865Z Successfully loaded extractor powershell (powershell) from D:\a\_temp\codeql3000\distribution\codeql\powershell. 2024-07-20T16:50:18.0688539Z Created skeleton CodeQL database at D:\a\_temp\codeql3000\d\powershell. This in-progress database is ready to be populated by an extractor. 2024-07-20T16:50:18.1802371Z Trace process level: 1 2024-07-20T16:50:20.0845540Z Looking for process tree index 4 (trace process level 1) 2024-07-20T16:50:20.0847283Z Process tree index 0 (trace process level -3): Win32_Process: powershell.exe (Handle = "1088") 2024-07-20T16:50:20.0848220Z Process tree index 1 (trace process level -2): Win32_Process: java.exe (Handle = "732") 2024-07-20T16:50:20.0849055Z Process tree index 2 (trace process level -1): Win32_Process: codeql.exe (Handle = "4640") 2024-07-20T16:50:20.0849857Z Process tree index 3 (trace process level 0): Win32_Process: node.exe (Handle = "2264") 2024-07-20T16:50:20.0850649Z Process tree index 4 (trace process level 1): Win32_Process: Agent.Worker.exe (Handle = "4412") 2024-07-20T16:50:20.0851490Z Found Agent.Worker.exe process which means we are running on Azure Pipelines 2024-07-20T16:50:20.0852029Z Aborting search early and using process: Win32_Process: Agent.Worker.exe (Handle = "4412") 2024-07-20T16:50:20.0852455Z Injecting into process: Win32_Process: Agent.Worker.exe (Handle = "4412") 2024-07-20T16:50:20.0852733Z 2024-07-20T16:50:20.1011813Z To initialize indirect tracing, some environment variables need to be set in the context the build will run in. 2024-07-20T16:50:20.1014162Z A number of scripts to do this have been created in D:\a\_temp\codeql3000\d\temp\tracingEnvironment. 2024-07-20T16:50:20.1015429Z Please run one of these scripts before invoking your build command. 2024-07-20T16:50:20.1025482Z 2024-07-20T16:50:20.1026956Z Based on your operating system, we recommend you run: 2024-07-20T16:50:20.1027700Z D:\a\_temp\codeql3000\d\temp\tracingEnvironment\start-tracing.ps1 2024-07-20T16:50:20.1257894Z ##[verbose]CLI exit code: 0 2024-07-20T16:50:20.1260229Z ##[verbose]Setting variables from file: D:\a\_temp\codeql3000\d\temp\tracingEnvironment\start-tracing.json 2024-07-20T16:50:20.1330524Z ##[verbose]Database initialized and tracing started 2024-07-20T16:50:20.1331604Z Successfully initialized the database 2024-07-20T16:50:20.1332368Z ##[verbose]Saving environment properties. 2024-07-20T16:50:20.1333048Z ##[verbose]Saving configuration variables. 2024-07-20T16:50:20.1333699Z ##[verbose]Saving status properties. 2024-07-20T16:50:20.1334413Z __________________________________________________________________________________________________________________________________________ 2024-07-20T16:50:20.1335251Z |Language |Status |Message | 2024-07-20T16:50:20.1336029Z |______________________|___________________________|_____________________________________________________________________________________| 2024-07-20T16:50:20.1358955Z |cpp |database initialized |Will attempt to trace the build process of compiled language | 2024-07-20T16:50:20.1359745Z | | |Check the Finalize task logs for status | 2024-07-20T16:50:20.1360799Z |______________________|___________________________|_____________________________________________________________________________________| 2024-07-20T16:50:20.1361655Z |javascript |database initialized |Will attempt to create database for interpreted language | 2024-07-20T16:50:20.1362416Z | | |Check the Finalize task logs for status | 2024-07-20T16:50:20.1363300Z |______________________|___________________________|_____________________________________________________________________________________| 2024-07-20T16:50:20.1364333Z |powershell |database initialized |Will attempt to create database for interpreted language | 2024-07-20T16:50:20.1365077Z | | |Check the Finalize task logs for status | 2024-07-20T16:50:20.1365872Z |______________________|___________________________|_____________________________________________________________________________________| 2024-07-20T16:50:20.1366711Z |python |database initialized |Will attempt to create database for interpreted language | 2024-07-20T16:50:20.1367397Z | | |Check the Finalize task logs for status | 2024-07-20T16:50:20.1368272Z |______________________|___________________________|_____________________________________________________________________________________| 2024-07-20T16:50:20.1368873Z 2024-07-20T16:50:20.1369478Z If there were failures, check the logs for the detailed error messages. 2024-07-20T16:50:22.7053930Z ##[section]Finishing: Initialize CodeQL (auto-injected) ```

I wonder if maybe the mechanism there is somehow interfering with system tools in a way that pydistcheck cannot handle? The logs sort of make it look like core parts of the Windows API are being replaced with instrumented versions (but I'm very unsure about this).

I'm going to put up a PR trying to turn those jobs off.

jameslamb commented 1 month ago

I think that #6563 fixed this.

https://github.com/microsoft/LightGBM/pull/6563#issuecomment-2241928232

But I think we should leave it open until we see a few more successful CI runs on merges. I'm putting the awaiting response label on it so it'll be automatically closed after 30 days (just in case we forget to come back and close it).

StrikerRUS commented 1 month ago

@jameslamb Ah, thank you very much for finding this hidden rogue! I remember we switched of this check in the past: #5175.

I have no idea what is it, but we haven't requested these checks. 😕 They just increase our overall CI time and are potential places for project-unrelated CI failures.

github-actions[bot] commented 2 weeks ago

This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one. Thank you for taking the time to improve LightGBM!