sporadic CI failures deleting files on windows

[rbtcollins]

The Windows CI jobs fail from time to time deleting files during self upgrade; its likely a genuine bug, and the normal first step would be a procmon or WPA trace to identify what process is holding a handle open on the file - so we can see whether we are stepping on our own feet, or whether it is a virus scanner race we need more retries for.

High level we need to:

• install procmon as part of Windows CI jobs
• have an s3 bucket somewhere with autodelete of content
• run procmon around each windows test
• if that one test fails, upload the procmon PML file to s3 and provide a link in the error output in the console

[tesuji]

Here is the log file created by procmon while running cli-v1 test:

cargo test --release --test cli-v1 -- remove_toolchain_then_add_again run_command rustc_no_default_toolchain update_channel update_on_channel_when_date_has_changed

Logfile.PML.zip

[kinnison]

Oooh well done @lzutao -- now we need to see if the log gives us what we need to diagnose. Is there a procmon reader for Linux?

[tesuji]

procmon could generated XML and CSV format if you prefer.

[rbtcollins]

[rbtcollins]

Thats STATUS_TOO_MANY_LINKS https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/69643dd3-b518-465d-bb0e-e2e9c5b7875e indeed.

[rbtcollins]

I think I might know whats going on: we're linking the rustup.exe from target into the test case. Presumably we're linking all the aliases too - cargo, rustc etc.

NTFS has a link limit per file of 1K.

So a mere 100 tests making 10 links each would be able to exceed this limit. And and any laziness in accounting (or in test working dir cleanup equally) would lead to issues.

My recommendation: change the test suite to take a copy of the the rustup.exe we're testing with for each test, so that all the links being made are siloed within each test, then we are going to be under that threshold.

[rbtcollins]

https://en.wikipedia.org/wiki/NTFS#Hard_links

[tesuji]

Also see #995

[rbtcollins]

I think we can probably close this though we might want to put a comment in the code base about why it is a copy vs a hard link of the binary.

[workingjubilee]

@rustbot label: +O-windows