Support for self-hosted Windows runners

nazar-pc commented 1 year ago

I tried this action on self-hosted runner and while Ubuntu worked, Windows did't:

``` ##[debug]Evaluating condition for step: 'Install cargo-nextest' ##[debug]Evaluating: success() ##[debug]Evaluating success: ##[debug]=> true ##[debug]Result: true ##[debug]Starting: Install cargo-nextest ##[debug]Loading inputs ##[debug]Loading env Run taiki-e/install-action@0163f6cf65d9b9bb0e5d1f0e3ae19280a06be4b0 ##[debug]Evaluating: inputs.tool ##[debug]Evaluating Index: ##[debug]..Evaluating inputs: ##[debug]..=> Object ##[debug]..Evaluating String: ##[debug]..=> 'tool' ##[debug]=> 'cargo-nextest' ##[debug]Result: 'cargo-nextest' ##[debug]Evaluating: inputs.checksum ##[debug]Evaluating Index: ##[debug]..Evaluating inputs: ##[debug]..=> Object ##[debug]..Evaluating String: ##[debug]..=> 'checksum' ##[debug]=> 'true' ##[debug]Result: 'true' ##[debug]Evaluating condition for step: 'run' ##[debug]Evaluating: success() ##[debug]Evaluating success: ##[debug]=> true ##[debug]Result: true ##[debug]Starting: run ##[debug]Loading inputs ##[debug]Loading env Run bash --noprofile --norc "${GITHUB_ACTION_PATH:?}/main.sh" ##[debug]C:\cygwin64\bin\bash.EXE --noprofile --norc -e -o pipefail C:\actions-runner\_work\_temp\74c519f7-435e-4f1d-9e43-a0fae0148bd9.sh info: install-action does not support cargo-nextest; fallback to cargo-binstall info: installing cargo-binstall info: downloading null curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL curl: (3) URL using bad/illegal format or missing URL Error: Process completed with exit code 3. ##[debug]Finished: run ##[debug]Finishing: Install cargo-nextest ```

Would be convenient if it was supported in this environment as well.

taiki-e commented 1 year ago

Thanks for the report! There may be some issue with jq or path handling. Could you try aa0fb41b4db2493d2ef6e45c478fb01378bc2809?

nazar-pc commented 1 year ago

host platform: x86_64_windows
jq version: jq-1.6

https://github.com/subspace/subspace/actions/runs/6188492029/job/16800658877#step:7:1

taiki-e commented 1 year ago

host platform: x86_64_windows
jq version: jq-1.6
info: install-action does not support cargo-nextest; fallback to cargo-binstall
info: installing cargo-binstall@1.3.0
info: downloading null

Hmm. jq at least succeeds in parsing the manifest the first and second time (since "1.3.0" is parsed), but the subsequent comparison to "null" (even though the actual value is "null") appears to be evaluated as false.

https://github.com/taiki-e/install-action/blob/c33cd1d5e1ef655caac77b00640bbbd1731caa2b/main.sh#L209

Given that the jq version is 1.6 and that a flag was added in 1.7 to fix the carriage return problem, I guess this is probably related to https://github.com/jqlang/jq/issues/1854.

Could you try c86cf7e603a60840f49eb6ec40eadb133072f91d?

nazar-pc commented 1 year ago

I don't see anything helpful in the output: https://github.com/subspace/subspace/actions/runs/6188821326/job/16801617758#step:7:1 jq 1.7 if fairly new, I think there are plenty of environments where it is not yet present.

taiki-e commented 1 year ago

https://github.com/jqlang/jq/issues/1854 appears to indicate that the problem exists only with a particular type of jq binary, and the curl error (unable to write to hidden directory) appears to be specific to a particular build way. (Both work fine, at least when using the binaries included in the GitHub-provided runners.)

Could you try 77363e9d6d7d3414223a8f118f39650fb2d9c553? (added potential workaround for curl hidden directory issue)

nazar-pc commented 1 year ago

Hm...

host platform: x86_64_windows
jq version: jq-1.7-dirty
info: install-action does not support cargo-nextest; fallback to cargo-binstall
info: installing cargo-binstall@1.3.0
info: downloading https://github.com/cargo-bins/cargo-binstall/releases/download/v1.3.0/cargo-binstall-x86_64-pc-windows-msvc.zip
info: verifying sha256 checksum for cargo-binstall-x86_64-pc-windows-msvc.zip
info: cargo-binstall installed at 
+ cargo binstall -V
error: no such command: `binstall`

    Did you mean `install`?

    View all installed commands with `cargo --list`
Error: Process completed with exit code 101.

https://github.com/subspace/subspace/actions/runs/6189004596/job/16802168830#step:7:1

taiki-e commented 1 year ago

Could you try f2f6a1fee94a02c13db4374fd3ee9fa4f2b15530? Added a potential fix and some debug output.

nazar-pc commented 1 year ago

Didn't work either:

host platform: x86_64_windows
jq version: jq-1.7-dirty
info: install-action does not support cargo-nextest; fallback to cargo-binstall
error: no such command: `binstall`

    Did you mean `install`?

    View all installed commands with `cargo --list`
info: cargo-binstall already installed at /usr/local/bin/cargo-binstall.exe, but is not compatible version with install-action, upgrading
info: installing cargo-binstall@1.3.0
info: downloading https://github.com/cargo-bins/cargo-binstall/releases/download/v1.3.0/cargo-binstall-x86_64-pc-windows-msvc.zip
info: verifying sha256 checksum for cargo-binstall-x86_64-pc-windows-msvc.zip
Warning: cargo-binstall should be installed at /usr/local/bin; but cargo-binstall.exe not found in path
+ cargo binstall -V
error: no such command: `binstall`

    Did you mean `install`?

    View all installed commands with `cargo --list`

https://github.com/subspace/subspace/actions/runs/6189176189/job/16802753709#step:7:1

P.S. Every time I have an issue with CI it is never an exact science, but rather trial and error.

taiki-e commented 1 year ago

I found a bug related to determining the installation location that I can reproduce in other Windows environments. I think that is the cause of the remaining problems.

Could you try cc5181c92a8beda8da32d4bd837b8b9c700ab348?

nazar-pc commented 1 year ago

Something new this time:

host platform: x86_64_windows
jq version: jq-1.7-dirty
info: install-action does not support cargo-nextest; fallback to cargo-binstall
info: installing cargo-binstall@1.3.0
info: downloading https://github.com/cargo-bins/cargo-binstall/releases/download/v1.3.0/cargo-binstall-x86_64-pc-windows-msvc.zip
info: verifying sha256 checksum for cargo-binstall-x86_64-pc-windows-msvc.zip
info: cargo-binstall installed at /home/Administrator/.install-action/bin/cargo-binstall.exe
+ cargo binstall -V
error: could not execute process `C:\cygwin64\home\Administrator\.install-action\bin\cargo-binstall.exe binstall -V` (never executed)

Caused by:
  Access is denied. (os error 5)

https://github.com/subspace/subspace/actions/runs/6189633359/job/16804165414#step:7:1

taiki-e commented 1 year ago

I changed to prefer $RUNNER_TOOL_CACHE over $HOME in a84bb470d0b8469209c57eb14de2bf4f59bc96b1, so the permission issue should be fixed now.

nazar-pc commented 1 year ago

Nope, same error again: https://github.com/subspace/subspace/actions/runs/6190541151/job/16806980662#step:7:1 UPD: I used the old commit, let me try again :upside_down_face:

nazar-pc commented 1 year ago

No, updated commit didn't work either with a different error:

info: host platform: x86_64_windows
info: cargo is installed at /cygdrive/c/Users/Administrator/.cargo/bin/cargo
info: jq version: jq-1.7-dirty
info: install-action does not support cargo-nextest; fallback to cargo-binstall
info: installing cargo-binstall@1.3.0
info: downloading https://github.com/cargo-bins/cargo-binstall/releases/download/v1.3.0/cargo-binstall-x86_64-pc-windows-msvc.zip
info: verifying sha256 checksum for cargo-binstall-x86_64-pc-windows-msvc.zip
Warning: cargo-binstall should be installed at /c/actions-runner/_work/_tool/install-action/bin/cargo-binstall.exe; but cargo-binstall.exe not found in path
+ cargo binstall -V
error: no such command: `binstall`

    Did you mean `install`?

    View all installed commands with `cargo --list`
Error: Process completed with exit code 101.

https://github.com/subspace/subspace/actions/runs/6190591939/job/16807141866#step:7:1

taiki-e commented 1 year ago

Hmm, sorry, that did not seem to be enough to fix the problem. (On second thought, if installing on HOME was really the problem, then the jq run would have failed as well)

The difference between the way cargo-binstall and jq are executed is whether it goes through cargo or not. I have created a patch (79a868a4c0889f8bc9ec358aec0009aa0a887db1) that calls cargo binstall without going through cargo, could you try that?

nazar-pc commented 1 year ago

Still permission denied:

+ cargo-binstall binstall -V
C:\actions-runner\_work\_actions\taiki-e\install-action\79a868a4c0889f8bc9ec358aec0009aa0a887db1/main.sh: line 11: /home/Administrator/.install-action/bin/cargo-binstall: Permission denied
Error: Process completed with exit code 126.

https://github.com/subspace/subspace/actions/runs/6190724569/job/16807554656

Is there something like chmod +x on Windows? I didn't use Windows for anything work-wise for ~10 years now.

taiki-e commented 1 year ago

Is there something like chmod +x on Windows?

Good point. I found https://cygwin.com/cygwin-ug-net/using-filemodes.html

On FAT or FAT32 filesystems, files are always readable, and Cygwin uses the DOS read-only attribute to determine if they are writable. Files are considered to be executable if the filename ends with .bat, .com or .exe, or if its content starts with #!.

Whether or not the .exe is used at the time of the call is indeed another difference between the way cargo-binstall and jq were executed...

I updated my patch to use .exe (84cf87d08640570a4867937b58e6cab8d5b8fed9).

nazar-pc commented 1 year ago

info: host platform: x86_64_windows
info: cargo is located at /cygdrive/c/Users/Administrator/.cargo/bin/cargo
info: jq version: jq-1.7-dirty
info: install-action does not support cargo-nextest; fallback to cargo-binstall
info: installing cargo-binstall@1.3.0
info: downloading https://github.com/cargo-bins/cargo-binstall/releases/download/v1.3.0/cargo-binstall-x86_64-pc-windows-msvc.zip
info: verifying sha256 checksum for cargo-binstall-x86_64-pc-windows-msvc.zip
Warning: cargo-binstall should be installed at /home/Administrator/.install-action/bin/cargo-binstall.exe; but cargo-binstall.exe not found in path
+ cargo-binstall.exe binstall -V
C:\actions-runner\_work\_actions\taiki-e\install-action\84cf87d08640570a4867937b58e6cab8d5b8fed9/main.sh: line 11: cargo-binstall.exe: command not found
Error: Process completed with exit code 127.

https://github.com/subspace/subspace/actions/runs/6190869896/job/16808019785#step:7:1

taiki-e commented 1 year ago

Oh, that's probably a bug about adding PATHs, and it should be fixed in 6b495ce909cc90247ed5baa261cf4bcf03f75eff.

nazar-pc commented 1 year ago

Back to permission denied:

info: host platform: x86_64_windows
info: cargo is located at /cygdrive/c/Users/Administrator/.cargo/bin/cargo
info: jq version: jq-1.7-dirty
info: install-action does not support cargo-nextest; fallback to cargo-binstall
info: installing cargo-binstall@1.3.0
info: adding \home\Administrator\.install-action\bin to PATH
info: downloading https://github.com/cargo-bins/cargo-binstall/releases/download/v1.3.0/cargo-binstall-x86_64-pc-windows-msvc.zip
info: verifying sha256 checksum for cargo-binstall-x86_64-pc-windows-msvc.zip
info: cargo-binstall installed at /home/Administrator/.install-action/bin/cargo-binstall.exe
+ cargo-binstall.exe binstall -V
C:\actions-runner\_work\_actions\taiki-e\install-action\6b495ce909cc90247ed5baa261cf4bcf03f75eff/main.sh: line 11: /home/Administrator/.install-action/bin/cargo-binstall.exe: Permission denied

https://github.com/subspace/subspace/actions/runs/6191193015/job/16808987474#step:7:1

taiki-e commented 1 year ago

Hmm...

One of the other differences between jq and cargo-binstall is whether it is called with the full path or not, so changing that (582603535a55cc92add685115f1d18f98c8b020e) might work, but it probably would not resolve the underlying problem (except when there was a problem only with cargo-binstall).
Another possibility is that there is a problem with the unzipping process of cargo-binstall's zip file or with the binaries themselves. It might be helpful to see if cargo-llvm-cov, which uses tar.gz, or shfmt, which installs binaries directly, also has problems. (In this case, we may want to revert https://github.com/taiki-e/install-action/pull/183)

That said, it may be better here to try taiki-e/cache-cargo-install-action, which was created to support cases where install-action does not work.

NobodyXu commented 1 year ago

@nazar-pc Could that be the windows defender or other defending mechanism kicking in?

nazar-pc commented 1 year ago

Full path shouldn't be necessary. Just tried taiki-e/cache-cargo-install-action and it worked just fine, will go with it for now.

@nazar-pc Could that be the windows defender or other defending mechanism kicking in?

Anything is possible, asked DevOps person on our team to take a look there.

taiki-e commented 7 months ago

taiki-e / install-action

Support for self-hosted Windows runners #224