`npx` not found under Windows

[xobs]

I'm still trying to track this one down, and in the meantime I'm opening this issue to document my findings.

The espfs build will silently fail on Windows when compressing files using npx:

[2/7] Building espfs binary CMakeFiles/blackmagic.elf.dir/espfs
'"npx uglifycss"' is not recognized as an internal or external command,
operable program or batch file.
0abb65ac xterm.css                          file 4.1 KiB   -> 0.0 KiB   (0.0%)
0f71a853 flash                              dir
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
0fd72300 xterm.js                           file 255.9 KiB -> 0.0 KiB   (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
52372b2f debug.html                         file 1.6 KiB   -> 0.0 KiB   (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
66e3de48 index.js                           file 94 B      -> 0 B       (0.0%)
'"npx uglifycss"' is not recognized as an internal or external command,
operable program or batch file.
69d0be6a flash/style.css                    file 517 B     -> 0 B       (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
70aae05d flash/index.html                   file 1.8 KiB   -> 0.0 KiB   (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
ad1d1634 flash/140medley.min.js             file 827 B     -> 0 B       (0.0%)
'"npx html-minifier"' is not recognized as an internal or external command,
operable program or batch file.
af538a40 index.html                         file 6.7 KiB   -> 0.0 KiB   (0.0%)
The system cannot find the path specified.
'"npx uglifyjs"' is not recognized as an internal or external command,
operable program or batch file.
f4f97403 xterm-addon-fit.js                 file 2.3 KiB   -> 0.0 KiB   (0.0%)
[7/7] Generating binary image from built executable
esptool.py v3.2-dev

This is despite the fact that the npx command exists and is capable of running commands:

[15:31:33] E:/Code/esp/blackmagic-espidf> npx uglifycss --help
Usage: uglifycss [options] file1.css [file2.css [...]] > output
options:
--max-line-len n  add a newline every n characters
--expand-vars     expand variables
--ugly-comments   remove newlines within preserved comments
--cute-comments   preserve newlines within and around preserved comments
--convert-urls d  convert relative urls using the d directory as location target
--debug           print full error stack on error
--output f        put the result in f file
--help            show this help
--version         display version number
[15:31:47] E:/Code/esp/blackmagic-espidf>

The compilation does not fail when this occurs, and instead generates empty files.

Possible reasons for this:

• The command is getting quoted in its entirety, generating an invalid command -- indeed, there is no program named npx html-minifier, it's supposed to call "npx" "html-minifier"
• The idf.py shell is replacing PATH with one that is cleaned of any system PATH entries
• Something is generating paths inside the JS code that is using backslashes that are getting misinterpreted later on down the line
• It's bouncing back and forth between cmd.exe and powershell, however this is less likely because the error occurs as well with cmd.exe without powershell

There are two issues here:

1. A failure in the generation process is nonfatal
2. The build system is unable to find npx on Windows even though it exists and is on the PATH

[jkent]

Hopefully one of these days I'll get around to testing this on a Windows host machine. Have you made any more progress since your initial report?

[xobs]

I have discovered a workaround to this. I have a patch I've made to get mkespfsimage.py working.

I was getting ready to make a PR, but then I discovered that it no longer exists. I believe a similar patch would be:

diff --git a/tools/preprocess.py b/tools/preprocess.py
index 8358e0d..9dfb3ef 100755
--- a/tools/preprocess.py
+++ b/tools/preprocess.py
@@ -251,6 +251,10 @@ def preprocess(path, preprocessors):
             command = config['preprocessors'][preprocessor]['command']
             if command[0].startswith('tools/'):
                 command[0] = os.path.join(script_dir, command[0][6:])
+            # These are implemented as `.cmd` files on Windows, which explicitly
+            # requires them to be run under `cmd /c`
+            if os.name == 'nt':
+                command = ["cmd", "/c"] + command
             process = subprocess.Popen(command, stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE, shell=True)
             data = process.communicate(input=data)[0]

I'm working around #33 to try and test it. With this patch applied I have successfully generated an output directory.

The issue appears to be that npm stores its scripts in .cmd files that are roughly equivalent to .sh files on Posix systems. However, since Windows has no concept of a shebang it doesn't actually know to run it. The trick is to run it under a shell, similar to how you might run bash -c. I'm not sure why shell=True doesn't actually accomplish this, but prefixing the command with cmd /c does the trick.

[jkent]

I will no longer using npx in favor of calling stub scripts with node directly for running the various tools. In the mean time, this has been merged.