search

magnusbaeck / logstash-filter-verifier

Apache License 2.0
192 stars 27 forks source link

Error on Windows 10: fails to find an existing executable found among Logstash path candidates #71

Closed fbaligand closed 4 years ago

fbaligand commented 4 years ago

Hi,

I tried last version (1.5.1) on Windows 10. And when I start this command: C:\dev\elastic-stack\products\logstash-filter-verifier_1.5.1\logstash-filter-verifier.exe --logstash-output --logstash-path=C:\dev\elastic-stack\products\logstash-6.8.2\bin\logstash.bat testcase.json filter.logstash.conf

I get the following error: Error locating Logstash: no existing executable found among candidates: C:\dev\elastic-stack\products\logstash-6.8.2\bin\logstash.bat, /opt/logstash/bin/logstash, /usr/share/logstash/bin/logstash

What did I miss?

magnusbaeck commented 4 years ago

I suspect it's the conditional at https://github.com/magnusbaeck/logstash-filter-verifier/blob/1.5.1/logstash-filter-verifier.go#L96 that's always false on Windows and therefore causes it to not accept logstash.bat. Could you please run LFV with --loglevel DEBUG?

fbaligand commented 4 years ago

I confirm that it is this line: I had this debug log line: Logstash path candidate not an executable regular file: /path/to/logstash.bat

I also tried to run with cygwin, setting execute right on file with chmod. But it doesn't fix the problem.

Any idea to fix the issue?

magnusbaeck commented 4 years ago

I suggest we just skip those conditions if we're running on Windows, i.e. change the line to this:

if runtime.GOOS != "windows" && (!stat.Mode().IsRegular() || stat.Mode().Perm()&0111 != 0111) {
fbaligand commented 4 years ago

That would be great!

fbaligand commented 4 years ago

Hi @magnusbaeck,

I cloned the repo, made the change, and recompiled the binary (BTW, very easy!).

And when I launched again my test, I had the next level error :(

Here's the log with DEBUG level:

call C:\workspace\logstash-filter-verifier\logstash-filter-verifier.exe --logstash-output --logstash-path=C:/dev/logstash-6.8.2/bin/logstash.bat --logstash-version=6.8.2 --keep-env=JAVA_HOME --loglevel=DEBUG testcase.json filter.logstash.conf
2019/11/19 19:36:21 Reading test case file: %s (%s) testcase.json C:\dev\logstash-unit-test\testcase.json
2019/11/19 19:36:21 Logstash path candidate accepted: %s C:/dev/logstash-6.8.2/bin/logstash.bat
2019/11/19 19:36:21 Preparing configuration file directory %s with these files: %v C:\Temp\114979567\pipeline.d [filter.logstash.conf]
Running tests in testcase.json...
2019/11/19 19:36:21 Starting %q with args %q. C:/dev/logstash-6.8.2/bin/logstash.bat [-w 1 --debug -f C:\Temp\114979567\pipeline.d -b 1 -l C:\Temp\114979567\log --path.settings C:\Temp\114979567\config --path.data C:\Temp\114979567\data]
2019/11/19 19:36:21 Waiting for child with pid %d to terminate. 169312

Process output:
'findstr' is not recognized as internal or external command, an executable program or a commands file.
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54: warning: It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
[ERROR] 2019-11-19 19:36:28.514 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (GemspecError)
[!] There was an error while loading `logstash-core-plugin-api.gemspec`: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory
Does it try to require a relative path? That's been removed in Ruby 1.9. Bundler cannot continue.

 #  from C:/dev/logstash-6.8.2/logstash-core-plugin-api/logstash-core-plugin-api.gemspec:1
 #  -------------------------------------------
 #    end
 >  end
 #  # -*- encoding: utf-8 -*-
 #  -------------------------------------------

The process wrote nothing to its logfile.
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed

I don't reproduce this error with same version and same arguments on Ubuntu 18. Any idea about the problem?

magnusbaeck commented 4 years ago

Googling the "copy embedded jar" error message indicates that something's up with the temp directory. The volume containing the temp directory isn't full or inaccessible or anything? Presumably not the latter since LFV is clearly able to create C:\Temp\114979567 and copy various files to it. Could you try pointing the TEMP environment variable elsewhere?

But maybe it's "findstr is not recognized" that's the root cause? Have we screwed up the PATH variable somehow? What happens if you drop --keep-env=JAVA_HOME, does everything blow up because the JRE isn't found? What if you add --keep-env=PATH (even though that should be included by default).

fbaligand commented 4 years ago

Defining a custom TEMP folder does the job! Big thanks for your suggestion!

Some remarks:

Finally, thanks for your great work on this tool! This is really useful and efficient!

Speaking about tools, if you wish a completion tool for your Logstash pipeline configuration, I developed a Visual Studio Code extension for that: https://marketplace.visualstudio.com/items?itemName=fbaligand.vscode-logstash-editor

magnusbaeck commented 4 years ago

Great feedback, thanks! I don't have time to look into everything right now, but here's one finding:

My custom TEMP directory is used by Logstash itself, but not by logstash-filter-verifier.

You probably still have TMP set to C:\Temp. Quoting the documentation of os.TempDir():

On Windows, it uses GetTempPath, returning the first non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory.

fbaligand commented 4 years ago

Hi @magnusbaeck,

Thanks for your suggestion. I just added set TMP=%TEMP% and it works like a charm!

If you're interested to, I can submit a PR to bring windows compatibility. What do you think about?

magnusbaeck commented 4 years ago

Sorry for the delayed feedback. I've been ill and very busy.

And BTW, generated files are not removed after tests are finished.

Which files are left behind? I'm sure it's the annoying Windows behavior of not allowing deletion of files with open handles, but I'm curious exactly which files aren't deleted.

So that your Logstash Filter Verifier works, I have to put cygwin "bin" directory in my path, as it requires "diff" command that is not available on Windows 10. Maybe it would be nice to indicate it in README.md.

Yes. As you can tell I've never tested it on Windows.

In INFO logs, it would be nice to log a "SUCCESS" message for each test case passed successfully.

I disagree. If nothing is logged then the success of the testcase is implied. Adding a "success" line will double the amount of output from a successful run.

I look at your generated pipeline, and realize that you create C:\Temp\1234 temp directory, but set C:\Temp\6789 as output file path. Why not put this output file in your created directory?

No particular reason, probably just laziness.

finally, when I built logstash-filter-verifier, I didn't use make command (not available in Go SDK), but go build. Maybe it would be more relevant in Logstash Filter Verifier README?

Yes, I clearly forgot the readme file when I migrated to modules.

If you're interested to, I can submit a PR to bring windows compatibility.

Please do! Earlier I prepared but didn't push a commit for the executable check condition so it's up to you.

magnusbaeck commented 4 years ago

@fbaligand, were you planning on sending one or more PR for the issues you found? Just so I know if I should work on them myself.

fbaligand commented 4 years ago

Hi, Sorry for my late feedback.

So :

fbaligand commented 4 years ago

PR done!

fbaligand commented 4 years ago

Concerning documentation on how to build binary (go build), do you want a PR, or do you do it?

magnusbaeck commented 4 years ago

The first time I launched the tool, I saw no log message at all and thought that there was an error somewhere that prevents the tool to go to the end of its task.

Aha, right. We could have a summary line at the end, that'd be fine.

Concerning documentation on how to build binary (go build), do you want a PR, or do you do it?

I have a documentation overhaul planned anyway so I can include what we talked about.