Using Karate to test cli

[maxandersen]

Thought that our discussion could be useful as an issue to not get lost in the dark depths of twitter.

My initial goal is to find a way to just test my basic jbang cli; but same could be used for testing lets say kubectl, git or even the karate bash script it self.

i.e. to test that jbang --version always returns your version string format (i.e. 0.32.1)

Given cmd jbang
And arg '--version'
When cmd run
Then exit 0
And match out \d+\.\d+\.\d+

or error situations:

Given cmd jbang
And arg 'filedoesnotexist.java'
When cmd run
Then exit -1
And match err 'Could not read `filedoesnotexist.java`'

and be able to combine both:

Given cmd jbang
And arg '--someflag'
And arg '--init'
And arg 'hello.java'
When cmd run
Then exit 0
And match out '`hello.java` initialized.'
And match err 'WARN: Unknown argument `--someflag`'

make the cmd use cmd /c on windows and sh -c on linux/osx by default and you got a cross-platform cli testing tool.

Combine it with karate nice to/from csv/xml/json/etc. and expression tests then wether your cli streams content on out/err or files created karate seem to have nice ways to assert on that.

== Stream/async tests

beyond these basics could be to add some notion of making assertions against out and err as streams - not sure if karate has similar to assert against streaming http requests/responses ?

[ptrthomas]

I think all of this is now possible. here's the updated gist: https://gist.github.com/ptrthomas/b0f81875e30390e7c8c097b9a8e91b4e

out and err as streams

the ping example already does it - it intercepts each line of console output

make the cmd use cmd /c on windows and sh -c on linux/osx by default

yes that makes sense, we already have a util that can tell you which os we're on

[maxandersen]

I'll give develop branch a go - but how about adding the syntactic sugar level to make the cli testing less verbose ?

[ptrthomas]

@maxandersen when I started Karate - I was in "keyword" mode, which is how the API testing looks like, e.g. see the hello world: https://gist.github.com/ptrthomas/d5a2d9e15d0b07e4f1b46f692a599f93

over time, Karate evolved to do UI testing and here I started adding JS-like API-s - the 2 primary advantages are simply because it is more complex, you need to do "fluent-api" style method-chaining and you can start even passing functions around. this I think is needed for advanced CLI testing. for e.g. this code below. BTW this is similar how we do web-sockets testing

* def listener = 
"""
function(line) { 
  var temp = karate.get('count');
  if (line.contains('bytes from')) {
    temp = temp + 1;
    karate.set('count', temp);
    karate.log('count is', temp);
  }
  if (temp == 5) {    
    var proc = karate.get('proc');
    karate.signal(proc.buffer);
  }
}
"""
* def proc = karate.fork({ args: ['ping', 'google.com'], listener: listener })
* def output = karate.listen(10000)
* print 'console output:', output

see how everything is tied together, the process, the listener and the wait for process to "signal" etc. trying to fit all that into syntax sugar will complicate things IMO and be very hard to implement. I also have a selfish motive of not bloating the keyword surface-area of karate, this strategy of having "less common" stuff hanging off API "helper" objects has served the project well

that said - if this is really compelling and there's a lot of interest, we could take the concepts and create a Karate "extension" (or CLI specific clone) as a separate OSS project - but I personally would prefer karate to continue on it's path to be a "one stop shop"

for example, the reason I worked on karate.fork() and got it to this point is because I am currently trying to solve for Windows app automation and hopefully eventually even Mac and Linux. this may be of interest to you because we have the capability to call windows api-s and dll-s (via JNA) but it is a separate, optional karate "module": https://github.com/intuit/karate/tree/develop/karate-robot

I suggest we get things working with the API - once that works we can level-up to thinking if "keywords" make sense.

[maxandersen]

ack - I get the power and I get why you would want that for the more complex scenarious.

I can just say that if the testing scripts stays this verbose there would be not much value in me using karate for basic cli testing. It would then be as complex to just do it in java (with a small utility wrapper). For me a lot of cli testing (imagine cli tests of git, karate, docker, kubectl) where its just one command and then verify output. would be 10x fold much more readable using the API/fluent apis.

But definitely +1 on at least adding the capability to launch/fork and testing on stdout/err - that is definitely useful to do.

[ptrthomas]

@maxandersen yes - agree that a small java utility may be sufficient - but I think it is a fair amount of work. people tend to forget that karate has bunch of things, assertions, parallel execution, the reports, data-driven testing, a debugger etc.

readability is good to have, but especially in this domain - I would argue that you are not solving for a lot of non-tech people, right :)

anyway I will work on the stderr and smart-os-detection, but also do take a look at these:

https://twitter.com/KarateDSL/status/1144458169822806016 - you can wrap the API into nice readable functions - so it comes close to your examples, just a few extra round brackets. for example:

Given exec(['jbang', '--someflag', '--init', 'hello.java'])
Then assert exitCode == 0
And match sysOut == 'foo'
And match sysErr 'WARN: Unknown argument `--someflag`'

it just struck me that the exec() function can set context variables e.g. karate.set('exitCode', proc.exitCode) so you actually can get very close to your ideal state. WDYT ?

[maxandersen]

that is getting pretty close yes :)

[maxandersen]

just tried it using develop and getting this is awesome :)

great to see the output directly in the report.

Definitely lots of potential here!

btw. is there a way to rewrite assert exitCode == 0 to make the report include what exitCode actually was ?

[maxandersen]

btw. is there a way to rewrite assert exitCode == 0 to make the report include what exitCode actually was ?

found it match proc.getExictCode() == 1 and you get it. awesome.

[ptrthomas]

awesome.

great to see the output directly in the report.

oh yes, that is one of the selling points !

[maxandersen]

readability is good to have, but especially in this domain - I would argue that you are not solving for a lot of non-tech people, right :)

I get what you are saying but like - with this I could see users and even my self much faster write tests just because lot of the superflous syntax is removed.

And would also be nice to use the tests as a kind of documentation of what is feasible - here if users can read and even copy the command under test to their shell that would be nice.

i.e.

Given exec(['jbang', '--someflag', '--init', 'hello.java'])
Then assert exitCode == 0
And match sysOut == 'foo'
And match sysErr 'WARN: Unknown argument `--someflag`'

is super close - even better would be:

Given command 'jbang --someflag --init hello.java'
Then assert exitCode == 0
And match sysOut == 'foo'
And match sysErr 'WARN: Unknown argument `--someflag`'

would be even better.

But that is minor - if I can get the first part up and it launch the same on windows as linux/osx then I'm definitely going to use it instead of my current bash only testing

[ptrthomas]

@maxandersen just checked in the changes to see system.err separately. and the autoPrefix of sh -c is working - you may have to test windows. also I couldn't test system.err was too lazy to simulate - let me know what would be an easy way

and this works. note that I renamed the getter for buffer. this discussion has been super-useful because it had never occurred to me to use karate.set() in this fashion, I think it is pretty neat !

Feature:

Background:
* def command =
"""
function(line) {
  var proc = karate.fork({ redirectErrorStream: false, useShell: true, line: line });
  proc.waitSync();
  karate.set('sysOut', proc.sysOut);
  karate.set('sysErr', proc.sysErr);
  karate.set('exitCode', proc.exitCode);
}
"""

Scenario: java cli
* command('java -version')
* match sysOut == '#regex java version (.+[\\r\\n])+'
* assert exitCode == 0

Scenario: jave fail
* command('java -foo')
* match sysOut contains 'Unrecognized option: -foo'
* assert exitCode == 1

EDIT: that command() function can even be injected into the "global scope" from karate-config.js like this

var command = read('classpath:utils/command.js');
return { command: command };

so you get your custom DSL (well almost).

[maxandersen]

trying to use this now and hitting weird error which not sure if type error or something.

I did this:

Feature:

Background:
* def command =
"""
function(line) {
  var proc = karate.fork({ redirectErrorStream: false, useShell: true, line: line });
  proc.waitSync();
  karate.set('out', proc.sysOut);
  karate.set('err', proc.sysErr);
  karate.set('exit', proc.exitCode);
}
"""

Scenario: jbang version
* command('jbang version')
* match out == '#regex .*'
* match exit == 0

Scenario: fail on missing file
* command('jbang notthere.java')
* match out contains 'Could not read script argument notthere.java'
* assert exitCode == 1

match exit == 0 works, but assert exit == 0 evaluates to false - how come ?..

[maxandersen]

another issue, when running on windows it runs in a loop ...with a tons of hs_err*log files with out of memory errors.

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=

[maxandersen]

var command = read('classpath:utils/command.js'); return { command: command };

how would that command.js look like ? couldn't get it to work in my attempts.

[ptrthomas]

@maxandersen JS can be just this part, the fn name is optional to keep IDE-s quiet and is ignored. more details here:

function fn(line) {
  var proc = karate.fork({ redirectErrorStream: false, useShell: true, line: line });
  proc.waitSync();
  karate.set('out', proc.sysOut);
  karate.set('err', proc.sysErr);
  karate.set('exit', proc.exitCode);
}

example: https://github.com/intuit/karate/blob/master/karate-junit4/src/test/java/karate-config.js

[ptrthomas]

@maxandersen maybe exit is a reserved word in JS or Nashorn pulls all of System into scope :)

[maxandersen]

@maxandersen maybe exit is a reserved word in JS or Nashorn pulls all of System into scope :)

but why isn't osx affected then ?

btw. I got my current progress on migrating to Karate at https://github.com/jbangdev/jbang/tree/master/itests

Things are looking good thus far - but for some reason I never get stdErr output even though my cli are printing directly to it. Trying to figure out why.

[maxandersen]

opened issue on the stderr thing specifically at https://github.com/intuit/karate/issues/1195

[ptrthomas]

0.9.6 released

[Rajpratik71]

HI @ptrthomas ,

On the same topic, does karate support passing some arguments to CLIs on runtime?

[ptrthomas]

@Rajpratik71 as far as I know yes. refer: https://stackoverflow.com/a/62911366/143475