facebookarchive / contest

Run continuous and on-demand system testing for real and virtual hardware
MIT License
32 stars 15 forks source link

Add exec test step plugin #303

Closed mimir-d closed 3 years ago

mimir-d commented 3 years ago

Description

The exec plugin runs a given binary or executable and determines whether the test passed based on criteria specified by options. The executables are run using a transport that abstracts the mechanism of launching the process.
More info in readme.md in plugin folder.

Testing

unit tests

> go test ./plugins/teststeps/exec
ok      github.com/facebookincubator/contest/plugins/teststeps/exec     0.003s

integration tests

codecov[bot] commented 3 years ago

Codecov Report

Merging #303 (c3b0e60) into main (3fbf31a) will decrease coverage by 2.55%. The diff coverage is 52.38%.

:exclamation: Current head c3b0e60 differs from pull request most recent head 5df58f4. Consider uploading reports for the commit 5df58f4 to get more accurate results Impacted file tree graph

@@            Coverage Diff             @@
##             main     #303      +/-   ##
==========================================
- Coverage   68.63%   66.08%   -2.56%     
==========================================
  Files         141      152      +11     
  Lines        7991     8970     +979     
==========================================
+ Hits         5485     5928     +443     
- Misses       1952     2400     +448     
- Partials      554      642      +88     
Flag Coverage Δ
e2e 48.69% <0.00%> (-1.60%) :arrow_down:
integration 57.95% <49.14%> (-1.81%) :arrow_down:
unittests 48.38% <13.43%> (-2.03%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
plugins/teststeps/exec/ocp_parser.go 27.14% <27.14%> (ø)
plugins/teststeps/exec/runner.go 43.54% <43.54%> (ø)
plugins/teststeps/exec/exec.go 32.60% <44.44%> (ø)
...gins/teststeps/exec/transport/ssh_process_async.go 50.29% <50.29%> (ø)
...lugins/teststeps/exec/transport/local_transport.go 54.54% <54.54%> (ø)
plugins/teststeps/exec/transport/transport.go 57.14% <57.14%> (ø)
plugins/teststeps/exec/transport/ssh_process.go 61.81% <61.81%> (ø)
plugins/teststeps/exec/transport/ssh_transport.go 63.41% <63.41%> (ø)
pkg/test/param_expander.go 66.66% <66.66%> (ø)
plugins/teststeps/exec/events.go 66.66% <66.66%> (ø)
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3fbf31a...5df58f4. Read the comment docs.

mimir-d commented 3 years ago

forgot a method rename in the last commit; the test was passing but it resulted in the controlled remote process hanging around because it wasnt actually killed (there is no wait for the controlled process to die, it just sends the sigkill)