search

layer5io / meshery-smp-action

GitHub Action for pipelining microservices and Kubernetes performance testing with Meshery
https://layer5.io/projects/nighthawk
Apache License 2.0
29 stars 22 forks source link

Stability issues wrt Scheduled Benchmark Tests on Self Hosted Runner #60

Closed hershd23 closed 2 years ago

hershd23 commented 2 years ago

Description

Multiple issues of the Scheduled Benchmark Tests on Self Hosted Runner generate different results

While here just one combination failed :- https://github.com/layer5io/meshery-smp-action/runs/7770247406 (linkerd-wrk-soak)

Here a couple failed :- https://github.com/layer5io/meshery-smp-action/actions/runs/2832186044

And here one runner startup itself failed :- https://github.com/layer5io/meshery-smp-action/actions/runs/2833335073

Expected Behavior

The behaviour should be consistent across Scheduled Benchmark Test run on self hosted runners. Given we are working with external hardware where connections can fail, ideally we should have a retry mechanism and we should work to considerably reduce the consistency

Logs

In the logs Auth seems to be a common culprit

Opening Meshery (http://localhost:31391) in browser.
Failed to open Meshery in browser, please point your browser to http://localhost:31391 to access Meshery.
authentication failed: Get "http://localhost:31[39](https://github.com/layer5io/meshery-smp-action/runs/7765562299?check_suite_focus=true#step:5:40)1/api/providers": dial tcp [::1]:31391: connect: connection refused
Verifying prerequisites...
Authentication token not found. please supply a valid user token with the --token (or -t) flag. or login with `mesheryctl system login`
Onboarding application... Standby for few minutes...
Error: Authentication token not found. please supply a valid user token with the --token (or -t) flag. or login with `mesheryctl system login`

This is seen for istio

Opening Meshery (http://192.168.49.2:32398/) in browser.
Failed to open Meshery in browser, please point your browser to http://192.168.49.2:32398/ to access Meshery.
Verifying prerequisites...
Adapter for required mesh not found
Onboarding application... Standby for few minutes...
rpc error: code = Unknown desc = no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"

Error from server (NotFound): namespaces "istio-system" not found
Error from server (NotFound): namespaces "istio-system" not found
Service Mesh: Istio - ISTIO
Gateway URL: [http://192.168.49.2:](http://192.168.49.2/)

Environment:

hershd23 commented 2 years ago

@gyohuangxin @leecalcote

leecalcote commented 2 years ago

@hershd23 thanks for this! @alphaX86 will you kindly investigate the auth issue? // @theBeginner86

alphaX86 commented 2 years ago

Rog' seems I need to see the script file again and tackle the auth issue

hershd23 commented 2 years ago

https://github.com/layer5io/meshery-smp-action/tree/master/.github/workflows/scripts @alphaX86

hershd23 commented 2 years ago

Just to point out the auth issue doesn't happen everytime, it's flaky but this seems to be one of the most clear reasons as to why random tests are failing

alphaX86 commented 2 years ago

Understood, I kinda figured where it went wrong, but still I need to investigate further...

The fix for this will be done by tomorrow for sure. Thanks @hershd23 👍

hershd23 commented 2 years ago

Great. Do let me know if the scripts need to change in lieu of your changes to meshery or ctl