search

nuweba / faasbenchmark

Run or add generic tests to accurately benchmark FaaS providers. https://faastest.com/
Apache License 2.0
36 stars 7 forks source link

Cannot start GUI and CLI #43

Closed joe4dev closed 4 years ago

joe4dev commented 4 years ago

GUI

Starting the GUI currently fails due to a missing debug bool in NewGlobalConfig:

# github.com/nuweba/faasbenchmark/tui
tui/main.go:89:40: not enough arguments in call to config.NewGlobalConfig
    have (provider.FaasProvider, string, report.Top)
    want (provider.FaasProvider, string, report.Top, bool)

Passing an extra bool value (e.g., true) at line 89 fixes the GUI startup. I can then successfully deploy and undeploy stacks on AWS but I couldn't figure out how to start actual performance tests.

CLI

The command go run main.go immediately returns (with successful exit code). How do I invoke faasbenchmark list or faasbenchmark run? faasbench-cli

Is go run main.go supposed to open an interactive shell or should I pipe faasbenchmark commands into go run main.go? Thanks for clarification!

nwb-ilan commented 4 years ago

Thanks for filing the issue!
We'll be looking into it.
In the meantime, an example for using the cli would be:
go run main.go run aws Example1
This will deploy a simple stack with one function to aws, invoke it 3 times, and delete the stack.
There are other test stacks you can try as well, for example IncreasingCPULoadLvl1.
To get a full list of tests and their description you can use:
go run main.go list tests

nwb-ilan commented 4 years ago

fixed #44

nwb-ilan commented 4 years ago

@joe4dev let me know if you have any other issues

joe4dev commented 4 years ago

Thanks for the quick comment and fix.

The tests run nicely on AWS after ensuring the user has the role AWSLambdaRole (https://stackoverflow.com/questions/37498124/accessdeniedexception-user-is-not-authorized-to-perform-lambdainvokefunction).

The Google setup has some undocumented conventions:

The Azure credentials setup works as described in the Serverless docs: https://serverless.com/framework/docs/providers/azure/guide/credentials/ However, trying to run any azure test (e.g., go run main.go run azure Example1) yields the following error:

2020-01-31T12:17:43.012+0100    INFO    main    config/global.go:46 starting tests
failed reading new stack: failed parsing stack: failed to capture stdout or stderr
exit status 1
nwb-ilan commented 4 years ago

Thanks for the quick comment and fix.

The tests run nicely on AWS after ensuring the user has the role AWSLambdaRole (https://stackoverflow.com/questions/37498124/accessdeniedexception-user-is-not-authorized-to-perform-lambdainvokefunction).

The Google setup has some undocumented conventions:

  • Credentials are assumed to be located under /tmp/.gcloud/faasbenchmark.json
  • The project is assumed to be called faastest
  • Need to enable "Google Cloud Functions API" and "Cloud Deployment Manager V2 API" in the API & Services Dashboard: https://console.cloud.google.com/apis/dashboard
  • Some permission setup is required to fix the error Error: Required 'deploymentmanager.deployments.list' permission for 'projects/faastest' (see serverless/serverless-google-cloudfunctions#52)

The Azure credentials setup works as described in the Serverless docs: https://serverless.com/framework/docs/providers/azure/guide/credentials/ However, trying to run any azure test (e.g., go run main.go run azure Example1) yields the following error:

2020-01-31T12:17:43.012+0100  INFO    main    config/global.go:46 starting tests
failed reading new stack: failed parsing stack: failed to capture stdout or stderr
exit status 1

Thanks for the detailed report.
We'll be looking into all of these issues (three of them linked above).