search

bojand / ghz

Simple gRPC benchmarking and load testing tool
https://ghz.sh
Apache License 2.0
3.05k stars 273 forks source link

Recommendation for merging multiple reports #106

Open pritishshah opened 5 years ago

pritishshah commented 5 years ago

We want to have load test using ghz (go package). We also want to use ghz-web.

We integrated one RPC using runner, and wrote http client which upload a report to by calling api/projects/<project_identifier>/ingest end-point.

We have following requirements:

  1. Load test 3 different RPCs.
  2. For Each RPC, load test different scenario.

Based on what we read so far, we need to create a runner for each scenario, which is fine. We wanted to create one project for our service, and manage all RPCs underneath. Is it possible to provide grouping based on RPC?

If not, do you recommend to have separate project for each RPC OR each scenario of RPC?

Looking for recommendation around how to group these reports for given RPC and service together.

pritishshah commented 5 years ago

@bojand let me know if you find this confusing. you guys have developed awesome tool. really want to help here to make it to the next stage.

bojand commented 5 years ago

Hello, thanks for kind words regarding the tool 😄 . If I understand correctly... this is basically the hypothetical scenario:

We have a service, for example:

service UserService {
  rpc CreateUser (NewUserRequest) returns (User) {}
  rpc GetUser (Id) returns (User) {}
  rpc ListUsers (Empty) returns (stream User) {}
}

And we want to test each RPC calls, and we have different scenarios for each RPC, for example we may want to test ListUser with concurrency of 10, 100 and 1000, or vary some other parameters.

The question is whether all of this can be grouped together under a project "UserService". The simple answer is yes, but that would not make sense, at least the way the data relationships are set up right now. Right now "project" really represents a single conceptual "test" that represents something you would run as code changes and want to keep track of change in performance over time. So for example you could set up multiple projects "UserService - ListUser - Scenario A", "UserService - ListUser - Scenario B", "UserService - ListUser - Scenario C", and alike for other calls, like "UserService - GetUser - Scenario A", etc...

Originally I did have an idea for different structure of data, with additional level of encapsulation. Something like "Projects", "Tests", and "Reports". In this structure "Project" would be just a simple representation of some conceptual project or service, so like "UserService" and could add some description and perhaps settings. Then under tests you would create what we currently represent as "projects", and individual reports would live under there just as they do today.

Eventually I did not implement this as I felt it added an extra level or data that was not strictly necessary and I wanted the core functionality of the ghz-web project coded and working.

It is something I am open to but I think the current data structure is workable. Let me know if I understand the issue correct and other thoughts.

pritishshah commented 5 years ago

@bojand very nice explanation. The way I vision this web app is one centralized place for reporting load test results for all services. For now, its fine to setup multiple projects, based on each scenario, but once we want onboard more services, we may need one or more level of grouping, which allows us to manage different reports for each service at one location.

Possible hierarchy: Service > Build# > RPC > Tests > Reports.

  1. One project for each service.
  2. Every run of load test will have build# OR unique ID
  3. Each load test will run set of RPCs.
  4. Each RPCS will have different tests (scenarios).
  5. Each Scenario will have report.

I totally understand your thoughts, and idea is to support general audience. If we want to add more level of grouping, are you open for accepting contribution from us?

Another thing also we can do is, I can fork this repository, and make necessary changes in web application code.

Looking forward for your response.

bojand commented 5 years ago

Hello, sorry for taking so long to reply. I just want to make sure I understand the structure described above? Taking the user service from above, you describe something like this:

Is this something like what is proposed? I guess I am a little bit confused by the build # aspect mentioned in point 2.

As I mentioned I am open to restructure and improvements where they make sense, and for something that would be generally practical and usable. PR's are welcome. Changing the structure of data is not an insignificant change, so it would be good to narrow down the design and api's first before any substantial work is done.

As I get some free time I do plan on making some changes to ghz-web project in the future, but they would mostly be some frontend improvements, and maybe some feature requests such as adding thresholds.

pritishshah commented 5 years ago

@bojand apologies for delayed response.

We recently experimented around this. Having listing of services at top level would definitely help, that way we can see all projects (scenarios) inside that.

Point#2 is not needed, as it can be shoved in as part of name of the report.

If we allow service at the top-level, then ghz-web can be hosted for different services, and single place to view results of load tests. If we do this, then project has to go inside service.

e.g.

Service APIs:

  1. POST /api/services "create a service"
  2. GET /api/services "list services"
  3. GET /api/services/:sid "get a service by identifier"
  4. PUT /api/services/:sid "update a service by identifier"
  5. DELETE /api/services/:sid "delete a service by identifier"

New Project APIs (if we want to have service as top-level):

  1. POST /api/services/:sid/projects "create a project for given service identifier" OR POST /api/services/:sid/projects/:pid "add a project to given service (User has to create a project before invoking this API)"
  2. GET /api/services/:sid/projects "list all projects for given service"

Let me know your thoughts. Also, if we agreed to make this change, can give you PRs for adding these APIs.

pritishshah commented 5 years ago

@bojand checking with you again on this.

bojand commented 5 years ago

Hello, sorry I think I was away when you comment was posted and just missed it somehow.

The proposed API is fine by me, but I am still not sure I completely understand the proposed hierarchy heh. Would the rest of the API live under /api/services/:sid/... as is now? All the projects (which you could use as "RPC"s), reports, etc... for each "service"? We are just basically adding another level for a service above everything else?

That sound fine by me. PR's are welcome! But this may not be insignificant amount of work. Some technical work required:

Regards,

Bojan

pritishshah commented 5 years ago

yes we are adding extra layer. Currently, its hard to see all RPC tests under one umbrella when we have 10+ services. If you have only single service, then it becomes unnecessary wrapper.

With proposed way, UI will show all services firsts, and then when you click on service, it will show all RPC load tests as ghz is showing right now.

I will start working on this probably next week.

Should I give you separate pull requests for the technical work you mentioned?

bojand commented 5 years ago

Hello,

Thanks for willing to work on this. Yup a single PR for the complete feature, including all the steps for above, would be sufficient.

Looking forward to it :)