Open tyler-mauthe-hs opened 7 years ago
Sorry, if it wasn't clear I'd like the load generation abstraction to also be a sort of adaptor -- the current method of internally generated random load would be supported in this way.
Ah, I see I'd also need to add the ability to attach the Lambda to a subnet in a VPC to enable private testing.
I was doing some POC testing of this concept. While SNS does work for this I think I'll get higher throughput and more optimal Lambda utilization by using Kinesis streams.
@tyler-mauthe-hs I'm currently in the process of defining the roadmap for OLX internal development of GOAD... which for the moment is just focusing on the HTTP use case. Indeed we want to go for a solution that would break the current 5min limit related to the current implementation. Please, let me know with a pvt message your email address and I'll try to look you in for a planning meeting.
@gserra-olx - Sorry for the delayed response! I've added my email to my profile.
To allow for plugging of different load- and functional-tests while keeping goad as a platform to orchestrate we'll introduce a unified interface for goad runners.
Since goad makes heavy use of AWS lambda functions we try to keep the variety of options to run code in AWS untouched. We'll try to specify the minimal interface for goad to be able to create a lambda function and the mechanism to communicate back the results of the load test run inside the lambda function.
We'll use the existing SQS queue infrastructure as a means to signal results back to the cli client.
The current default runner for goad looks somewhat like this:
defaut-runner
└── lambda <- folder mapped to /var/task inside the lambda
├── goad-lambda <- compiled go program executed by Javascript
└── index.js <- Javascript lambda function
Current parameters passed into the runner are the following:
general settings | specific to default-runner |
---|---|
sqsurl | concurrency |
queue-region | requests |
aws-region | execution-time |
client-timeout | |
frequency | |
method | |
body | |
header | |
url |
Apart from the aws and sqs settings, parameters seem specific to the current implementation of goads default-runner.
The results written to the sqs queue have the following scheme:
TotalReqs int `json:"total-reqs"`
TotalTimedOut int `json:"total-timed-out"`
TotalConnectionError int `json:"total-conn-error"`
AveTimeToFirst int64 `json:"ave-time-to-first"`
TotBytesRead int `json:"tot-bytes-read"`
Statuses map[string]int `json:"statuses"`
AveTimeForReq int64 `json:"ave-time-for-req"`
AveReqPerSec float32 `json:"ave-req-per-sec"`
AveKBytesPerSec float32 `json:"ave-kbytes-per-sec"`
Slowest int64 `json:"slowest"`
Fastest int64 `json:"fastest"`
Region string `json:"region"`
FatalError string `json:"fatal-error"`
Finished bool `json:"finished"`
FinishedLambdas int `json:"finished-lambdas"`
FinishedLambdas
is only used locally and should not be part of the interface. Statuses
is never displayed and should also not become part of the interface. The other values could be represented more concise since most of them are calculated properties. This interface has the implied property that if no result is written for some time the cli will time out and stop execution.
We pick the existing interface and abstract the necessary parts to allow for a more generalized use-case.
The configuration of the extended behavior should be inside the goad.ini file in the current working directory. We'll introduce a new section to describe goad tasks to execute.
Individual folders/paths will hold the custom runners that are locally implemented. You can specify folders that hold configurations for individual tasks in the configuration. Goad loads them to s3 before executing the lambda functions. The lambda functions can then access the payload from s3.
Though the ini format is somewhat limited we can map settings of individual tasks like this:
[task]
[task.run-simple-load]
runner = goad-siege-runner
task-directory = tests/run-simple-load
concurrency = 100
task-specific-variable = xyz
[task.run-simple-load.regions]
us-east-1 ;N.Virginia
[task.run-complex-sequence]
runner = goad-sequence-runner
The working directory for goad whould then look like this:
goad-workdir
├── custom-sequence-runner
│ └── lambda
│ └── index.js
├── goad.ini
├── goad-siege-runner
│ └── lambda
│ └── index.js
└── tests
└── run-simple-load
└── url-list.txt
A goad runner specifies the executable lambda function launched by goad. You can define a custom runner by creating a directory containing the code to execute inside the lambda function like this:
custom-sequence-runner
└── lambda <- folder mapped to /var/task inside the lambda
├── static-data-payload <- data passed over to the lambda
└── index.js <- Javascript lambda function
The new execution model allows us to place a static configuration inside the AWS lambda. The default runner makes used of this to place the goad-lambda binary alongside the Javascript.
We will reduce the interface to pass parameters as environment variables to the lambda function, as recommended by the lambda documentation. We pass along by default the aws-queue
, queue-region
and s3-payload
if a folder was uploaded to s3.
Also we allow for any key value pair set by the user to be passed to the lambda function as environment variable. We would pass in the default runners parameters this way and allow user created runner to have various configuration options.
To test the implementation of custom runners we can create as dry-run option in goad. This option would run the selected runner or the default runner inside the locally simulated lambda container and not on aws.
This allows for easier development and testing of new features and runners.
The goad default-runner has a build in mechanism to resume execution on a new lambda function if it runs into the aws timeout. Custom runners would need to recreate a similar behavior.
I suggest a four step route to implement a first version of the interface:
Awesome stuff -- thanks @cwaltken-edrans!
I'll take a stab at writing something up based on this spec using Mongo's official nodejs client: https://mongodb.github.io/node-mongodb-native/
I made an attempt at implementing the interface, I wasn't able to perform my test -- but I think the interface is generally sound.
Some comments:
Ah great catch on your second point there Tyler. I think we should put that into the interface configuration and keep the current values as defaults. 👍
In that same location we could expose the connection to the VPC interface as mentioned in the AWS docs.
ok, this is on hold until we clean up all the other tickets... the current data aggregation model & metrics reporting is broken
Hello! I'm thinking of building out a very similar load testing platform, but I figure I'll try pitching the difference here to try and avoid XKCD#927. Goad seems like the coolest and closest tool to what I'd like, so hopefully we can come to an agreement.
At a high level, what I'd like to do is to have goad emit it's load via an adaptor which would be specified when launching the swarm. This adaptor would send load in various formats to the specified target. In addition to refactoring the existing HTTP driver out, I'd also be adding a driver for MongoDB.
Sending different types of load means the way load is generated also needs to be abstracted. In the spirit of keeping it Lambda-native, I'd like to suggest ingesting load from SNS.
So goad will create an SNS topic and attach the Lambdas (in VPC / subnets if specified) to the topics and start emitting load via SNS.
Let me know if you like this at a high level and I'll start to get familiar with the codebase and propose an implementation.