Add Support for Other Protocols

[tyler-mauthe-hs]

Hello! I'm thinking of building out a very similar load testing platform, but I figure I'll try pitching the difference here to try and avoid XKCD#927. Goad seems like the coolest and closest tool to what I'd like, so hopefully we can come to an agreement.

At a high level, what I'd like to do is to have goad emit it's load via an adaptor which would be specified when launching the swarm. This adaptor would send load in various formats to the specified target. In addition to refactoring the existing HTTP driver out, I'd also be adding a driver for MongoDB.

Sending different types of load means the way load is generated also needs to be abstracted. In the spirit of keeping it Lambda-native, I'd like to suggest ingesting load from SNS.

So goad will create an SNS topic and attach the Lambdas (in VPC / subnets if specified) to the topics and start emitting load via SNS.

Let me know if you like this at a high level and I'll start to get familiar with the codebase and propose an implementation.

[tyler-mauthe-hs]

Sorry, if it wasn't clear I'd like the load generation abstraction to also be a sort of adaptor -- the current method of internally generated random load would be supported in this way.

[tyler-mauthe-hs]

Ah, I see I'd also need to add the ability to attach the Lambda to a subnet in a VPC to enable private testing.

[tyler-mauthe-hs]

I was doing some POC testing of this concept. While SNS does work for this I think I'll get higher throughput and more optimal Lambda utilization by using Kinesis streams.

SNS

• SNS API is slow and a single Lambda is invoked for each message send -- if each message is a unit of load then this is a lot of invocations
• With SNS, you will spawn concurrent Lambda executions until you exceed your concurrent execution limit
• Single message per invocation negates any benefit from concurrency options (like spawning 1k goroutines per Lambda)

Kinesis

• Lambda trigger only invokes one Lambda per shard - implicit backpressure to avoid exceeding concurrent execution limit
• Lambda receives a batch of work from the Kinesis stream - enables concurrency benefits
• Can dispatch batches of work to Kinesis through various options

[gserra-olx]

@tyler-mauthe-hs I'm currently in the process of defining the roadmap for OLX internal development of GOAD... which for the moment is just focusing on the HTTP use case. Indeed we want to go for a solution that would break the current 5min limit related to the current implementation. Please, let me know with a pvt message your email address and I'll try to look you in for a planning meeting.

[tyler-mauthe-hs]

@gserra-olx - Sorry for the delayed response! I've added my email to my profile.

[cwaltken-edrans]

GOAD Lambda Runners API

To allow for plugging of different load- and functional-tests while keeping goad as a platform to orchestrate we'll introduce a unified interface for goad runners.

Prerequisites

Since goad makes heavy use of AWS lambda functions we try to keep the variety of options to run code in AWS untouched. We'll try to specify the minimal interface for goad to be able to create a lambda function and the mechanism to communicate back the results of the load test run inside the lambda function.

We'll use the existing SQS queue infrastructure as a means to signal results back to the cli client.

The current GOAD runner

The current default runner for goad looks somewhat like this:

defaut-runner
└── lambda  <- folder mapped to /var/task inside the lambda
    ├── goad-lambda  <- compiled go program executed by Javascript
    └── index.js     <- Javascript lambda function

Parameters passed in

Current parameters passed into the runner are the following:

general settings	specific to default-runner
sqsurl	concurrency
queue-region	requests
aws-region	execution-time
	client-timeout
	frequency
	method
	body
	header
	url

Apart from the aws and sqs settings, parameters seem specific to the current implementation of goads default-runner.

Results returned

The results written to the sqs queue have the following scheme:

TotalReqs            int            `json:"total-reqs"`
TotalTimedOut        int            `json:"total-timed-out"`
TotalConnectionError int            `json:"total-conn-error"`
AveTimeToFirst       int64          `json:"ave-time-to-first"`
TotBytesRead         int            `json:"tot-bytes-read"`
Statuses             map[string]int `json:"statuses"`
AveTimeForReq        int64          `json:"ave-time-for-req"`
AveReqPerSec         float32        `json:"ave-req-per-sec"`
AveKBytesPerSec      float32        `json:"ave-kbytes-per-sec"`
Slowest              int64          `json:"slowest"`
Fastest              int64          `json:"fastest"`
Region               string         `json:"region"`
FatalError           string         `json:"fatal-error"`
Finished             bool           `json:"finished"`
FinishedLambdas      int            `json:"finished-lambdas"`

FinishedLambdas is only used locally and should not be part of the interface. Statuses is never displayed and should also not become part of the interface. The other values could be represented more concise since most of them are calculated properties. This interface has the implied property that if no result is written for some time the cli will time out and stop execution.

A Plugable Interface

We pick the existing interface and abstract the necessary parts to allow for a more generalized use-case.

Configuration

The configuration of the extended behavior should be inside the goad.ini file in the current working directory. We'll introduce a new section to describe goad tasks to execute.

Individual folders/paths will hold the custom runners that are locally implemented. You can specify folders that hold configurations for individual tasks in the configuration. Goad loads them to s3 before executing the lambda functions. The lambda functions can then access the payload from s3.

Though the ini format is somewhat limited we can map settings of individual tasks like this:

[task]

[task.run-simple-load]
runner = goad-siege-runner
task-directory = tests/run-simple-load
concurrency = 100
task-specific-variable = xyz

[task.run-simple-load.regions]
us-east-1 ;N.Virginia

[task.run-complex-sequence]
runner = goad-sequence-runner

The working directory for goad whould then look like this:

goad-workdir
├── custom-sequence-runner
│   └── lambda
│       └── index.js
├── goad.ini
├── goad-siege-runner
│   └── lambda
│       └── index.js
└── tests
    └── run-simple-load
        └── url-list.txt

Goad Runner

A goad runner specifies the executable lambda function launched by goad. You can define a custom runner by creating a directory containing the code to execute inside the lambda function like this:

custom-sequence-runner
└── lambda  <- folder mapped to /var/task inside the lambda
    ├── static-data-payload  <- data passed over to the lambda
    └── index.js             <- Javascript lambda function

The new execution model allows us to place a static configuration inside the AWS lambda. The default runner makes used of this to place the goad-lambda binary alongside the Javascript.

We will reduce the interface to pass parameters as environment variables to the lambda function, as recommended by the lambda documentation. We pass along by default the aws-queue, queue-region and s3-payload if a folder was uploaded to s3.

Also we allow for any key value pair set by the user to be passed to the lambda function as environment variable. We would pass in the default runners parameters this way and allow user created runner to have various configuration options.

Local Execution to Test GOAD Runners

To test the implementation of custom runners we can create as dry-run option in goad. This option would run the selected runner or the default runner inside the locally simulated lambda container and not on aws.

This allows for easier development and testing of new features and runners.

AWS Execution Time-Limit

The goad default-runner has a build in mechanism to resume execution on a new lambda function if it runs into the aws timeout. Custom runners would need to recreate a similar behavior.

Roadmap

I suggest a four step route to implement a first version of the interface:

1. We'll create the dry-run functionality to support development and testing of the feature.
2. We'll recreate the existing default runner as a plugin.
3. We'll implement the siege runner with support for upload of a file list with URLs to load test in parallel with goad.
4. We document the solution and create a way to share implementations of goad runners.

[tyler-mauthe-hs]

Awesome stuff -- thanks @cwaltken-edrans!

I'll take a stab at writing something up based on this spec using Mongo's official nodejs client: https://mongodb.github.io/node-mongodb-native/

[tyler-mauthe-hs]

I made an attempt at implementing the interface, I wasn't able to perform my test -- but I think the interface is generally sound.

Some comments:

• I need to attach my Lambda to a VPC in order to access the resource I want to test (unlike a website, my MongoDB instance isn't publicly accessible 😄 ).
• A number of assumptions are made here which might be better made configurable (lambda name, handler, etc).
• It will probably be very helpful for me to put up a wiki page that outlines the process I went through to create my package, I'll keep notes on how I did things to help the community. 🙌
• It would also probably be helpful to have some helper modules written in various Lamba-supported languages that define this standard interface and any common helper functions (like writing results to the SQS).

[cwaltken-edrans]

Ah great catch on your second point there Tyler. I think we should put that into the interface configuration and keep the current values as defaults. 👍

In that same location we could expose the connection to the VPC interface as mentioned in the AWS docs.

[gserra-olx]

ok, this is on hold until we clean up all the other tickets... the current data aggregation model & metrics reporting is broken