Runscope plugin for Live Traffic Alerts

[mansilladev]

While most people think of Runscope for functional API testing and monitoring, there's another product feature called Live Traffic Alerts. It monitors live product traffic for matches based on any parts of the HTTP request or response, including header values, JSON or XML data, connection details and more.

A Runscope plugin for Kong would utilize the Runscope API to post API request/response data from Kong into a Runscope bucket, which in turn gets analyzed by Live Traffic Alerts. The API resource is Messages, and the documentation is here. The only configuration settings required are a bearer token (an application personal access token can be obtained from the Applications settings) and a bucket key (unique bucket identifier) which is a path-based parameter in the method.

I'd love to collaborate on this plugin.

[ahmadnassri]

:+1:

[harlow]

@mansilladev I'd be happy to help out on this too if you wanna pair or just do some code reviews

[sonicaghi]

@mansilladev here few good resources for a PR:

1. http://tylerneylon.com/a/learn-lua/
2. https://getkong.org/docs/0.5.x/plugin-development/
3. http://streamdata.io/blog/developing-an-helloworld-kong-plugin/ (not fully up to date)
4. https://getkong.org/plugins/galileo/ (implementation looks similar to Galileo)

[mansilladev]

Digging into the log-serializers -- how is the response body (content) accessed?

[thibaultcha]

Aggregated in the body_filter_by_lua phase: https://github.com/Mashape/kong/blob/master/kong/plugins/mashape-analytics/handler.lua#L74

[mansilladev]

@thibaultCha Woot. Thanks.

I've built a plugin that was based off of http-log. Works great. However, I did just notice that mashape-analytics has a batching mechanism. I'm going to implement that, and then hopefully, have something for folks to try out.

[thibaultcha]

I recommend that you use lua-resty-logger-socket for the batching mechanism.

[thibaultcha]

BTW, I am concerned that multiple logging plugins perform the response body aggregation logic. Because this operation consumes memory and resources for each request, and if someone has, say, the mashape-amalytics plugin enabled along another plugin doing response body aggregation, one of the two is doing work already done by another, eating up resources from the Nginx worker.

The only solution I can see to that is for "official" plugins to aggregate in a namespaced ngx.ctx variable, only if that variable was not initialized previously. Hence, only one plugin would aggregate the value in body_filter_by_lua, but all of them can access the response's content in log_filter_by_lua.

[mansilladev]

Well, if we can settle on not just the namespace, but also common content/operations that would be universally useful (right now it's req_body, res_body and req_post_args that are in ngx.ctx.analytics) then I'd be totally down with adhering to that pattern/standard to access those variables for the Runscope plugin.

[mansilladev]

BTW: https://github.com/mansilladev/kong/ -- initial commit. After enabling the plugin runscope-log in the kong.yaml, add the plugin via the admin API:

Request body:

{
  "name": "runscope-log",
  "config.access_token": "{{yourRunscopePersonalAccessToken}}",
  "config.bucket_key": "{{yourRunscopeBucketKey}}",
  "config.log_body": true
}

Make some calls through Kong and the traffic should flow into your bucket.

[thibaultcha]

Awesome! Do you plan on making a PR? I just see a few tweaks that we can discuss in it, otherwise it looks good.

I think we can merge this without taking into account the unique request/response bodies we discussed, and create an issue to enhance that in the near future.

[ahmadnassri]

https://getkong.org/plugins/runscope/