How do you deal with user session to replay the traffic correctly?

[lbdremy]

Hello @buger,

Thanks for writing gor. I would like to use gor to replay the production traffic to the staging server but I have an issue doing this, I'm gonna explain.

Each request played has an access token, the access token is generated when an user successfully signs in. The generation of this access token is completely random so when the login request is played on the staging server the access token generated is different from the access token generated by the production server. The environments don't share and can't share/sync (in my situation) the same database where the access tokens are stored thus the only way I see is to maintain a mapping between the access token in production and the corresponding access token in staging.

Can I plug something into gor to do this? And do we have access to the response in order to store the access tokens generated ?

Thanks

[buger]

Currently i'm working on branch which will support middleware, basic idea is quite simple, middleware can be any program that accepts requests and responses on STDIN and emits modified requests on STDOUT https://github.com/buger/gor/tree/input-modifier. So workflow will look like this:

Script receives original request, then original and replayed responses, on the first response you remember mapping between original token and token received from replayed response, and for all future requests rewrite token value.

[lbdremy]

The idea of middleware is exactly what's needed there. We have implemented a solution which uses gor to replay the traffic to a proxy server (where the request is transformed) before the traffic is sent to the staging server.

Thanks for your reply

[buger]

I will still keep this issue open since this is common question, until it will be fixed.

[gmontard]

Hello @buger, firstable thanks for your great work!

We have a great use case where the middleware could be really useful - our sessions data live inside the DB, so we'd like to use the middleware to query our DB with the session_id passed in the headers of the request, grab the production session data from the DB and then forward it to our staging env inside a new header.

I guess this would require a few line of SQL query in GO inside a middleware addon.

Do you plan to let us add our own middleware, I've read your current code but I don't really see how to extend the available middleware without forking the repository ? (I'm not an export in Go..)

Thanks!

[buger]

@gmontard @lbdremy Happy to announce working middleware draft: https://github.com/buger/gor/pull/162

Any testing and feedback invaluable. Read pull request PR for documentation and see examples and specs. Links to binaries included into PR description as well. Thanks!

[buger]

@gmontard @lbdremy middleware released in 0.10.0

[gmontard]

:+1:

[gmontard]

Hello @buger,

I've started playing with the middleware, it seems to work great but I noticed a strange behavior when using a regexp. If you specific a regexp to disallow URL for example, the middleware still catch everything, I guess the regexp is applied too late and should be before the middleware is called, what do you think?

Thanks,

Guillaume.

[buger]

@gmontard yeah it could be, for now disallow this urls in your own middleware: just not send this requests to stdout.

[gmontard]

Yes sure I'll do that for the time being but if you can change this in a futur release I guess it will be a nice improvement.

Thanks.

[buger]

@gmontard just curios, what is your first thoughts about middleware? Did you finished integration?

[buger]

Here is project which does similar work and inspired by Gor https://github.com/Springest/rack-request_replication

[GiovanniPaoloGibilisco]

Hi, revamping an old but still actual thread.

We are trying to build a middleware that eases the integration of GoReplay into a performance test environment. We are now experiencing ploblems correlating Cookies among requests and responses due to the Open Loop nature of GoReplay when replaying traffic from a recorded file.

We have described the issue in details, here but the main point is that GoReply is forwarding requests to the middleware as soon as it reads them from the file so the middleware should not only modify the content of the request to update Cookies according to those received by the replayed responses (like all the availbale examples do), but also modify the time field GoReplay uses to schedule the request.

OT: @buger, you can also have a look at the way we use the prometheus client for python to export statistics on the replayed responses if you like.

[nikolai-derzhak-distillery]

@GiovanniPaoloGibilisco It seems it will make many queries in replayed traffic rejected. Do you have any work arounds ? As we would like to load test cluster. It means we always will hit that point when responses are slower then next queries. On other side this is how we could measure cluster performance. RPS at which it can handle traffic without hiccups will be about the same level of performance as source cluster. Does it sound right ?

[nikolai-derzhak-distillery]

Oh ! I believe you did this in your correlation middleware. Nice ! But I agree that would be great to have 'native' support. At least I feel it can be discussed.

https://github.com/GiovanniPaoloGibilisco/GorMW/blob/master/examples/correlation.py

[karussell]

The middleware works fine for us, except in the described situation with replay responses being slower then follow-up requests. Have explained it here.

Could it be that the provided searchResponses in the JS middleware has the same problem if the follow up messages of the authentication request arrive too fast? Would it make sense to provide some kind of a re-queue method for these cases?

Update: have implemented for my use case: https://github.com/karussell/goreplay-java-middleware/

[Vani261196]

Hi , my problem with goreplay is that my application has stateful queries , and if we delete some item in production then that id would be deleted , so replaying that id again is impossible , so we were hoping there would be someway where we can stall the middleware till we get new ids , then replacing the deleted ids with the new id .