The only tricky bit here is 47750d4: I discovered that the logic for logout detection was slightly wrong. Before, when a logged-out request arrived, we thought that this was the request that caused logout, and did not retry it, but retried all next requests. But the actual logout request could arrive a little later in fact, and we could incorrectly mark a different request as logout. To fight this I allow each request be retried a couple of times, and if it is logged out all the time, then we don't repeat it.
Middleware is tested without splash here. Splash tests are still done in the undercrawler, not sure if it makes sense to add them here too. It's not a lot of new code, just much slower build time.
The only tricky bit here is 47750d4: I discovered that the logic for logout detection was slightly wrong. Before, when a logged-out request arrived, we thought that this was the request that caused logout, and did not retry it, but retried all next requests. But the actual logout request could arrive a little later in fact, and we could incorrectly mark a different request as logout. To fight this I allow each request be retried a couple of times, and if it is logged out all the time, then we don't repeat it. Middleware is tested without splash here. Splash tests are still done in the undercrawler, not sure if it makes sense to add them here too. It's not a lot of new code, just much slower build time.