fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers πŸ•ΈοΈ. It also smartly handles traffic routing πŸ”€ to minimize bans and increase success rates πŸš€.
http://scrapoxy.io
MIT License
1.89k stars 232 forks source link

[BUG] Socket hang out when scrapoxy is overloaded #242

Open jmaitrehenry opened 4 weeks ago

jmaitrehenry commented 4 weeks ago

Current Behavior

When the scrapoxy instance is overloaded, we could have some socket hang up.

Expected Behavior

We do not have any error.

Steps to Reproduce

  1. Send as many request as possible and wait

Failure Logs

ERROR [MasterService] request_error: socket hang up from proxy

### Scrapoxy Version

4.9

### Custom Version

- [ ] No
- [X] Yes

### Deployment

- [X] Docker
- [ ] Docker Compose
- [ ] Kubernetes
- [ ] NPM
- [ ] Other (Specify in Additional Information)

### Operating System

- [X] Linux
- [ ] Windows
- [ ] macOS
- [ ] Other (Specify in Additional Information)

### Storage

- [X] File (default)
- [X] MongoDB & RabbitMQ
- [ ] Other (Specify in Additional Information)

### Additional Information

It's not an easy bug and if I investigate the TCP traffic with Wireshark after disabling the SSL between scrapoxy and proxy, I could find this error send back from the proxy to scrapoxy:

HTTP/1.1 500 connect_error X-Scrapoxy-Proxyerror: write after end



![image](https://github.com/fabienvauchelles/scrapoxy/assets/593562/8059c697-0521-493e-9d26-b687694edc04)

You can assign me this issue as I'm working on it.
fabienvauchelles commented 4 weeks ago

Thanks :)

fabienvauchelles commented 4 weeks ago

which tool are you using for load test ?

jmaitrehenry commented 3 weeks ago

I'm using DDosify mainly but we also use it with real traffic and we see the same issue.