search

fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.
http://scrapoxy.io
MIT License
2.06k stars 237 forks source link

[BUG] Socket hang out when scrapoxy is overloaded #242

Open jmaitrehenry opened 5 months ago

jmaitrehenry commented 5 months ago

Current Behavior

When the scrapoxy instance is overloaded, we could have some socket hang up.

Expected Behavior

We do not have any error.

Steps to Reproduce

  1. Send as many request as possible and wait

Failure Logs

ERROR [MasterService] request_error: socket hang up from proxy

### Scrapoxy Version

4.9

### Custom Version

- [ ] No
- [X] Yes

### Deployment

- [X] Docker
- [ ] Docker Compose
- [ ] Kubernetes
- [ ] NPM
- [ ] Other (Specify in Additional Information)

### Operating System

- [X] Linux
- [ ] Windows
- [ ] macOS
- [ ] Other (Specify in Additional Information)

### Storage

- [X] File (default)
- [X] MongoDB & RabbitMQ
- [ ] Other (Specify in Additional Information)

### Additional Information

It's not an easy bug and if I investigate the TCP traffic with Wireshark after disabling the SSL between scrapoxy and proxy, I could find this error send back from the proxy to scrapoxy:

HTTP/1.1 500 connect_error X-Scrapoxy-Proxyerror: write after end



![image](https://github.com/fabienvauchelles/scrapoxy/assets/593562/8059c697-0521-493e-9d26-b687694edc04)

You can assign me this issue as I'm working on it.
fabienvauchelles commented 5 months ago

Thanks :)

fabienvauchelles commented 5 months ago

which tool are you using for load test ?

jmaitrehenry commented 5 months ago

I'm using DDosify mainly but we also use it with real traffic and we see the same issue.

roxanebellot commented 2 months ago

Same issue here :+1:

fuad00 commented 2 months ago

Same issue. i had to run another instance as a temp fix