search

fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.
http://scrapoxy.io
MIT License
1.97k stars 233 forks source link

[BUG][Azure] Sometime the deployment create more proxies than needed #241

Open jmaitrehenry opened 3 months ago

jmaitrehenry commented 3 months ago

Current Behavior

Sometime, scrapoxy create more proxy than the maximum allowed and the request could be block by a quota limit:

Error: Subscription 2edf240e-097e-4b04-a0a8-0a3c76592cdc has a quota of 20 for resources of type PublicIpAddress with sku Basic. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress
    at /app/scrapoxy.js:2:415409

Actually, we have a max of 18 proxies in the region and we have a quota of 20 of them.

Expected Behavior

Scrapoxy do not over provision proxies or have an option for that where we can control how much over provisionning we allow.

Steps to Reproduce

  1. Don't know, it happen when scrapoxy recycle proxies or when we delete more than one proxy

Failure Logs

Error: Subscription 2edf240e-097e-4b04-a0a8-0a3c76592cdc has a quota of 20 for resources of type PublicIpAddress with sku Basic. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress
    at /app/scrapoxy.js:2:415409


### Scrapoxy Version

4.9

### Custom Version

- [ ] No
- [X] Yes

### Deployment

- [X] Docker
- [ ] Docker Compose
- [ ] Kubernetes
- [ ] NPM
- [ ] Other (Specify in Additional Information)

### Operating System

- [X] Linux
- [ ] Windows
- [ ] macOS
- [ ] Other (Specify in Additional Information)

### Storage

- [X] File (default)
- [ ] MongoDB & RabbitMQ
- [ ] Other (Specify in Additional Information)

### Additional Information

Low issue as it a temporary issue.
fabienvauchelles commented 3 months ago

Does it happen when:

  1. You have a quota for 20 instances
  2. You are using 20 instances
  3. You ask Scrapoxy to remove more than 2 instances

?

Let me try. I think Scrapoxy will try to have temporary 18 (running) + 2 (deleting) + 2 (creating) = 22 instances

jmaitrehenry commented 3 months ago

In my case, I have 18 instance with a quota of 20 but yes, it's exactly the case you expose, scrapoxy create more instance without waiting the old instances are down and deleted.

fabienvauchelles commented 2 months ago

I think the issue is solved. Can I close it @jmaitrehenry ?