fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers πŸ•ΈοΈ. It also smartly handles traffic routing πŸ”€ to minimize bans and increase success rates πŸš€.
http://scrapoxy.io
MIT License
2.06k stars 237 forks source link

Add Azure spot instance and new instance types, also fix an issue when Azure is too slow to provision a VM #238

Closed jmaitrehenry closed 5 months ago

jmaitrehenry commented 5 months ago

Changes

In this PR 2 changes are made:

Add Spot instance and new instance size for Azure

So basically, now we have the possibility to use Standard_A1_v2 and Standard_D2plds_v5 VM. I use the A1_v2 VM on my side and it works. For the ARM image, I have an issue with it right now when we create the VM image as the VM image don't have the arch parameter, Azure try to create an amd64 image instead of an arm64 image.

Also, you can't use the Spot instance pricing with the B* VM series.

Why I add the Spot pricing? because the A1_V2 VM price is almost the same (3.14$/month) as the B1ls VM (3.80$/month) with 1 full vPCU and 2 G of memory and, normally, a better network throughput.

And the D2plds_v5 is5.6$/month with 2vCPU and 4G of memory with a far better bandwidth.

Add a delay of 180s before removing an unused IP

Actually, we have a race condition, when we create a deployment, Azure will create the public IP first, then a NIC and, at the end, the VM. But, in the same time, if scrapoxy do a listProxy, it will delete unused resources like public IP and NIC. That why we add a 180s delay before a resource is deleted.

fabienvauchelles commented 5 months ago

I've cloned the branch from my repository and rebased it onto the master branch.

I retained the Spot Instance functionality but replaced the IP timeout mechanism with a different solution: listing deployments to identify instances in creation, eliminating the need for a timer.

Big thanks to you! This is the first accepted PR for Scrapoxy V4.

The release is V4.10.1. I've quoted you in the changelog => https://scrapoxy.io/intro/changelog#_4-10-1

Best, Fabien

fabienvauchelles commented 5 months ago

Merged