search

fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.
http://scrapoxy.io
MIT License
1.98k stars 233 forks source link

AWS Spot Instance #82

Closed DimaAhmedov closed 6 years ago

DimaAhmedov commented 6 years ago

Is there support for AWS Spot instances at the moment? If not can this be added ? I can donate for this feature to go asap.

fabienvauchelles commented 6 years ago

Hello, You can do that! Thx

DimaAhmedov commented 6 years ago

Hi, Thank you for the reply. Is there any documentation for this? I have tried adding this code inside the "instance" but it gives me error:

2017-12-08T08:43:57.686Z - error: [Manager] Error: Cannot update or adjust instances: UnexpectedParameter: Unexpected key 'InstanceMarketOptions' found in params

"InstanceMarketOptions": {
    "MarketType": "spot",
    "SpotOptions": {
      "BlockDurationMinutes": "0",
      "InstanceInterruptionBehavior": "terminate",
      "MaxPrice": "0.0023",
      "SpotInstanceType": "one-time",
      "ValidUntil": "Wed Dec 31 2018 16:00:00 GMT-0800 (PST)"
    }
  },
fabienvauchelles commented 6 years ago

Hello,

Spot instances are special: you ask an instance and it could take a long time to get it.

I tried to plug the feature but without success (sometimes it takes too long to get an instance).

If you want to try, you have to make a pull request and code a provider like here: https://github.com/fabienvauchelles/scrapoxy/tree/develop/server/providers/awsec2

thanks, fabien

anisgandoura commented 6 years ago

Hello,

I have used spot instances on another project.

To get spot instances immediately you need to diversify the instance type (ex: nano, small, etc ...), and the availability region. The best strategy would be to diversify the region too.

You can summon many spot instances but requesting the same type in the same region / zone will throttle your requests.

Actually, Scrapoxy does not support multi regions unfortunately.

Anis

fabienvauchelles commented 6 years ago

Hello,

Now, Scrapoxy uses many region.

Spot won't be supported (to complex).

Thanks, Fabien.