search

yadayada / acd_cli

An unmaintained command line interface and FUSE filesystem for Amazon (Cloud) Drive
Other
1.35k stars 166 forks source link

Appspot users may get throttled starting September 23rd #416

Open yadayada opened 8 years ago

yadayada commented 8 years ago

This is a public service announcement.

Please adjust your Amazon Drive API usage

Thank you for integrating your app with Amazon Drive.

We noticed abnormal call patterns from your app into Amazon Drive's APIs. Your customers are calling the getChanges method up to 2,900 times per hour; higher than what we saw when we approved your app. We recommend that you call getChanges a maximum of 15 times per hour per customer. In order to maintain our service performance at optimal levels for all of our customers, we will throttle your app at developer levels starting on September 23rd, 2016.

If you would like the throttle removed, please re-submit your app for approval.

Best regards, Amazon Cloud Drive 3P Support

I do not know what it means exactly for the app to be on developer limits. [This is not for a lack of asking; Amazon would not tell me what limitations there are.]

So, if any acd_cli customers experience issues starting on the 23rd, I suggest creating your own security profile.

msh100 commented 8 years ago

Your customers are calling the getChanges method up to 2,900 times per hour

Does this mean there's a single user calling getChanges 2,900 times an hour? or all users combined?

It sounds like the former. They recommend limiting it to 15, maybe time to add some throttle/burst logic into acd_cli itself?

EldonMcGuinness commented 8 years ago

What makes the getChanges call, is it just the sync command? Is there anything users can do to lower the number of calls to the service?

yadayada commented 8 years ago

Does this mean there's a single user calling getChanges 2,900 times an hour? or all users combined?

I'm not sure anymore, but judging from the number of roughly 17,500 requests per day to the App Engine instance there should be 730 users per hour. So, that's 4 requests per hour per user on average and already well below 15 rph.

What makes the getChanges call, is it just the sync command?

Yeah, it's the sync action and the autosync function that's disabled by default.

They recommend limiting it to 15, maybe time to add some throttle/burst logic into acd_cli itself?

Limiting the (auto)sync command to run only once per 4 minute would be fairly easy, but getting every user to update is another thing.

Axcy commented 8 years ago

How hard would it be to change your app key and disable the old one? This could be a way to force people to upgrade perhaps?

a5m0 commented 8 years ago

@yadayada or anyone else would you be able to provide updated steps for how to get our own security profile working? I am able to create one but I don't see how to get anything about ACD working with it... like there seems to be no ability to apply for ACD api access to the new security profile?

breunigs commented 8 years ago

@a5m0 request an invitation: https://developer.amazon.com/public/apis/experience/cloud-drive

a5m0 commented 8 years ago

I did this a couple weeks ago, has anyone actually received an email back?

On Sun, Sep 25, 2016, 2:14 AM Stefan notifications@github.com wrote:

@a5m0 https://github.com/a5m0 request an invitation: https://developer.amazon.com/public/apis/experience/cloud-drive

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yadayada/acd_cli/issues/416#issuecomment-249405046, or mute the thread https://github.com/notifications/unsubscribe-auth/AExEejZ-Ar-Dx5f_A6Vv0A6g4ZfUgn8lks5qthFbgaJpZM4KBQjW .

bgemmill commented 8 years ago

@yadayada looking at bug #400 what is the correct way of dealing with the end not reached for change requests now that incomplete syncs need to be spaced by multiple minutes to make Amazon happy?

I've switched over to old syncing for my uses since I can't seem to get a complete incremental sync at all.

Craeckie commented 8 years ago

@a5m0 I received this response some time last month:

Thank you for your interest in using the Amazon Drive developer tools. We need more information in order to consider your request. Please let us know more about your application and/or future plans for using the Amazon Drive developer tools.

• What is the use case you plan to support with your app? • What type of customers, small business, or enterprise does your app support? • How frequently will your app need to make requests to our service? • Please provide a link to your website and link to download your app. • Please provide your phone number in case we have additional questions.

Best Regards,

The Amazon Drive Team

I answered them several weeks ago without any response..

vx28643 commented 8 years ago

I just received this same email and answered as truthfully as possible, that I want to whitelist my security profile for personal file backup, giving the link here to acd_cli

Will see what happens.

ghost commented 8 years ago

@vx28643 Any update at the Amazon side? I replied weeks ago. No answer to the moment.

vx28643 commented 8 years ago

@maiaftv no, I will post here when I get any reply. They took a few weeks to reply to my initial request, so I will be patient.

vx28643 commented 8 years ago

Just received a response as follows:

Thank you for your interest in using the Amazon Drive developer tools. We consider a variety of factors as we evaluate each invite request, including the use case your application will support. While our goal is to offer a broad and compelling assortment of applications to Amazon Drive customers, we are currently unable to approve your request for access.

I suppose they don't want to grant api access for developers wishing to use acd for personal file backup. Will report back if I am able to change their mind...

bgemmill commented 7 years ago

@yadayada should we be defaulting to old-sync if an incremental one fails? There are examples in the wild of people incremental syncing in tight loops to make it work when #400 strikes them: https://github.com/jlippuner/acd_cli/commit/cc05b38a0bf495d702a1d266c4af314bf913c79e

a5m0 commented 7 years ago

Received the same negative response email as @vx28643

ghost commented 7 years ago

@vx28643 and @a5m0, same thing here. Denied.

vx28643 commented 7 years ago

I suggest those who are receiving negative responses should all be replying to Amazon, asking them to reconsider and pointing them to this page. By them visiting this page they may see that they are turning away some potential paying customers.

I mean, we are all wanting to be paying customers of Amazon Drive and have a legitimate reason for wanting to use the API for our personal file storage and backups.

As I hope to be a long time customer of Amazon Drive I would much prefer to whitelist my own security profile than simply rely on the shared Appspot profile that acd_cli provides.

By helping Amazon to understand our use case perhaps they will see a way to change their mind.

I will reply to them and post here if anything changes.

ghost commented 7 years ago

I guess they do not want people like us using their services, @vx28643. Being able to mount and upload huge amounts of encrypted data could not be what they had on mind for the business plan of Amazon Cloud Drive.

If you get any response, please update this issue.

vx28643 commented 7 years ago

@maiaftv I realize this and while amazon promote Amazon Drive as "Unlimited Storage" for $59.99 per year I assume they must have some kind of throttling in place for unreasonable users.

That seems to be what triggered the problems with the Appspot profile that acd_cli provides, and is one of the reasons I feel the need for my own security profile if I were going to use Amazon Drive as part of my long term backup plan.

You are probably right - but I thought it worth asking Amazon to have another look before I completely give up on Amazon Drive as a backup option. I suppose we will see what happens. I can only speak for myself but my uses would be ~1TB total storage, not huge amounts.

Will update if I hear something. Cheers.

vx28643 commented 7 years ago

Tired of waiting for Amazon so have opted for one of the affordable backup/storage hosting plans to be found on www.lowendtalk.com I was able to find one at less than $50/year for 1TB. Will update if I do hear anything from Amazon

gdragotto commented 7 years ago

@yadayada Actually, after the --RequestError 429 "Rate Exceeded--, i can't access to Amazon api both with acd_cli init and custom security profile. What i get is

We're sorry!

An error occurred when we tried to process your request. Rest assured, we're already working on the problem and expect to resolve it shortly. Show Details

Error Summary

400 Bad Request Unknown client_id

joeydh commented 7 years ago

Seems like acd_cli has been banned or maybe a rate limit

bgemmill commented 7 years ago

acdcli is up on my end, so probably not an outright ban. If you were getting 429s, I've observed amazon shutting me out for a length of time after a high number of operations. You could be in a state like that; I'd wait a day and see if it's still down.