yadayada / acd_cli

An unmaintained command line interface and FUSE filesystem for Amazon (Cloud) Drive
Other
1.35k stars 165 forks source link

429 Too Many Requests problems - sequential writes on a mount #383

Closed Thinkscape closed 8 years ago

Thinkscape commented 8 years ago

Tools that perform many (smaller) writes will often trigger Amazon's 429:

HTTP/1.1 429 Too Many Requests

Can not create temp file: /mnt/acd/file
creat: Bad address

[...]
send: <MultipartEncoder: OrderedDict([('metadata', '{"name": "ioz", "parents": ["ZbpBl4OdSdeG4nZu4me5aQ"], "kind": "FILE"}'), ('content', ('ioz', <_io.BytesIO object at 0x7f9213562ca8>, 'application/octet-stream'))])>
sendIng a read()able
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Content-Type header: Date header: Server header: x-amzn-RequestId header: Content-Length header: Connection 16-08-21 13:35:15.468 [DEBUG] [requests.packages.urllib3.connectionpool] - "POST /cdproxy/nodes?suppress=deduplication HTTP/1.1" 429 120
16-08-21 13:35:15.469 [ERROR] [acdcli.acd_fuse] - create: RequestError: 429, {"logref":"54510c7e-6793-11e6-96d9-6f62fd43f55a","message":"Concurrent Access on same node. Please back off.","code":""}
16-08-21 13:35:15.469 [DEBUG] [acdcli.acd_fuse] - <- create '[Unhandled Exception]'
Traceback (most recent call last):
  File "/root/acd_cli/acdcli/acd_fuse.py", line 561, in create
    r = self.acd_client.create_file(name, p.id)
  File "/root/acd_cli/acdcli/api/content.py", line 98, in create_file
    raise RequestError(r.status_code, r.text)
acdcli.api.common.RequestError: RequestError: 429, {"logref":"54510c7e-6793-11e6-96d9-6f62fd43f55a","message":"Concurrent Access on same node. Please back off.","code":""}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python3.5/dist-packages/fuse.py", line 736, in create
    fi.fh = self.operations('create', path, mode)
  File "/root/acd_cli/acdcli/acd_fuse.py", line 279, in __call__
    ret = getattr(self, op)(path, *args)
  File "/root/acd_cli/acdcli/acd_fuse.py", line 565, in create
    FuseOSError.convert(e)
  File "/root/acd_cli/acdcli/acd_fuse.py", line 84, in convert
    exc = FuseOSError.code_mapping[e.status_code]
KeyError: 429

This can be easily replicated with something like rsync (small files) or iozone, i.e.:

acd_cli mount /mnt/acd iozone -a -g 4096 -K -f /mnt/acd/ioz

I don't think there's any reasonable workaround other than implementing a time-backed write-back cache (buffer). Hammering Amazon servers is bad for obvious reasons and could potentially trigger an account suspension (looking at ACD TOS).

Thinkscape commented 8 years ago

@bgemmill I've been using your #374 branch for those tests. It's somewhat tricky to test, because I have to figure out if the mount is crashing because of some bug, or if amazon is bouncing requests. Anyway, none of that would be possible without your work 👍

bgemmill commented 8 years ago

@Thinkscape oddly, I don't see this at all with rsync and many-file directories in #374. Since I haven't seen the error, I'm just guessing here... this may be a side effect of moving from WriteStream and it's background thread to WriteBuffers and flushing temp files directly.

Is iozone threaded?

Thinkscape commented 8 years ago

Is iozone threaded?

I don't believe so. A quick run shows that it runs everything from the main thread. It just performs a lot of sequential writes as it does its benchmark - i.e.: image

... but this will happen every time ACD servers get hammered too much. acd_cli doesn't have a retry routine for mounts yet - so when amazon server yells "back off" the read/write will fail and it crashes the operation.

A quick google search of the fuse write-back reveals that it (the new FUSE flag feature) has been added to some FUSE drivers, like the google cloud services suite.

Thinkscape commented 8 years ago

Conversation moved to issue #374 https://github.com/yadayada/acd_cli/pull/374