Closed Thinkscape closed 8 years ago
@bgemmill I've been using your #374 branch for those tests. It's somewhat tricky to test, because I have to figure out if the mount is crashing because of some bug, or if amazon is bouncing requests. Anyway, none of that would be possible without your work 👍
@Thinkscape oddly, I don't see this at all with rsync and many-file directories in #374. Since I haven't seen the error, I'm just guessing here... this may be a side effect of moving from WriteStream and it's background thread to WriteBuffers and flushing temp files directly.
Is iozone threaded?
Is iozone threaded?
I don't believe so. A quick run shows that it runs everything from the main thread. It just performs a lot of sequential writes as it does its benchmark - i.e.:
... but this will happen every time ACD servers get hammered too much. acd_cli doesn't have a retry routine for mounts yet - so when amazon server yells "back off" the read/write will fail and it crashes the operation.
A quick google search of the fuse write-back reveals that it (the new FUSE flag feature) has been added to some FUSE drivers, like the google cloud services suite.
Conversation moved to issue #374 https://github.com/yadayada/acd_cli/pull/374
Tools that perform many (smaller) writes will often trigger Amazon's 429:
This can be easily replicated with something like rsync (small files) or iozone, i.e.:
I don't think there's any reasonable workaround other than implementing a time-backed write-back cache (buffer). Hammering Amazon servers is bad for obvious reasons and could potentially trigger an account suspension (looking at ACD TOS).