Using this library for very high volumes

[chris-erickson]

Any strategies for best using this at very high volumes/in parallel? I'm using it in an AWS Lambda and we can easily hit a rate limit for /1.1/account/verify_credentials/ when posting on behalf of one account (with the new developer API setup).

I've instantiated the client outside of functions so it shouldn't be re-negotiating on each request (it was before, we fixed that and some other issues). I've verified with twitter that we are well below our other account limits. It's plausible that the library is still doing some auth work that isn't well cached in one way or another (by me, most likely), or we just have too many parallel requests.

[jeremylow]

Huh, very strange! Are you trying to get users' email addresses? I don't see anywhere in the code that you'd be hitting that endpoint without specifically calling it. If it's for something you can cache, I'd suggest that route, but maybe it's another issue. Would be interested to hear more about how you're calling that function.

[chris-erickson]

Really all I use is .PostUpdate(message, media=media_url). I instantiate an API at the top of a module file like

if STAGE == "prod":
    account_config = {
        "account_the_first": {
            "api": twitter.Api(**ACCOUNTS["account_the_first"]),
...

Then I reference the ['api'] in the few places that (more or less) do account_config['account_the_first']['api'].PostUpdate(message, media=media_url)

My working theory is that instantiating the API triggers a call the the verify endpoint (or perhaps it's lazily done). With a lambda, I might have in the 10-20's of concurrent executions, so maybe that grinds through auth limits pretty fast? So my next thought is how I might globally cache the auth call for X minutes, perhaps in an external store like redis. Not sure if that's possible?

FWIW, this app is entirely for our org, and posts to a twitter account we control. Due to the new API, we had to go through all the shenanigans to set up a dev account and link it as an app in our regular twitter account.