search

wesselt / bunq2ynab

Upload bunq transactions to ynab
Other
78 stars 22 forks source link

adds sleep for get_notification_filters #44

Closed rubenvitt closed 1 year ago

rubenvitt commented 1 year ago

This MR resolves the following issue for auto_sync.py:

Aug 07 12:03:55 ubuntu python3[357647]: Exception: Too many requests. You can do a maximum of 3 calls per 3 second to this endpoint.
Aug 07 12:03:55 ubuntu python3[357647]: 2023-08-07 12:03:55,792 | ERROR | auto_sync.py:193 | <module> | Failed 12 times, waiting 10 seconds for retry.
Aug 07 12:04:05 ubuntu python3[357647]: 2023-08-07 12:04:05,802 | INFO | sync.py:60 | populate | Retrieving bunq accounts...
Aug 07 12:04:06 ubuntu python3[357647]: 2023-08-07 12:04:06,585 | INFO | sync.py:62 | populate | Retrieving ynab accounts...
Aug 07 12:04:07 ubuntu python3[357647]: 2023-08-07 12:04:07,080 | INFO | auto_sync.py:80 | setup_callback | Host has a public IP...
Aug 07 12:04:11 ubuntu python3[357647]: 2023-08-07 12:04:11,439 | ERROR | auto_sync.py:189 | <module> | Error: Too many requests. You can do a maximum of 3 calls per 3 second to this endpoint.
Aug 07 12:04:11 ubuntu python3[357647]: 2023-08-07 12:04:11,439 | ERROR | auto_sync.py:190 | <module> | Traceback (most recent call last):
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/./auto_sync.py", line 175, in <module>
Aug 07 12:04:11 ubuntu python3[357647]:     setup_callback()
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/./auto_sync.py", line 114, in setup_callback
Aug 07 12:04:11 ubuntu python3[357647]:     bunq_api.add_callback(acc["bunq_user_id"], "bunq2ynab-autosync", url)
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/lib/bunq_api.py", line 10, in add_callback
Aug 07 12:04:11 ubuntu python3[357647]:     set_callbacks(bunq_user_id, url_end, {
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/lib/bunq_api.py", line 41, in set_callbacks
Aug 07 12:04:11 ubuntu python3[357647]:     get_notification_filters(bunq_user_id)]
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/lib/bunq_api.py", line 116, in get_notification_filters
Aug 07 12:04:11 ubuntu python3[357647]:     return bunq.get(method)
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/lib/bunq.py", line 206, in get
Aug 07 12:04:11 ubuntu python3[357647]:     return call('GET', method)
Aug 07 12:04:11 ubuntu python3[357647]:   File "/home/bunq2ynab/bunq2ynab/lib/bunq.py", line 197, in call
Aug 07 12:04:11 ubuntu python3[357647]:     raise Exception(result["Error"][0]["error_description"])
Aug 07 12:04:11 ubuntu python3[357647]: Exception: Too many requests. You can do a maximum of 3 calls per 3 second to this endpoint.
Aug 07 12:04:11 ubuntu python3[357647]: 2023-08-07 12:04:11,440 | ERROR | auto_sync.py:193 | <module> | Failed 13 times, waiting 10 seconds for retry.
wesselt commented 1 year ago

Thanks for the PR, looks good! Two questions:

1) There's one notification filter per user. How many users are you synching to trigger the rate limit? 2) Have you tried shorter sleep values?

rubenvitt commented 1 year ago
  1. Only my own user, but more than three bank accounts in there. 🤷‍♂️
  2. My fault. Can be .33 - I'll change that.
wesselt commented 1 year ago

That's curious, there is only one notification filter per user. Perhaps the limit applies to a group of APIs.

I just noticed the error message you got: "You can do a maximum of 3 calls per 3 second to this endpoint." That sounds like 1 per second.

rubenvitt commented 1 year ago

Your right, I was too fast with editing & change it back.

From their API docs:

Rate limits If you are receiving the error 429, please make sure you are sending requests at rates that are below our rate limits. Our rate limits per IP address per endpoint: GET requests: 3 within any 3 consecutive seconds POST requests: 5 within any 3 consecutive seconds PUT requests: 2 within any 3 consecutive seconds Callbacks: 2 callback URLs per notification category We have a lower rate limit for /session-server: 1 request within 30 consecutive seconds.