Rate limit fails, rate increases dramatically, eventually crashes process

[qbatten]

So, I havent been able to complete a full download of my stripe data because something funky is going on with the rate limiter.

This is the context:

• The command I'm using is tdog --json-file config.json.
• I'm using a SQLite db, and I'm sure it's able to interact with the DB bc tdog successfully adds tables to the empty db at thestart of the run.
• I have max_requests_per_second=20

Here's my config.json, click to expand

I redacted the key of course ``` { "cmd": { "fn": "download", "args": { "from": { "stripe": { "secret_key": "rk_redacted", "max_requests_per_second": 20 } }, "to": { "sqlite": { "file": "main.db" } }, "options": { "watch": false, } } }, "log": "trace" } ```

What happens:

• It runs for a while, seems like it's doing fine.
• At some point, I get the "Database inserts are too slow" error message. On one run, the bad stuff (queue & rate limit blowing up) happened right after this warning. But on another run it kept going without issues for another 45min-1hr. So I think this is not correlated, but I'm not sure.
• Then, at some point...
  • the queue grows
  • The req/sec increases dramatically, surpassing my rate limit
• Eventually stripe starts returning 429s, and the program waits a bit, but then it keeps running way above stripes rate limit, immediately gets a 429 again... and eventually stripe just starts closing the connection.

[qbatten]

I've attached two log files from different runs where this issue occurs.

• debug_shared_log_1.log is a run with log=debug. The issue appears at L3642, or you can search for the timestamp 2022-07-21 15:54:46.625.
• debug_shared_log_2.log is a run with log=info. The issue appears at L2070, or you can search for the timestamp 2022-07-21 04:07:09.546.

Thanks for making this awesome program and let me know if I can help further!

debug_shared_1.log debug_shared_2.log

[emadda]

Hello @qbatten,

Thanks.

Can you try two things:

1. Request a rate limit increase from Stripe.
2. Set a lower max_requests_per_second, perhaps starting at 5?

After the first full download you will be able to apply the events to the DB to keep it up to date. Applying the events should take a few seconds.

The logs are helpful. Let me know how it goes.