search

jonny-rimek / wowmate

combatlog analysis for world of warcraft players
http://wowmate.io
MIT License
4 stars 0 forks source link

Parallelize timestream upload #302

Closed jonny-rimek closed 3 years ago

jonny-rimek commented 3 years ago

For a single file upload the duration is now down to ~2.2sec from ~25sec, because I make all API calls in parallel (361). This means my lambdas are way cheaper because I don't pay for a bunch of cpu that idles to make api calls.

as a result the api throttles/returns errors way sooner, that meant I had to reduce the reserved concurrency dramatically. I experienced intermittent errors on 5 and went down to 2. Zero errors at 2 tho.

I could implement that only a certain number of goroutines can run concurrently, to not overwhelm the api to much, as a result I'd have longer running lambdas, which means higher cost. I will stick with a limit of 2 and optimize later

jonny-rimek commented 3 years ago

Doesn' work for large file, because it tries to send to many api calls at once

failed to write to timestream: RequestError: send request failed caused by: Post "https://ingest.timestream.us-east-1.amazonaws.com/": dial tcp: lookup ingest.timestream.us-east-1.amazonaws.com on 169.254.78.1:53: dial udp 169.254.78.1:53: socket: too many open files: errorString null

jonny-rimek commented 3 years ago

Limited it to 10 concurrent api calls, there is some room for improvement, but I'm limited on queries anyway.

I'm now at 4,4 seconds with 15 concurrent lambdas and 10 concurrent goroutines.