Extract out timestream write to min memory lambda and add a step that uploads to s3

[jonny-rimek]

First try writing records concurrently to timestream!

!!!!!!!

I need to upload the converted result to s3, check how long that takes, if it is as low as uploading to timestream, there is no point

!!!!!!!!

let s3 call convert directly and put messages into queue and let timestreamwrite poll from that

most of the time is spent uploading to timestream, but we dont need a huge lambda for that, it's just cost ineffective

2sec are spent covnerting the file, ~12sec writing to timestream.

seperating those concerns would also allow me to make a bigger convert lambda, that would benefit my users because they can upload larger files

architecture:

convert lambda uploads to s3, pushes to the object key to sns which invokes the timestream write lambda (min memory) which downloads the data from s3 and writes it to timestream, after that it pushes the combatlog uuid to SQS, which also has a delay of 10seconds until the next lambda is notified, the following lambda just publishes the message to SNS.

I can't do SQS > SNS because the trace will be lost :s

[jonny-rimek]

parallel uploads fixed the problem