This will affect the time Sfn needs and affects the proper cronjob rate (shall not run concurrent cronjob, only one cronjob should be running at a time). Currently we always see around 45 concurrency that's the max. If so, given the stories usually around 87~130, this mean we need up to 3 pass. Each pass is up to 13 minutes. Total of 39 minutes each Sfn could take, which is a bit slow. It means to process all the landing pages, it's going to take 40min * 818 landings / 60 / 24 ~~ 22 days.
This SO someone says this is expected and usually AWS StepFunction only gives you around 50, and AWS Sfn decides the actual concurrency. Is this true, can we get more source confirming this?
This AWS blog does not specify any ceiling limit of concurrency.
If there's a (semi) hard limit on concurrency per Sfn, another way is to run multiple Sfn at the same time! Say, cronjob instead of starting one Sfn, start 3 Sfn executions instead:
[x] Let cronjob run up to 3 Sfn
[x] Still we need to increase cronjob rate > 45min, because each Sfn takes up to 40min, need some buffer there.
This will affect the time Sfn needs and affects the proper cronjob rate (shall not run concurrent cronjob, only one cronjob should be running at a time). Currently we always see around 45 concurrency that's the max. If so, given the stories usually around 87~130, this mean we need up to 3 pass. Each pass is up to 13 minutes. Total of 39 minutes each Sfn could take, which is a bit slow. It means to process all the landing pages, it's going to take
40min * 818 landings / 60 / 24~~ 22 days.This SO someone says this is expected and usually AWS StepFunction only gives you around 50, and AWS Sfn decides the actual concurrency. Is this true, can we get more source confirming this?
Workaround
If there's a (semi) hard limit on concurrency per Sfn, another way is to run multiple Sfn at the same time! Say, cronjob instead of starting one Sfn, start 3 Sfn executions instead: