Shortly after starting the ingestion for https://github.com/NASA-IMPACT/csdap-cumulus/issues/323, I noticed that the ingest/publish rate was maxing out at 18K granules/hr, which is about half the best rate we've achieved. Even at 36K/hr, we're still much below where we'd like to aim, which is ~60K/hr (i.e., ~1K/min).
We must adjust our throttling configuration for the background queue to allow us to achieve ~1K/min, but only as far as possible without reaching the AWS quota for concurrent Lamba executions, which defaults to 1000, however, I now see that my request to increase the quota to 2000 concurrent executions has been applied! See https://us-west-2.console.aws.amazon.com/servicequotas/home/services/lambda/quotas/L-B99A9384
Shortly after starting the ingestion for https://github.com/NASA-IMPACT/csdap-cumulus/issues/323, I noticed that the ingest/publish rate was maxing out at 18K granules/hr, which is about half the best rate we've achieved. Even at 36K/hr, we're still much below where we'd like to aim, which is ~60K/hr (i.e., ~1K/min).
We must adjust our throttling configuration for the background queue to allow us to achieve ~1K/min, but only as far as possible without reaching the AWS quota for concurrent Lamba executions, which defaults to 1000, however, I now see that my request to increase the quota to 2000 concurrent executions has been applied! See https://us-west-2.console.aws.amazon.com/servicequotas/home/services/lambda/quotas/L-B99A9384