search

linz / geostore

Central storage, management and access for important geospatial datasets
MIT License
33 stars 2 forks source link

Bug: PynamoDB error while importing data #449

Closed billgeo closed 3 years ago

billgeo commented 3 years ago

Bug Description

PynamoDB error while importing data

How to Reproduce**

  1. Run: 
    aws lambda invoke --function-name nonprod-dataset_versions-endpoint --invocation-type RequestResponse --payload '{"httpMethod": "POST", "body": {"id": "3a8e05df8c4211ebbfae69792f7b74c4", "metadata-url": "s3://linz-data-lake-raster-nonprod/aerial-imagery/new-zealand/auckland-05m-rural-2010-2012_RGBA_bill_test/auckland-05m-rural-aerial-photos-2010-2012_RGBA/collection.json", "type": "RASTER"}}' --region ap-southeast-2 /dev/stdout
  2. Step function fails on valdiation like this https://ap-southeast-2.console.aws.amazon.com/states/home?region=ap-southeast-2#/executions/details/arn:aws:states:ap-southeast-2:632223577832:execution:nonproddatasetversioncreation8597518E-sUQ3QIrF41EC:f72778ed8c5111ebbc25ed88a9bd0951
  3. See error similar to https://ap-southeast-2.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-2#logsV2:log-groups/log-group/$252Faws$252Fbatch$252Fjob/log-events/checkstacmetadatataskch-f9c68036667dffb$252Fdefault$252Fad82ec6d805344668bb107cd1b743603 Last step in Stack trace 
    pynamodb.exceptions.PutError: Failed to put item: An error occurred (ProvisionedThroughputExceededException) on request (4VR8PLF40P5O8CVS044IC479VBVV4KQNSO5AEMVJF66Q9ASUAAJG) on table (nonprod-processing-assets) when calling the PutItem operation: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. | pynamodb.exceptions.PutError:  Failed to put item: An error occurred  (ProvisionedThroughputExceededException) on request  (4VR8PLF40P5O8CVS044IC479VBVV4KQNSO5AEMVJF66Q9ASUAAJG) on table  (nonprod-processing-assets) when calling the PutItem operation: The  level of configured provisioned throughput for the table was exceeded.  Consider increasing your provisioning level with the UpdateTable API.

Software and OS versions

Additional context

imincik commented 3 years ago

We just need to increase Dynamodb provisioned throughput, if I remember correctly we are using some very low values there.