awslabs / dynein

DynamoDB CLI written in Rust.
https://github.com/awslabs/dynein
Apache License 2.0
359 stars 36 forks source link

BatchWriteError Error when `dy import` #259

Open YN-CTC opened 1 month ago

YN-CTC commented 1 month ago

I'm encountering an issue while trying to import data into a table using Dynein. The process starts, but eventually fails with a BatchWriteError. It seems like the problem might be related to the aws-sdk-rust. Below are the details of the attempts and the errors received.

$ dy import -t <table_name> -i import_data.json
WARN: For the best performance on import/export, dynein recommends OnDemand mode. However the target table is Provisioned mode now. Proceed anyway? 
[WARN: For the best performance on import/export, dynein recommends OnDemand mode. However the target table is Provisioned mode now. Proceed anyway? yes
8350 items processed (1201.56 items/sec)Error: BatchWriteError(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source:CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } }))

$ dy import -t <table_name> -i import_data.json
1975 items processed (1026.56 items/sec)Error: BatchWriteError(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source:ProviderError(ProviderError { source: Unexpected(Unexpected { source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s })}) }), connection: Unknown } }))
StoneDot commented 1 month ago

Thank you for using dynein and reporting the error. A credential provider's throttling might cause this issue. The current implementation does not reuse clients. I think that the code change on our side should fix this issue. https://github.com/awslabs/dynein/blob/main/src/batch.rs#L240

YN-CTC commented 1 month ago

Thank you for your response. Due to the need for immediate use in business, I modified the context function in app.rs to load the authentication information and instantiate the DynamoDbSdkClient. Additionally, by reusing the instantiated client within the context function, I confirmed that the above-mentioned bug no longer occurs.

StoneDot commented 1 month ago

Thank you for sharing your status. I am working on improving the import/export command, and I expect the new implementation will resolve this issue. Therefore, I do not have plans to address this issue individually. I will update this thread once the work is complete. Is this acceptable to you? Please feel free to let me know if you have any other concerns.

YN-CTC commented 1 month ago

There's no problem with this. I am looking forward to the improvements.