search

matheussilvasantos / dynamocli

Utilities for interaction with AWS DynamoDB
MIT License
26 stars 5 forks source link

The provided key element does not match the schema #1

Closed gconcepts closed 4 years ago

gconcepts commented 5 years ago

I'm getting the below exception when trying to push data from a CSV to a dynamodb table

{code} USD1601311:DataGenerator ooshiber$ dynamocli import APIReleases.csv --table ethos-affinity-api-releases /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call': The provided key element does not match the schema (Aws::DynamoDB::Errors::ValidationException) from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-dynamodb-1.34.0/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:117:incall' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:incall' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in call' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/response_paging.rb:10:incall' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/seahorse/client/plugins/response_target.rb:23:in call' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/seahorse/client/request.rb:70:insend_request' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-dynamodb-1.34.0/lib/aws-sdk-dynamodb/client.rb:823:in batch_write_item' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli/import.rb:45:inblock in write_records_to_dynamodb_table' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli/import.rb:44:in each' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli/import.rb:44:ineach_slice' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli/import.rb:44:in each' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli/import.rb:44:inwrite_records_to_dynamodb_table' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli/import.rb:18:in start' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/lib/dynamocli.rb:17:inimport' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:ininvoke_command' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/base.rb:466:instart' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.4/bin/dynamocli:6:in <top (required)>' from /usr/local/bin/dynamocli:22:inload' from /usr/local/bin/dynamocli:22:in `

' {code}

My dynamodb table looks as below {code} image {code}

The data in the CSV looks like below {code} releaseId,releaseName,releaseVersion 1,Test,1.17.0 2,Test,1.18.0 3,Test,1.19.0 4,Test,1.20.0 5,Test,1.21.0 {code}

What do you think could be the issue?

matheussilvasantos commented 5 years ago

Thank you for reporting this issue.

I was able to reproduce your error, the problem was that the program wasn't converting the data to the right type of data, in other words, the same type defined for the keys in the DynamoDB table.

I've just released a new version, could you test if it is working? It is the version 0.1.5.

gconcepts commented 5 years ago

Thanks. Will try that out now and report back.

gconcepts commented 5 years ago

I still seem to be getting the same error and I even uninstalled the older version and installed the new one.

matheussilvasantos commented 5 years ago

What you see when you run gem list dynamocli?

gconcepts commented 5 years ago

gem list dynamocli

LOCAL GEMS

dynamocli (0.1.5)

matheussilvasantos commented 5 years ago

That's weird, could you paste the error again?

gconcepts commented 5 years ago

dynamocli import APIReleases.csv --table ethos-affinity-api-releases /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call': The provided key element does not match the schema (Aws::DynamoDB::Errors::ValidationException) from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-dynamodb-1.34.0/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:117:incall' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:incall' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in call' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/aws-sdk-core/plugins/response_paging.rb:10:incall' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/seahorse/client/plugins/response_target.rb:23:in call' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-core-3.62.0/lib/seahorse/client/request.rb:70:insend_request' from /Library/Ruby/Gems/2.3.0/gems/aws-sdk-dynamodb-1.34.0/lib/aws-sdk-dynamodb/client.rb:823:in batch_write_item' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:59:inblock in write_records_to_dynamodb_table' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:in each' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:ineach_slice' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:in each' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:inwrite_records_to_dynamodb_table' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:19:in start' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/lib/dynamocli.rb:17:inimport' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:ininvoke_command' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' from /Library/Ruby/Gems/2.3.0/gems/thor-0.20.3/lib/thor/base.rb:466:instart' from /Library/Ruby/Gems/2.3.0/gems/dynamocli-0.1.5/bin/dynamocli:6:in <top (required)>' from /usr/local/bin/dynamocli:22:inload' from /usr/local/bin/dynamocli:22:in `

'

matheussilvasantos commented 5 years ago

Does your table have some index?

gconcepts commented 5 years ago

Yes it does have an index. I will try deleting the index and try it out again.

gconcepts commented 5 years ago

Same error with the index deleted. Will wait a bit in case AWS index deletion is still synching

gconcepts commented 5 years ago

Same error. Does it work for you?

matheussilvasantos commented 5 years ago

Yes, it does :cry:.

image

gconcepts commented 5 years ago

Weird. Let me delete the table and recreate without the index and i'll see if it works

gconcepts commented 5 years ago

Same error still. Can you send me a screenshot of your table overview page?

matheussilvasantos commented 5 years ago

Here it is: image

gconcepts commented 5 years ago

This is baffling. Still getting the same error after recreating the table. Are you on a Mac?

matheussilvasantos commented 5 years ago

No, I'm running Linux.

matheussilvasantos commented 5 years ago

Maybe Ruby is not running the right version, you can try cloning the repository and execute in the repository folder ./bin/dynamocli import APIReleases.csv --to ethos-affinity-api-releases

gconcepts commented 5 years ago

Ok. Will try that out and keep you posted

gconcepts commented 5 years ago

Same error. I will try it out from a linux VM and see if the same error occurs. Will keep you updated.

gconcepts commented 5 years ago

Same error on a Linux based distro.

matheussilvasantos commented 5 years ago

It doesn't make much sense but maybe can be your credentials, what permissions your machine have on AWS?

gconcepts commented 5 years ago

The same account has full admin permission in AWS. I can use the same account to perform the tasks manually within AWS console

AntonERG commented 4 years ago

Hey guys, I have a same issue, already checked permissions and creds. Any solution? (I'm also working on Mac)

matheussilvasantos commented 4 years ago

Hi, could you post the error here too?

AntonERG commented 4 years ago

Antons-MacBook-Pro:iosbaliblogger admin$ dynamocli import aeguides-baliblogger-main.csv --to aeguides-baliblogger-main Traceback (most recent call last): 22: from /usr/local/bin/dynamocli:23:in <main>' 21: from /usr/local/bin/dynamocli:23:inload' 20: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/bin/dynamocli:6:in <top (required)>' 19: from /Library/Ruby/Gems/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:instart' 18: from /Library/Ruby/Gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' 17: from /Library/Ruby/Gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:ininvoke_command' 16: from /Library/Ruby/Gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' 15: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli.rb:17:inimport' 14: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:19:in start' 13: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:inwrite_records_to_dynamodb_table' 12: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:in each' 11: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:ineach_slice' 10: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:58:in each' 9: from /Library/Ruby/Gems/2.6.0/gems/dynamocli-0.1.5/lib/dynamocli/import.rb:59:inblock in write_records_to_dynamodb_table' 8: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-dynamodb-1.38.0/lib/aws-sdk-dynamodb/client.rb:816:in batch_write_item' 7: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/seahorse/client/request.rb:70:insend_request' 6: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/seahorse/client/plugins/response_target.rb:23:in call' 5: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/aws-sdk-core/plugins/response_paging.rb:10:incall' 4: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in call' 3: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:incall' 2: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call' 1: from /Library/Ruby/Gems/2.6.0/gems/aws-sdk-dynamodb-1.38.0/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:117:incall' /Library/Ruby/Gems/2.6.0/gems/aws-sdk-core-3.77.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The provided key element does not match the schema (Aws::DynamoDB::Errors::ValidationException) Antons-MacBook-Pro:iosbaliblogger admin$

AntonERG commented 4 years ago

Just for testing I:

Really like ur tool, hope you will be able to fix this issue=)

matheussilvasantos commented 4 years ago

Hi, thanks for the feedback.

I've followed the steps you provided and now I know why it's crashing.

That is the header from the CSV exported from AWS Web Console:

"id (N)","name (S)","age (N)","source (S)","tree (S)"

dynamocli expects the exact name of the field and here we have id (N) instead of id.

I suggest you edit the header of the CSV file by now because I don't know if it is a good idea to remove the (N) part on my tool. I'll think about it and let you know if I implement it. You can give suggestions if you want :smile:.

AntonERG commented 4 years ago

Heey Matheus! Thanks for reply, now works like a charm 🎉

Well, you know it might be no need to add support of "id (S)" style header, but it might be useful if you put a chapter in Readme.md with an example of correct header and csv file format.

Still, it is a case that lots of people will try to use Export csv feature from AWS console and after look for solution to import it. To change header line in csv by hands is not a problem at all, but it's nice to know that you have to do it =))

sergiorodriguez82 commented 4 years ago

Adjusting the headers worked! Nice work, Update your usage instructions to consider this scenario. I think is the most frequent!

matheussilvasantos commented 4 years ago

I've made a new release. Now, it is possible to call dynamocli import passing the option --exported-from-aws. This will modify the header. I hope you enjoy. :smile:.

matheussilvasantos commented 4 years ago

Closing this issue.