search

Dynamoid / dynamoid

Ruby ORM for Amazon's DynamoDB.
MIT License
580 stars 195 forks source link

Error on listing HABTM associations with sort key #365

Open mightymatth opened 5 years ago

mightymatth commented 5 years ago

This issue is somehow related to #364.

Consider having following models.

class Attender
  include Dynamoid::Document

  table name: :attenders

  range :full_name
  field :email

  has_and_belongs_to_many :groups, class: ::AttenderGroup, inverse_of: :members
end

class AttenderGroup
  include Dynamoid::Document

  table name: :attender_group

  field :name
  field :description

  has_and_belongs_to_many :members, class: ::Attender, inverse_of: :groups
end

Now I can create Attender, AttenderGroup and add attender to group.

Creating attender: 

2.5.5 :004 > ana = Attender.create(full_name: 'Ana')
D, [2019-06-20T19:16:47.473066 #6307] DEBUG -- : list_tables | Request "{}" | Response "{\"TableNames\":[]}"
D, [2019-06-20T19:16:47.473594 #6307] DEBUG -- : (436.71 ms) LIST TABLES
D, [2019-06-20T19:16:47.473648 #6307] DEBUG -- : (437.16 ms) CACHE TABLES
I, [2019-06-20T19:16:47.473696 #6307]  INFO -- : Creating attenders table. This could take a while.
D, [2019-06-20T19:16:47.594624 #6307] DEBUG -- : create_table | Request "{\"TableName\":\"attenders\",\"ProvisionedThroughput\":{\"ReadCapacityUnits\":100,\"WriteCapacityUnits\":20},\"KeySchema\":[{\"AttributeName\":\"id\",\"KeyType\":\"HASH\"},{\"AttributeName\":\"full_name\",\"KeyType\":\"RANGE\"}],\"AttributeDefinitions\":[{\"AttributeName\":\"id\",\"AttributeType\":\"S\"},{\"AttributeName\":\"full_name\",\"AttributeType\":\"S\"}]}" | Response "{\"TableDescription\":{\"AttributeDefinitions\":[{\"AttributeName\":\"id\",\"AttributeType\":\"S\"},{\"AttributeName\":\"full_name\",\"AttributeType\":\"S\"}],\"TableName\":\"attenders\",\"KeySchema\":[{\"AttributeName\":\"id\",\"KeyType\":\"HASH\"},{\"AttributeName\":\"full_name\",\"KeyType\":\"RANGE\"}],\"TableStatus\":\"ACTIVE\",\"CreationDateTime\":1561051007.527,\"ProvisionedThroughput\":{\"LastIncreaseDateTime\":0.000,\"LastDecreaseDateTime\":0.000,\"NumberOfDecreasesToday\":0,\"ReadCapacityUnits\":100,\"WriteCapacityUnits\":20},\"TableSizeBytes\":0,\"ItemCount\":0,\"TableArn\":\"arn:aws:dynamodb:ddblocal:000000000000:table/attenders\",\"BillingModeSummary\":{\"BillingMode\":\"PROVISIONED\",\"LastUpdateToPayPerRequestDateTime\":0.000}}}"
D, [2019-06-20T19:16:47.594959 #6307] DEBUG -- : (121.25 ms) CREATE TABLE
D, [2019-06-20T19:16:47.665408 #6307] DEBUG -- : put_item | Request "{\"TableName\":\"attenders\",\"Item\":{\"created_at\":{\"N\":\"1561051007.595388\"},\"updated_at\":{\"N\":\"1561051007.595639\"},\"id\":{\"S\":\"94326b26-dc3b-4a1e-aa2b-20f7d623bd49\"},\"full_name\":{\"S\":\"Ana\"}},\"Expected\":{\"id\":{\"Exists\":false},\"full_name\":{\"Exists\":false}}}" | Response "{}"
D, [2019-06-20T19:16:47.665960 #6307] DEBUG -- : (69.14 ms) PUT ITEM - ["attenders", {:created_at=>0.1561051007595388e10, :updated_at=>0.1561051007595639e10, :id=>"94326b26-dc3b-4a1e-aa2b-20f7d623bd49", :full_name=>"Ana", :email=>nil, :groups_ids=>nil}, {}]
 => #<Attender:0x00007f9a0bc91b78 @new_record=false, @attributes={:created_at=>Thu, 20 Jun 2019 19:16:47 +0200, :updated_at=>Thu, 20 Jun 2019 19:16:47 +0200, :id=>"94326b26-dc3b-4a1e-aa2b-20f7d623bd49", :full_name=>"Ana", :email=>nil, :groups_ids=>nil}, @associations={}, @attributes_before_type_cast={:created_at=>Thu, 20 Jun 2019 19:16:47 +0200, :updated_at=>Thu, 20 Jun 2019 19:16:47 +0200, :id=>"94326b26-dc3b-4a1e-aa2b-20f7d623bd49", :full_name=>"Ana", :email=>nil, :groups_ids=>nil}, @attributes_changed_by_setter={}, @mutations_from_database=nil, @validation_context=nil, @errors=#<ActiveModel::Errors:0x00007f9a0bc80b70 @base=#<Attender:0x00007f9a0bc91b78 ...>, @messages={}, @details={}>, @previously_changed={}, @mutations_before_last_save=nil>

Creating attender group:

2.5.5 :005 > musicians = AttenderGroup.create(name: 'Musicians', description: 'Active musicians')
I, [2019-06-20T19:17:13.641094 #6307]  INFO -- : Creating attender_group table. This could take a while.
D, [2019-06-20T19:17:13.666881 #6307] DEBUG -- : create_table | Request "{\"TableName\":\"attender_group\",\"ProvisionedThroughput\":{\"ReadCapacityUnits\":100,\"WriteCapacityUnits\":20},\"KeySchema\":[{\"AttributeName\":\"id\",\"KeyType\":\"HASH\"}],\"AttributeDefinitions\":[{\"AttributeName\":\"id\",\"AttributeType\":\"S\"}]}" | Response "{\"TableDescription\":{\"AttributeDefinitions\":[{\"AttributeName\":\"id\",\"AttributeType\":\"S\"}],\"TableName\":\"attender_group\",\"KeySchema\":[{\"AttributeName\":\"id\",\"KeyType\":\"HASH\"}],\"TableStatus\":\"ACTIVE\",\"CreationDateTime\":1561051033.663,\"ProvisionedThroughput\":{\"LastIncreaseDateTime\":0.000,\"LastDecreaseDateTime\":0.000,\"NumberOfDecreasesToday\":0,\"ReadCapacityUnits\":100,\"WriteCapacityUnits\":20},\"TableSizeBytes\":0,\"ItemCount\":0,\"TableArn\":\"arn:aws:dynamodb:ddblocal:000000000000:table/attender_group\",\"BillingModeSummary\":{\"BillingMode\":\"PROVISIONED\",\"LastUpdateToPayPerRequestDateTime\":0.000}}}"
D, [2019-06-20T19:17:13.667083 #6307] DEBUG -- : (25.97 ms) CREATE TABLE
D, [2019-06-20T19:17:13.677350 #6307] DEBUG -- : put_item | Request "{\"TableName\":\"attender_group\",\"Item\":{\"created_at\":{\"N\":\"1561051033.667182\"},\"updated_at\":{\"N\":\"1561051033.667447\"},\"id\":{\"S\":\"e30db0bc-3820-49d2-987b-5b073a9378d2\"},\"name\":{\"S\":\"Musicians\"},\"description\":{\"S\":\"Active musicians\"}},\"Expected\":{\"id\":{\"Exists\":false}}}" | Response "{}"
D, [2019-06-20T19:17:13.677538 #6307] DEBUG -- : (9.93 ms) PUT ITEM - ["attender_group", {:created_at=>0.1561051033667182e10, :updated_at=>0.1561051033667447e10, :id=>"e30db0bc-3820-49d2-987b-5b073a9378d2", :name=>"Musicians", :description=>"Active musicians", :members_ids=>nil}, {}]
 => #<AttenderGroup:0x00007f9a0b109eb8 @new_record=false, @attributes={:created_at=>Thu, 20 Jun 2019 19:17:13 +0200, :updated_at=>Thu, 20 Jun 2019 19:17:13 +0200, :id=>"e30db0bc-3820-49d2-987b-5b073a9378d2", :name=>"Musicians", :description=>"Active musicians", :members_ids=>nil}, @associations={}, @attributes_before_type_cast={:created_at=>Thu, 20 Jun 2019 19:17:13 +0200, :updated_at=>Thu, 20 Jun 2019 19:17:13 +0200, :id=>"e30db0bc-3820-49d2-987b-5b073a9378d2", :name=>"Musicians", :description=>"Active musicians", :members_ids=>nil}, @attributes_changed_by_setter={}, @mutations_from_database=nil, @validation_context=nil, @errors=#<ActiveModel::Errors:0x00007f9a0b108720 @base=#<AttenderGroup:0x00007f9a0b109eb8 ...>, @messages={}, @details={}>, @previously_changed={}, @mutations_before_last_save=nil>

Assigning musician to her group:

2.5.5 :007 > musicians.members << ana
D, [2019-06-20T19:17:40.020569 #6307] DEBUG -- : put_item | Request "{\"TableName\":\"attender_group\",\"Item\":{\"created_at\":{\"N\":\"1561051033.667182\"},\"updated_at\":{\"N\":\"1561051059.993209\"},\"id\":{\"S\":\"e30db0bc-3820-49d2-987b-5b073a9378d2\"},\"name\":{\"S\":\"Musicians\"},\"description\":{\"S\":\"Active musicians\"},\"members_ids\":{\"SS\":[\"94326b26-dc3b-4a1e-aa2b-20f7d623bd49\"]}},\"Expected\":{}}" | Response "{}"
D, [2019-06-20T19:17:40.020814 #6307] DEBUG -- : (25.52 ms) PUT ITEM - ["attender_group", {:created_at=>0.1561051033667182e10, :updated_at=>0.1561051059993209e10, :id=>"e30db0bc-3820-49d2-987b-5b073a9378d2", :name=>"Musicians", :description=>"Active musicians", :members_ids=>#<Set: {"94326b26-dc3b-4a1e-aa2b-20f7d623bd49"}>}, nil]
D, [2019-06-20T19:17:40.038639 #6307] DEBUG -- : put_item | Request "{\"TableName\":\"attenders\",\"Item\":{\"created_at\":{\"N\":\"1561051007.595388\"},\"updated_at\":{\"N\":\"1561051060.021232\"},\"id\":{\"S\":\"94326b26-dc3b-4a1e-aa2b-20f7d623bd49\"},\"full_name\":{\"S\":\"Ana\"},\"groups_ids\":{\"SS\":[\"e30db0bc-3820-49d2-987b-5b073a9378d2\"]}},\"Expected\":{}}" | Response "{}"
D, [2019-06-20T19:17:40.038942 #6307] DEBUG -- : (17.48 ms) PUT ITEM - ["attenders", {:created_at=>0.1561051007595388e10, :updated_at=>0.1561051060021232e10, :id=>"94326b26-dc3b-4a1e-aa2b-20f7d623bd49", :full_name=>"Ana", :email=>nil, :groups_ids=>#<Set: {"e30db0bc-3820-49d2-987b-5b073a9378d2"}>}, nil]
 => #<Attender:0x00007f9a0bc91b78 @new_record=false, @attributes={:created_at=>Thu, 20 Jun 2019 19:16:47 +0200, :updated_at=>Thu, 20 Jun 2019 19:17:40 +0200, :id=>"94326b26-dc3b-4a1e-aa2b-20f7d623bd49", :full_name=>"Ana", :email=>nil, :groups_ids=>#<Set: {"e30db0bc-3820-49d2-987b-5b073a9378d2"}>}, @associations={:groups_ids=>#<Dynamoid::Associations::HasAndBelongsToMany:0x00007f9a0bb2c530 @query={}, @name=:groups, @options={:class=>AttenderGroup, :inverse_of=>:members}, @source=#<Attender:0x00007f9a0bc91b78 ...>, @loaded=false, @target=nil>}, @attributes_before_type_cast={:created_at=>Thu, 20 Jun 2019 19:16:47 +0200, :updated_at=>Thu, 20 Jun 2019 19:17:40 +0200, :id=>"94326b26-dc3b-4a1e-aa2b-20f7d623bd49", :full_name=>"Ana", :email=>nil, :groups_ids=>#<Set: {"e30db0bc-3820-49d2-987b-5b073a9378d2"}>}, @attributes_changed_by_setter={}, @mutations_from_database=nil, @validation_context=nil, @errors=#<ActiveModel::Errors:0x00007f9a0bc80b70 @base=#<Attender:0x00007f9a0bc91b78 ...>, @messages={}, @details={}>, @previously_changed={}, @mutations_before_last_save=nil>

But now when I want to list all my musicians in the group, I get this error:

2.5.5 :008 > musicians.members.all
D, [2019-06-20T19:17:47.913705 #6307] DEBUG -- : describe_table | Request "{\"TableName\":\"attenders\"}" | Response "{\"Table\":{\"AttributeDefinitions\":[{\"AttributeName\":\"id\",\"AttributeType\":\"S\"},{\"AttributeName\":\"full_name\",\"AttributeType\":\"S\"}],\"TableName\":\"attenders\",\"KeySchema\":[{\"AttributeName\":\"id\",\"KeyType\":\"HASH\"},{\"AttributeName\":\"full_name\",\"KeyType\":\"RANGE\"}],\"TableStatus\":\"ACTIVE\",\"CreationDateTime\":1561051007.527,\"ProvisionedThroughput\":{\"LastIncreaseDateTime\":0.000,\"LastDecreaseDateTime\":0.000,\"NumberOfDecreasesToday\":0,\"ReadCapacityUnits\":100,\"WriteCapacityUnits\":20},\"TableSizeBytes\":134,\"ItemCount\":1,\"TableArn\":\"arn:aws:dynamodb:ddblocal:000000000000:table/attenders\",\"BillingModeSummary\":{\"BillingMode\":\"PROVISIONED\",\"LastUpdateToPayPerRequestDateTime\":0.000}}}"
D, [2019-06-20T19:17:47.956185 #6307] DEBUG -- : batch_get_item | Request "{\"RequestItems\":{\"attenders\":{\"Keys\":[{\"id\":{\"S\":\"94326b26-dc3b-4a1e-aa2b-20f7d623bd49\"},\"full_name\":{\"NULL\":true}}]}}}" | Response "{\"__type\":\"com.amazon.coral.validate#ValidationException\",\"message\":\"Invalid attribute value type\"}"
Traceback (most recent call last):
       16: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter.rb:86:in `read'
       15: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter.rb:148:in `block (2 levels) in <class:Adapter>'
       14: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter.rb:54:in `benchmark'
       13: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter.rb:148:in `block (3 levels) in <class:Adapter>'
       12: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:192:in `batch_get_item'
       11: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:192:in `each'
       10: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/dynamoid-3.2.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:220:in `block in batch_get_item'
        9: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-dynamodb-1.30.0/lib/aws-sdk-dynamodb/client.rb:567:in `batch_get_item'
        8: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/seahorse/client/request.rb:70:in `send_request'
        7: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/seahorse/client/plugins/response_target.rb:23:in `call'
        6: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
        5: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
        4: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
        3: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
        2: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-dynamodb-1.30.0/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:117:in `call'
        1: from /Users/mpevec/.rvm/gems/ruby-2.5.5@attendance-tracker/gems/aws-sdk-core-3.54.2/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
Aws::DynamoDB::Errors::ValidationException (Invalid attribute value type)

I know that this is related to setting sort key on Attender; it only stores ids without store key (:groups_ids=>#<Set: {"e30db0bc-3820-49d2-987b-5b073a9378d2"}>) and when I try to list them, it uses batch_get_item method with wrong query parameters ({\"Keys\":[{\"id\":{\"S\":\"94326b26-dc3b-4a1e-aa2b-20f7d623bd49\"},\"full_name\":{\"NULL\":true}}]})

I think that, at the moment, querying on associations can be used only without using sort keys. Also, I'm not sure that my statements are all correct because I'm new in DynamoDB.

andrykonchin commented 5 years ago

Yeah, you are completely right. Dynamoid's associations don't work with sort keys and there is already issue about this problem (https://github.com/Dynamoid/dynamoid/issues/316).

Associations here are too simple and limited and require complete reimplementation. That's why this known issue still isn't fixed.

mightymatth commented 5 years ago

Sorry, I haven't look for open issue about that problem.

Anyway, is there any reason why we don't use query (docs here) method instead of batch_get_item when we have range key in our association? The same question for my previous issue #364.

andrykonchin commented 5 years ago

Hmm, I don't understand your question.

Current behavior I would consider as a bug and support of association with composite primary key should be added.

From the other hand BatchGetItem is the most efficient and cheapest way to load items by primary key for both simple scalar primary key and composite one.

mightymatth commented 5 years ago

From the other hand BatchGetItem is the most efficient and cheapest way to load items by primary key for both simple scalar primary key and composite one.

I agree with you, it's the most efficient way, but only if you have both; the primary key and composite one. Using Query API instead of BatchGetItem API, we could be able to get items just with primary key, which would resolve our problem while listing associations with range key. This is maybe less efficient way, but maybe it's better to use Query API in these cases before any bigger refactoring on associations are being done. This is just a proposal which acts as a workaround. Just speaking out loud.

andrykonchin commented 5 years ago

Using Query API instead of BatchGetItem API, we could be able to get items just with primary key, which would resolve our problem while listing associations with range key

I assume you meant partition key when wrote just with primary key.

Yeah, Query api call with condition on partition key like id: [id1, id2, id3, ...] will work for class with sort key (range key) and items with cached partition keys will be loaded. But result will contain associated models as well as some arbitrary models with different sort key (range key) but the same partition one. So result will contain both associated models and not associated.

I would say it would be very dangerous bug because it would be silent. Am I right?

Anyway, please provide some code example or prototype to ensure we are on the same page.

mightymatth commented 5 years ago

I assume you meant partition key when wrote just with primary key.

Yes, I was thinking about partition key.

I would say it would be very dangerous bug because it would be silent. Am I right?

I'm concerned about this too. I don't have so much practice with DynamoDB/Dynamoid so I don't have an answer at the moment. When I catch some time, I'll try to make changes and test if it works as expected, then we can discuss about it.