search

Dynamoid / dynamoid

Ruby ORM for Amazon's DynamoDB.
MIT License
579 stars 195 forks source link

DateTime fields need to be passed as strings or else we get an error #448

Open ychaker opened 4 years ago

ychaker commented 4 years ago

howdy! First of all I'd like to thank everyone for their effort on this gem, I truly appreciate all of you work.

I'm new to the gem and DynamoDB overall, so excuse my noob question.

According to the README (and rspec files), I was under the impression that we could pass a DateTime object to a field that is configured to be of type :datetime, however when trying to set a field to a value like this:

Time.at(str&.to_i).utc.to_datetime

I get an error saying unsupported type, expected Hash, Array, Set, String, Numeric, IO, true, false, or nil, got DateTime

you can see the stack trace here:

{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.211+00:00","v":0,"msg":"Data source: guidepoint"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"list_tables | Request \"{}\" | Response \"{\\\"TableNames\\\":[]}\""}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"(14.98 ms) LIST TABLES"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"(15.16 ms) CACHE TABLES"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":30,"time":"2020-06-12T22:27:48.227+00:00","v":0,"msg":"Creating telematics_local_events table. This could take a while."}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.243+00:00","v":0,"msg":"create_table | Request #<String \"{\\\"TableName\\\":\\\"telematics_local_events\\\",\\\"KeySchema\\\":[{\\\"AttributeName\\\":\\\"id\\\",\\\"KeyType\\\
":\\\"HASH\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"KeyType\\\":\\\"RANGE\\\"}],\\\"AttributeDefinitions\\\":[{\\\"AttributeName\\\":\\\"id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"AttributeType\\\":\\\"N\\\"},{\\\"A
ttributeName\\\":\\\"device_id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"event_type\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"provider\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"vin\\\",\\\"AttributeType\\
\":\\\"S\\\"}],\\\"BillingMode\\\":\\\"PROVISIONED\\\",\\\"ProvisionedThroughput\\\":{\\\"ReadCapacityUnits\\\":10,\\\"WriteCapacityUnits\\\":10},\\\"GlobalSecondaryIndexes\\\":[{\\\"IndexName\\\":\\\"telematics_local_events_index_device_id\\\",\\\"KeySchema\\\":[{\\\"A
ttributeName\\\":\\\"device_id\\\",\\\"KeyType\\\":\\\"HASH\\\"}],\\\"Projection\\\":{\\\"ProjectionType\\\":\\\"ALL\\\"},\\\"ProvisionedThroughput\\\":{\\\"ReadCapacityUnits\\\":100,\\\"WriteCapacityUnits\\\":20}},{\\\"IndexName\\\":\\\"telematics_local_events_index_ev
ent_type\\\",\\\"KeySchema\\\":[{\\\"AttributeName\\\":\\\"event_type\\\",\\\"KeyType\\\":\\\"HASH\\\"}],\\\"Projection\\\":{\\\"ProjectionType\\\":\\\"ALL\\\"},\\\"ProvisionedThroughput\\\":{\\\"ReadC\" ... (1722 bytes)> | Response #<String \"{\\\"TableDescription\\\":
{\\\"AttributeDefinitions\\\":[{\\\"AttributeName\\\":\\\"id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"AttributeType\\\":\\\"N\\\"},{\\\"AttributeName\\\":\\\"device_id\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\"
:\\\"event_type\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"provider\\\",\\\"AttributeType\\\":\\\"S\\\"},{\\\"AttributeName\\\":\\\"vin\\\",\\\"AttributeType\\\":\\\"S\\\"}],\\\"TableName\\\":\\\"telematics_local_events\\\",\\\"KeySchema\\\":[{\\\"A
ttributeName\\\":\\\"id\\\",\\\"KeyType\\\":\\\"HASH\\\"},{\\\"AttributeName\\\":\\\"recorded_at\\\",\\\"KeyType\\\":\\\"RANGE\\\"}],\\\"TableStatus\\\":\\\"ACTIVE\\\",\\\"CreationDateTime\\\":1592000868.240,\\\"ProvisionedThroughput\\\":{\\\"LastIncreaseDateTime\\\":0.
000,\\\"LastDecreaseDateTime\\\":0.000,\\\"NumberOfDecreasesToday\\\":0,\\\"ReadCapacityUnits\\\":10,\\\"WriteCapacityUnits\\\":10},\\\"TableSizeBytes\\\":0,\\\"ItemCount\\\":0,\\\"TableArn\\\":\\\"arn:aws:dynamodb:ddblocal:000000000000:table/telematics_local_events\\\"
,\\\"GlobalSecondaryIndexes\\\":[{\\\"IndexName\\\":\\\"telematics_local_events_index_device_id\\\",\\\"KeySchema\\\":[{\\\"AttributeName\\\":\\\"device_id\\\",\\\"KeyType\\\":\\\"HASH\\\"}],\\\"Projection\\\":{\\\"ProjectionType\\\":\\\"ALL\\\"},\\\"IndexStatus\\\"\" .
.. (2888 bytes)>"}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.243+00:00","v":0,"msg":"(15.86 ms) CREATE TABLE"}
/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:158: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:281: warning: The called method `update_time_to_live' is defined here
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.260+00:00","v":0,"msg":"update_time_to_live | Request \"{\\\"TableName\\\":\\\"telematics_local_events\\\",\\\"TimeToLiveSpecification\\\":{\\\"AttributeName\\\":\\\"expires_at
\\\",\\\"Enabled\\\":true}}\" | Response \"{\\\"TimeToLiveSpecification\\\":{\\\"Enabled\\\":true,\\\"AttributeName\\\":\\\"expires_at\\\"}}\""}
{"name":"runtime","hostname":"docker-desktop","pid":14,"level":20,"time":"2020-06-12T22:27:48.261+00:00","v":0,"msg":"(17.15 ms) UPDATE TIME TO LIVE - [{:table_name=>\"telematics_local_events\", :attribute=>\"expires_at\"}]"}
Error raised from handler method
{
  "errorMessage": "unsupported type, expected Hash, Array, Set, String, Numeric, IO, true, false, or nil, got DateTime",
  "errorType": "Function<ArgumentError>",
  "stackTrace": [
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:53:in `format'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:31:in `block in format'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:30:in `each'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:30:in `with_object'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:30:in `format'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/attribute_value.rb:16:in `marshal'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:195:in `translate'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:189:in `block in map'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:188:in `each'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:188:in `with_object'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:188:in `map'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:205:in `translate_complex'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:197:in `translate'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:170:in `block in structure'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:169:in `each'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:169:in `with_object'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:169:in `structure'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:157:in `apply'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:126:in `translate_input'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:116:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/seahorse/client/plugins/response_target.rb:23:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.99.2/lib/seahorse/client/request.rb:70:in `send_request'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/aws-sdk-dynamodb-1.49.1/lib/aws-sdk-dynamodb/client.rb:3536:in `put_item'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter_plugin/aws_sdk_v3.rb:451:in `put_item'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:150:in `block (3 levels) in <class:Adapter>'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:55:in `benchmark'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:150:in `block (2 levels) in <class:Adapter>'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/adapter.rb:70:in `write'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence/save.rb:23:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence/save.rb:7:in `call'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence.rb:300:in `block (2 levels) in save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:135:in `run_callbacks'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence.rb:299:in `block in save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:135:in `run_callbacks'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/persistence.rb:298:in `save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/dirty.rb:48:in `save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/validations.rb:18:in `save'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/dynamoid-3.5.0/lib/dynamoid/identity_map.rb:64:in `save'",
    "/opt/lib/ingestion/handlers/kinesis.rb:10:in `block in call'",
    "/opt/lib/ingestion/handlers/kinesis.rb:8:in `map'",
    "/opt/lib/ingestion/handlers/kinesis.rb:8:in `call'",
    "/opt/lib/ingestion/handler.rb:15:in `call'",
    "/var/task/handler.rb:8:in `lambda_handler'"
  ]
}

However, if I change the value to be:

Time.at(str&.to_i).utc.to_datetime.to_formatted_s(:iso8601)

while keeping the type as :datetime everything works. I feel it's odd that I'd have to pass a String, which if I'm not mistaken, then gets converted to a DateTime by the type caster, and then to a number.

Am I doing something wrong?

andrykonchin commented 4 years ago

Could you please provide a code snippet to reproduce the issue?

The issue is DynamoDB doesn't support Date/Time/DateTime type internally so such columns are stored either as number of seconds or as a string. Looks like an attribute value somehow isn't type casted by Dynamoid.

jmonsanto commented 4 years ago

Something like this:

record.update do |r|
  r.set data: data
  r.set synced_at: DateTime.current
end

When getting a record from DynamoDB there are no issues casting a DynamoDB string to ruby datetime, however when attempting to save a datetime object, it isn't casted to string to persist on DynamoDB.

But shouldn't datetime values be auto-casted if the declared field attribute is string?

andrykonchin commented 4 years ago

I see. Yes, it's definitely a bug in the #update/#update! methods.

ychaker commented 4 years ago

thank you @jmonsanto for providing the example, I dropped the ball on that when I saw the notification and then completely forgot about it, sorry!