search

sunitparekh / data-anonymization

Want to use production data for testing, data-anonymization can help you.
MIT License
459 stars 92 forks source link

undefined method `to_sym' for nil:NilClass (NoMethodError) when anonymizing json column in mysql #48

Closed databu closed 7 years ago

databu commented 7 years ago

I'm trying to anonymize a json column stored in a mysql table. The structure of the json looks like this: [[{ "target": "<name>", ... } ... ] ... ].

I'm mapping the "target" values to "some name" like so: 

anonymize('some_json_field') do |jsonField|
  dataArray = JSON::parse jsonField.value
  mappedArray = dataArray.map { |path| path.map { |node| node.merge({ "target" => "some name" }) } }
  JSON::generate(mappedArray)
end

However this throws an error:

E, [2017-05-29T09:46:45.362674 #43754] ERROR -- : Errors while processing table '<some table>':
E, [2017-05-29T09:46:45.362761 #43754] ERROR -- : undefined method `to_sym' for nil:NilClass (NoMethodError)

Full stack trace:

anonymize.rb:4:in `<main>'
E, [2017-05-29T09:46:45.364929 #43754] ERROR -- : ~/.rvm/gems/ruby-2.4.0/gems/composite_primary_keys-8.1.6/lib/composite_primary_keys/sanitization.rb:25:in `block in expand_hash_conditions_for_aggregates'
        ~/.rvm/gems/ruby-2.4.0/gems/composite_primary_keys-8.1.6/lib/composite_primary_keys/sanitization.rb:18:in `each'
        ~/.rvm/gems/ruby-2.4.0/gems/composite_primary_keys-8.1.6/lib/composite_primary_keys/sanitization.rb:18:in `expand_hash_conditions_for_aggregates'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/relation/query_methods.rb:957:in `build_where'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/relation/query_methods.rb:584:in `where!'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/relation/query_methods.rb:574:in `where'
        ~/.rvm/gems/ruby-2.4.0/gems/composite_primary_keys-8.1.6/lib/composite_primary_keys/relation.rb:63:in `_update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/persistence.rb:515:in `_update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/composite_primary_keys-8.1.6/lib/composite_primary_keys/locking/optimistic.rb:6:in `_update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/attribute_methods/dirty.rb:129:in `_update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:310:in `block in _update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
        ~/.rvm/gems/ruby-2.4.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_update_callbacks'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:310:in `_update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/timestamp.rb:70:in `_update_record'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/persistence.rb:504:in `create_or_update'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:302:in `block in create_or_update'
        ~/.rvm/gems/ruby-2.4.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
        ~/.rvm/gems/ruby-2.4.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_save_callbacks'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:302:in `create_or_update'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/persistence.rb:142:in `save!'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/validations.rb:43:in `save!'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:291:in `block in save!'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:220:in `transaction'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:291:in `save!'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/strategy/blacklist.rb:14:in `process_record'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/strategy/base.rb:151:in `process_record_if'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/strategy/base.rb:114:in `block in process_table'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:46:in `each'
        ~/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:46:in `each'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/strategy/base.rb:111:in `process_table'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/strategy/base.rb:97:in `process'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/core/database.rb:68:in `block in anonymize'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/core/database.rb:66:in `each'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/core/database.rb:66:in `anonymize'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/core/database.rb:46:in `anonymize'
        ~/.rvm/gems/ruby-2.4.0/gems/data-anonymization-0.7.4/lib/core/dsl.rb:10:in `database'
        anonymize.rb:4:in `<main>'
databu commented 7 years ago

Oh, maybe I should add that the MySQL type of some_json_field is just text, not the native JSON datatype supported by newer MySQL versions.

databu commented 7 years ago

Nevermind. The problem was something completely different: I didn't specify the primary key column. I thought you don't have to do it if it's called "id", but in this table, it isn't the first column, which seems to break the default mechanism.