bitshares / bitshares-core

BitShares Blockchain node and command-line wallet
https://bitshares.github.io/
Other
1.17k stars 643 forks source link

Store object arrays in ES as `nested` type instead of `object` #2568

Open abitmore opened 2 years ago

abitmore commented 2 years ago

User Story

Object arrays stored in ES are flattened by default (see https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html).

For example, for account_auths in account history object with ID 2.9.671969, the original input was

"account_auths": [
    [ "1.2.121", 30 ],
    [ "1.2.2204", 15 ],
    [ "1.2.3284", 10 ]
] 

After processed by our code in ES plugin (#2565), it becomes

"account_auths_object": [
    { "key_string": "1.2.121", "data_int": 30 },
    { "key_string": "1.2.2204", "data_int": 15 },
    { "key_string": "1.2.3284", "data_int": 10 }
]

But in ES it got flattened as

"account_auths_object.key_string": [ "1.2.121", "1.2.2204", "1.2.3284" ],
"account_auths_object.data_int": [ 30, 15, 10 ]

Screenshot: image

If we query with "key_string" : "1.2.121" and "data_int" : 15, this record will be returned. This behavior is not desired.

To fix this, we need to store account_auths_object as nested type but not automatically (by the default dynamic mapping rules) as object. It means we need to specify our own explicit mappings.

And there are more fields. The most complex case is multi-level nested proposals, although most of them were malformed unexpectedly.

The challenges are

Impacts Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.

CORE TEAM TASK LIST

abitmore commented 2 years ago

There was a template mentioned in another issue: https://github.com/bitshares/bitshares-core/issues/681#issuecomment-367224169

I'm using template to pre-define the settings:

$ curl -XPUT 'http://localhost:9200/_template/graphene' -d '{
  "index_patterns" : ["graphene-*"],
  "settings": { "number_of_shards": 2,
    "index": {
      "translog": {
        "retention": {
          "size": "512mb", "age": "300s"
        }
      }
    }
  }
}' -H 'Content-Type: application/json