search

The-Shadowserver-Foundation / api_utils

Sample programs to access the API
GNU General Public License v3.0
64 stars 7 forks source link

Error: merging dicts would replace value #3

Closed AyoobAli closed 1 year ago

AyoobAli commented 1 year ago

I faced an issue while trying to use the ElasticSearch script, I was getting the following error for the compromised_website report:

TypeError: Type mismatch at key `server`: merging dicts would replace value `{'naics': '517311'}` with `openresty`. This is likely due to dotted keys in the event dict being turned into nested dictionaries, causing a conflict.

It turned out it needs some adjustment in the mapping file.

The reason for this is that the Elastic field mapper has the server field as a dict which includes naics and sector. And in some reports of shadowserver we have a column with the name server and this is causing the ecs_logging to replace the server column with the dict when merging the values.

This can be fixed by adding the server column to the mapper. You can do that by adding the following line to the map section in map.json

"server" : "extra.server.name",

NOTE: for now make sure to disable the auto_update in the config file so it doesn't replace the edited mapper.

This need to be fix for all in the following file:

https://interchange.shadowserver.org/elasticsearch/v1/map
elsif2 commented 1 year ago

Thank you for your support. The mapping has been updated.