search

sonyxperiadev / lumber-mill

Where logs are cut into lumber
Apache License 2.0
12 stars 6 forks source link

Implement checksum function to ease deduplication of events with document_id #10

Closed jrask closed 8 years ago

jrask commented 8 years ago

Something like this for an S3 downloaded file

map (checksum ( 
    source: '{key} | {bucket} | {row}'
))

Since is fairly safe fo most cases

map (checksum ( 
    source: '{message}'
))

Results in

{
    ....
    "fingerprint": "...."
}