Hashsets Hits duplicate entries

[doomguy]

Software Version: Autopsy 4.7.0 Windows

Hi there,

when I run the "Hash Lookup" Ingester a second time on the same Data Source, the "Hashset Hits" will be filled with duplicates.

The Ingester should check whether the entry is already present before adding it to the "Hashset Hits" again.

Regards

[marshalla99]

That happens with all the ingesters. Why do you need to run them a second time?

[doomguy]

Situations like:

• I forgot to set some filters and now Autopsy is ingesting 500.000 JPGs which takes 2 days to finish, but I'm only interested in ASCII files (like .php). So I abort the job.
• I added a new hashset with IOCs and rerun the hash ingester to flag the files.

Regards

From: "marshalla99" notifications@github.com To: "sleuthkit/autopsy" autopsy@noreply.github.com Cc: "doomguy" doomguy@foxac1d.com, "Author" author@noreply.github.com Sent: Saturday, June 16, 2018 12:52:49 AM Subject: Re: [sleuthkit/autopsy] Hashsets Hits duplicate entries (#3843)

That happens with all the ingesters. Why do you need to run them a second time?

— You are receiving this because you authored the thread. Reply to this email directly, [ https://github.com/sleuthkit/autopsy/issues/3843#issuecomment-397761696 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/ABv6YLxSGFSfEiXoQ9mL6INhX5-ldXtoks5t9DrBgaJpZM4Ug6FD | mute the thread ] .

[rcordovano]

Duplication of hash set hit artifacts when the hash lookup module is run more than once was eliminated in the Autopsy 4.9.0 release. We will be steadily working on preventing the duplication of other types of artifacts as we move forward. For the upcoming 4.10.0 release, we expect to prevent the creation of duplicate EXIF metadata artifacts and most duplicate interesting item artifacts.