This PR changes the summarize / aggregation phase of the IP beacons, proxy beacons, SNI beacons, and unique connection analyses to perform a total roll-up of the currently imported data for each host rather than a roll-up for the data that was just imported. This allows us to store a single copy of the roll-ups in the host collection for each internal host instead of a record per chunked import. In turn, we no longer have to worry about old roll-ups falling out of sync with new data. Closes #800
Testing:
I've tested this PR against the logs linked in #800 and ensured that the problem is fixed there. I am currently testing to ensure that we don't have any regressions on other datasets.
This PR changes the summarize / aggregation phase of the IP beacons, proxy beacons, SNI beacons, and unique connection analyses to perform a total roll-up of the currently imported data for each host rather than a roll-up for the data that was just imported. This allows us to store a single copy of the roll-ups in the host collection for each internal host instead of a record per chunked import. In turn, we no longer have to worry about old roll-ups falling out of sync with new data. Closes #800
Testing: I've tested this PR against the logs linked in #800 and ensured that the problem is fixed there. I am currently testing to ensure that we don't have any regressions on other datasets.