zmap / zmap

ZMap is a fast single packet network scanner designed for Internet-wide network surveys.
https://zmap.io
Apache License 2.0
5.43k stars 907 forks source link

Use Judy1 on 64-bit Systems for Deduplication #849

Open zakird opened 4 months ago

zakird commented 4 months ago

Cachehash uses JudyHS on the backend for our deduplication. We originally used this because we wanted ZMap to remain usable on 32-bit systems. However, given that deduplication maps to have be fairly large empirically (>1M items), it would be helpful to use Judy1 when we are on a 64-bit system to save memory. This will require having a bitmap-style cachehash library that uses J1 on the backend.