ndejong
commented
5 months ago The hibp-downloader tool does not download into a single file, although we provide a generate feature to create a single file if you'd really like one.
You might find it helpful to get a mental-model of how this tool works -
- (1) we download and write all the hibp data chunks as-is from the endpoint keeping it in gzip compressed format
- (2) we write each hibp data-chunk into a path defined by you using
--data-path - (3) when we later re-run we check the local TTL (by default 12 hours) to decide if we are going to make a request, and then use a request with an
etagto the hibp endpoint so it only sends us new data if the etag has changed.
This way a re-run of the tool inside the TTL (12 hours) we do not generate unnecessary requests; and if the HIBP data chunk has changed then you'll get the update.
If you really want to do a diff between before and after, then you could use the generate feature to create the export files before and after.
Hi! Can I download the updated data into separate files? For example, the original database contained 100k hashes, in a month I will want to update. Where will new hashes be saved?