search

HaveIBeenPwned / PwnedPasswordsDownloader

A tool to download all Pwned Passwords hash ranges and save them offline so they can be used without a dependency on the k-anonymity API
BSD 3-Clause "New" or "Revised" License
629 stars 49 forks source link

Incorrect progress percent shown #73

Closed oyeaussie closed 3 months ago

oyeaussie commented 3 months ago 
$ haveibeenpwned-downloader pwnedpasswords -s false -o
Failed attempt #0 while fetching https://api.pwnedpasswords.com/range/3F6E7. Exception is HttpRequestException and message: Error while copying content to a stream..
Failed attempt #0 while fetching https://api.pwnedpasswords.com/range/3F6EA. Exception is HttpRequestException and message: Error while copying content to a stream..
Failed attempt #0 while fetching https://api.pwnedpasswords.com/range/41DF8. Exception is HttpRequestException and message: Error while copying content to a stream..
Failed attempt #0 while fetching https://api.pwnedpasswords.com/range/41DF9. Exception is HttpRequestException and message: Error while copying content to a stream..

Hash ranges downloaded ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  39% 00:00:00  

Finished downloading all hash ranges in 1,946,884ms (211.82 hashes per second).
We made 412,386 Cloudflare requests (avg response time: 284.26ms). Of those, Cloudflare had already cached 412,386 requests, and made 0 requests to the Have I Been Pwned origin server.

At 39% the download says that it finished downloading all hash ranges. Is this correct?

oyeaussie commented 3 months ago 
Failed attempt #0 while fetching https://api.pwnedpasswords.com/range/1D44E. Exception is HttpRequestException and message: Error while copying content to a stream..
Failed attempt #0 while fetching https://api.pwnedpasswords.com/range/1D450. Exception is HttpRequestException and message: Error while copying content to a stream..

Hash ranges downloaded ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  11% 00:00:00  

Finished downloading all hash ranges in 480,059ms (254.74 hashes per second).
We made 122,292 Cloudflare requests (avg response time: 197.59ms). Of those, Cloudflare had already cached 122,292 requests, and made 0 requests to the Have I Been Pwned origin server.

I tried it again and this time it stopped at 11%. The message "Finished downloading all hash ranges" is quite misleading. If there is a problem while downloading, should the app throw an exception or something.

oyeaussie commented 3 months ago

I was able to get a clean download this time

haveibeenpwned-downloader pwnedpasswords -s false

Hash ranges downloaded ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 00:00:00  

Finished downloading all hash ranges in 3,135,526ms (334.42 hashes per second).
We made 1,048,576 Cloudflare requests (avg response time: 22.69ms). Of those, Cloudflare had already cached 1,048,576 requests, and made 0 requests to the Have I Been Pwned origin server.

Are the numbers shown above are correct?

dlosyankov commented 3 months ago

@oyeaussie
Have the similar behavior with different numbers, but the tool in my case wasn't throwing any HttpRequestException:

C:\hvibeenpwnd_tool\haveibeenpwned-downloader.exe C:\hashes -s false -p 64
Hash ranges downloaded: 0%
Hash ranges downloaded: 0%
Hash ranges downloaded: 0%
Hash ranges downloaded: 0%
Hash ranges downloaded: 1%
Hash ranges downloaded: 1%
Hash ranges downloaded: 1%
Hash ranges downloaded: 1%
Hash ranges downloaded: 1%
Hash ranges downloaded: 2%
Finished downloading all hash ranges in 289,921ms (84.33 hashes per second).
We made 24,450 Cloudflare requests (avg response time: 745.07ms). Of those, 
Cloudflare had already cached 24,450 requests, and made 0 requests to the Have I
Been Pwned origin server.

May I ask you what is the total size of your pwd file?

oyeaussie commented 3 months ago

@dlosyankov

🤣 I got a bit frustrated and wrote my own app in PHP yesterday. Try it and let me know if there are any issues

https://github.com/oyeaussie/PHPPwnedPasswordsDownloader