Returned status: Forbidden

kristianrl commented 5 years ago

I received this error when running the plugin, after clicking OK in the settings prompt for the plugin. I haven't previously used the plugin, so I do not know if this is a temporary issue, but https://haveibeenpwned.com/ is working. I believe it could be an HTTP 403.

2019-01-17 09_01_51-haveibeenpwned checker

Let me know if I can provide more relevant information.

SigHunter commented 5 years ago

happens to me aswell, tried it for the first time today because of the "Collection #1" list

NathanGloyn commented 5 years ago

I fired up fiddler to see what the response was and it appears that the plugin has breached the acceptable use policy (html returned below) but it isn't apparent if this is a rate limiting issue or if it is too many requests from single IP


<head>
<title>Request Blocked</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<h1>You have been blocked from accessing this resource on Have I Been Pwned.</h1>

<p>This may be due to violating one or more of <a href="https://haveibeenpwned.com/API/v2#AcceptableUse">the acceptable use terms of the API</a>.</p>

<p>It may also be due to your traffic patterns being similar to other users who may have violated the acceptable use terms.</p>

<p>Tips to avoid requests being blocked include:</p>
<ol>
<li>Stick well within the published rate limit</li>
<li>Don't distribute requests over multiple IP addresses in an attempt to circumvent the rate limit</li>
<li>Only query the email addresses of people who have a reasonable expectation that you should do so</li>
<li>Avoid prolonged querying of the API over an extended period of time</li>
</ol>

<div class="cf-error-details cf-error-1020">
  <h1>Access denied</h1>
  <p>This website is using a security service to protect itself from online attacks.</p>
  <ul class="cferror_details">
    <li>Ray ID: 49a780168d393542</li>
    <li>Timestamp: 2019-01-17 08:37:31 UTC</li>
    <li>Your IP address: 79.68.4.48</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: haveibeenpwned.com/api/v2/breachedaccount/Nathan </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_21F109</li>
    <li>User-Agent: KeePass HIBP Checker/2.40.0.0</li>
  </ul>
</div>

</body>
</html>```

NathanGloyn commented 5 years ago

Having checked the Pwnd Password docs here I don't believe its anything to do with rate limiting as that should return a 429 but instead I'm seeing a 403

andrew-schofield commented 5 years ago

The plugin should always be rate limited in its requests to the API, so I wonder if it had accidentally triggered the "multiple IP addresses" check because obviously there will be a lot of requests from different IPs. The plugin sends a versioned user agent string too, so I might reach out to Troy to find out what's going wrong.

vdenisov commented 5 years ago

Could this be due to the sheer number of checks from a single IP? When I tried running it, it went through ~half my username list, then started getting an error for every entry, even if I waited for several seconds between attempts. Also - it's impossible to gracefully cancel plugin execution is this case, had to terminate KeePass process to abort.

vdenisov commented 5 years ago

One more thing I'd noticed: if progress indicator is to be believed, it definitely checks faster then once every 1.6s (I'd say, at least couple entries per second).

kapitanluffy commented 5 years ago

I was accessing the API manually earlier and got the same forbidden page. I think there is something wrong with hibp's API for now. Accessing other endpoints (pastes. pwned-pws) are accessible though.

I also tweeted haveibeenpwnd about it to make sure.

janKir commented 5 years ago

Although the problem might be API related, there should be the possiblity to cancel the process. These error messages pop up over and over again ...

Sanaki commented 5 years ago

Just ran site/service and username checks and had no issues myself. Either it's a temporary hiccup that resolved or my username count is severely low compared to yours.

marcelser commented 5 years ago

Works fine for me

peterpan123123 commented 5 years ago

Yep, works again since today.

NathanGloyn commented 5 years ago

I can reproduce the error when checking site or password by selecting the option "Check all supported breaches" without this option checked it runs successfully.

When trying to run the check via username even without the "Check all supported breaches" I'm getting the forbidden error but it appears that it is trying to get breaches so guessing its related.

marcofranssen commented 5 years ago

Same here Unable to check haveibeenpwned.com (return Status: Forbidden) See https://httpstatuses.com/403. Should this plugin be updated with credentials to authenticate against haveibeenpwned.com?

Even more anoying is that you have to dismiss the dialog for every single entry in your keepass file. That kept me busy for 15 minutes. :(

internationils commented 5 years ago

I'm stuck on this as well, under Ubuntu (see https://github.com/andrew-schofield/keepass2-haveibeenpwned/issues/58 ) ... Hint: you can close the progress dialog, and close KeePass to stop the dialog flood...

p3l4h0 commented 5 years ago

Same Error only with usernames @NathanGloyn

Having checked the Pwnd Password docs here I don't believe its anything to do with rate limiting as that should return a 429 but instead I'm seeing a 403

I'm not sure where to get the Error code 403 but this is about: no user agent: https://haveibeenpwned.com/API/v2#UserAgent

andrew-schofield commented 5 years ago

Same Error only with usernames @NathanGloyn

Having checked the Pwnd Password docs here I don't believe its anything to do with rate limiting as that should return a 429 but instead I'm seeing a 403

I'm not sure where to get the Error code 403 but this is about: no user agent: https://haveibeenpwned.com/API/v2#UserAgent

That shouldn't be the issue, as the plugin always sets the user agent appropriately https://github.com/andrew-schofield/keepass2-haveibeenpwned/blob/d78ade54c0710f4e1076f34a6a675a833f4f2515/HaveIBeenPwned/HaveIBeenPwnedExt.cs#L46 However if this is being stripped from the request for whatever reason, there's not much I can do.

strayge commented 5 years ago

However if this is being stripped from the request for whatever reason, there's not much I can do.

It's not stripped. Proper user-agent showed in cloudflare response.

Also same error happens for me by test link (from api page) opened in browser.

andrew-schofield commented 5 years ago

However if this is being stripped from the request for whatever reason, there's not much I can do.

It's not stripped. Proper user-agent showed in cloudflare response.

Also same error happens for me by test link (from api page) opened in browser.

Yeah, the direct link returns a 403 for me too.

duichwer commented 5 years ago

Requests to the breaches and pastes APIs are limited to one per every 1500 milliseconds each from any given IP address (an address may request both APIs within this period).

As mentioned on the Api-Website

And you have already added Delay but only for usernameChecks and not for the other checks. Maybe this causes this issue.

https://github.com/andrew-schofield/keepass2-haveibeenpwned/blob/d78ade54c0710f4e1076f34a6a675a833f4f2515/HaveIBeenPwned/BreachCheckers/HaveIBeenPwnedUsername/HaveIBeenPwnedUsernameChecker.cs#L102

andrew-schofield commented 5 years ago

Requests to the breaches and pastes APIs are limited to one per every 1500 milliseconds each from any given IP address (an address may request both APIs within this period).

As mentioned on the Api-Website

And you have already added Delay but only for usernameChecks and not for the other checks. Maybe this causes this issue.

keepass2-haveibeenpwned/HaveIBeenPwned/BreachCheckers/HaveIBeenPwnedUsername/HaveIBeenPwnedUsernameChecker.cs

Line 102 in d78ade5

await Task.Delay(1600);

Unless Troy has changed something and not updated the docs, I don't think this is the issue.

The API page specifically says

There is no rate limit on the Pwned Passwords API.

Additionally the site/service check only does 1 call to retrieve the entire list of breaches, and then compares entries locally.

NathanGloyn commented 5 years ago

I have installed latest release today and can confirm that all of the searches are working correctly (regardless of if "Check all breaches" is checked or unchecked 👍

p3l4h0 commented 5 years ago

Unfortunately not for me: Update reinstalled installed today, and with both ways, checked and uncheked keepass

andrew-schofield commented 5 years ago

Is this still an issue for people?

BenTen commented 5 years ago

I just check with Keepass 2.41 and the plugin 1.3.1, but the issue is not resolved ("Returned status: Forbidden")

andrew-schofield commented 5 years ago

Right, I've fired off an email to Troy about this. Hopefully he can offer some insight into why this is happening.

modelrockettier commented 5 years ago

I just got ran into this last night, here's what shows up when I manually attempt to query the API (example URL)

You have been blocked from accessing this resource on Have I Been Pwned

This may be due to violating one or more of the acceptable use terms of the API or for not complying with the API specifications. It may also be due to your traffic patterns being similar to other users who may have violated the acceptable use terms.

Tips to avoid requests being blocked include:

Stick well within the published rate limit

Don't distribute requests over multiple IP addresses in an attempt to circumvent the rate limit

Only query the email addresses of people who have a reasonable expectation that you should do so

Avoid prolonged querying of the API over an extended period of time

Clearly identify your app in the user agent string per the API docs.

If you believe your request meets these requirements and was still blocked, please send this entire response body along with any communication you send regarding the error.

Access denied

This website is using a security service to protect itself from online attacks.

Ray ID: 4a5297ee9b01c99f

Timestamp: 2019-02-07 02:58:12 UTC

Your IP address: 75.172.3.60

Requested URL: haveibeenpwned.com/api/v2/breachedaccount/test@example.com

Error reference number: 1020

Server ID: FL_28F150

User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

modelrockettier commented 5 years ago

So I've done a bit more investigation in my case, and the plugin is still popping up an error message saying Unable to check haveibeenpwned.com (returned Status: Forbidden).

All I have to do to reproduce it is check for breaches based on username and uncheck all boxes.

The error page from my previous comment seems to have been caused by the haveibeenpwned API rejecting requests from web browser user agents (as is documented in the API docs).

The above API URL does work over curl though (with the user agent KeePass HIBP Checker/1.3.1).

andrew-schofield commented 5 years ago

If you can reliably reproduce the error from within keepass, can you try capturing the error response by using fiddler (https://www.telerik.com/fiddler) or a similar tool?

Troy needs to see the exact response returned by cloudflare in order to debug this.

You're right that using a browser is not a valid test, as cloudflare rejects this based on the user agent, but it is interesting that the request fails from within keepass, but works via curl.

I haven't been able to reproduce the error on any of my systems (using at least 3 different IP addresses from different locations), either using the plugin, or with postman etc.

p3l4h0 commented 5 years ago

Downloaded and installed and "learned" Fiddler. Question

GET https://haveibeenpwned.com/api/v2/breachedaccount/****** HTTP/1.1
User-Agent: KeePass HIBP Checker/2.41.0.0
Accept: application/json
Host: haveibeenpwned.com
Cookie: __cfduid=dd4a76c0ce61505da0a219e7c1d168ae01549621013
Connection: Keep-Alive

First answer:

HTTP/1.1 403 Forbidden
Date: Fri, 08 Feb 2019 10:20:12 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Cache-Control: max-age=15
Expires: Fri, 08 Feb 2019 10:20:27 GMT
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Content-Type-Options: nosniff
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4a5d5cc34e15c4b0-DUS
Content-Length: 1951

<!DOCTYPE html>
<head>
<title>Request Blocked</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<h1>You have been blocked from accessing this resource on Have I Been Pwned</h1>

<p>This may be due to violating one or more of <a href="https://haveibeenpwned.com/API/v2#AcceptableUse">the acceptable use terms of the API</a> or for not complying with <a href="https://haveibeenpwned.com/API/v2">the API specifications</a>. It may also be due to your traffic patterns being similar to other users who may have violated the acceptable use terms.</p>

<p>Tips to avoid requests being blocked include:</p>
<ol>
<li>Stick well within the published rate limit</li>
<li>Don't distribute requests over multiple IP addresses in an attempt to circumvent the rate limit</li>
<li>Only query the email addresses of people who have a reasonable expectation that you should do so</li>
<li>Avoid prolonged querying of the API over an extended period of time</li>
<li>Clearly identify your app in the user agent string <a href="https://haveibeenpwned.com/API/v2#UserAgent">per the API docs</a>.</li>
</ol>
<p>If you believe your request meets these requirements and was still blocked, please send this entire response body along with any communication you send regarding the error.</p>
<div class="cf-error-details cf-error-1020">
  <h1>Access denied</h1>
  <p>This website is using a security service to protect itself from online attacks.</p>
  <ul class="cferror_details">
    <li>Ray ID: 4a5d5cc34e15c4b0</li>
    <li>Timestamp: 2019-02-08 10:20:12 UTC</li>
    <li>Your IP address: *********</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: haveibeenpwned.com/api/v2/breachedaccount/****** </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_48F23</li>
    <li>User-Agent: KeePass HIBP Checker/2.41.0.0</li>
  </ul>
</div>

</body>
</html>

Reconnect to provider some times to get new Ip adress -> Same answer

Start VPN it works

andrew-schofield commented 5 years ago

Thanks, I've forwarded the details on to Troy. It does look like cloudflare using an over enthusiastic IP range blocker that is causing this.

modelrockettier commented 5 years ago

So for me, this turned out to be an IP blocking issue.

I had previously been doing some testing with IE over a VPN and forgot to reset its proxy settings, and it looks like Keepass picks up the IE proxy settings, so all of the plugin's traffic was going over the VPN whereas curl, etc. did not.

Trying the same curl commands on the other end of the VPN failed, and after clearing the IE proxy settings, the keepass plugin started working again.

Braintelligence commented 5 years ago

Same problem. Fresh install of keepass and this plugin. No proxy involved. No other users on this IP even looking at haveibeenpwned.

mykey commented 5 years ago

I got the same error here when letting the plugin search for usernames and ticking the box "Check all breaches". I don't get the error when I search for passwords. I have fiddler and burp on the box and might set up a proxy later to help debug (but helas today and this weekend will not be that time. Way too much work).

andrew-schofield commented 5 years ago

OK everyone, I've chatted to Troy about this, and he says the only way for him to get to the bottom of this is for him to get copies of the error response returned by cloudflare, including the IP address (if this is missing, it's useless to him).

Since this isn't easily accessible when using KeePass (unless you're willing to run Fiddler, or another inspecting proxy), I'd like to have a show of hands of who would be willing to test a version of keepass that specifically saves these cloudflare error messages so we can send them to Troy.

Please 👍 this message if you're happy to take part so I know it's worth my time creating a debug build.

SoyRA commented 5 years ago

(...) is for him to get copies of the error response returned by cloudflare, including the IP address (...)

You mean this? Do I share the result here or what? 😅 * If you're saying you'll give us a version that will send the data automatically...I accept.

I've KeePass and Plugin updated, and I always have the same problem. :c!

andrew-schofield commented 5 years ago

@SoyRA No, that link will always fail because of the browser user-agent. We need the response from a valid request from KeePass. I'll work on getting a debug version created.

mykey commented 5 years ago

Thanks for your work. Happy to help out if needed.

Firehawke commented 5 years ago

This is hitting me as well, and I'd be willing to run a beta plugin for testing purposes if it'll help solve this once and for all.

andrew-schofield commented 5 years ago

I've attached a "rough-and-ready" debug version of the plugin which will store the cloudflare error responses returned from HIBP.com

This will dump a txt file into %TEMP%\KeePassHIBP and you can either attach the responses here, or email them to the.uncle.fungus@gmail.com I'll collate the responses and send them off to Troy so he can take a look.

Debug download: HaveIBeenPwned.zip

BenTen commented 5 years ago

I sent an email with debug file.

vdenisov commented 5 years ago

Here's two of my error response. Received when trying to run username check. Both site and password checks have worked.

srihwf1e.dj1.txt bcqq0wzq.k3x.txt

mykey commented 5 years ago

I've ran the beta plugin and also sent you an email with a number of debug files.

As stated previously I can only get errors by searching on username.

andrew-schofield commented 4 years ago

I believe this should be fixed now that HIPB API v3 has been released, however I'd be interested in knowing if people are still seeing this issue with >= v1.3.4 of the plugin.

YogSottot commented 4 years ago

Tested with 1.3.4 This problem has been resolved.

SoyRA commented 4 years ago

Let's see, everything works. The only "problem" is trying to use (...) based on username because it asks me for the API Key...But taking that out, everything's fine. :P * KeePass and this Plugins updated to the latest version.

andrew-schofield / keepass2-haveibeenpwned

Returned status: Forbidden #56

You have been blocked from accessing this resource on Have I Been Pwned

Access denied