search

rekby / lets-proxy2

Reverse proxy with automatically obtains TLS certificates from Let's Encrypt
MIT License
96 stars 18 forks source link

Allowed list of domains #147

Open indrora opened 3 years ago

indrora commented 3 years ago

I've had a lot of ratelimiting because of clients that send bogus HTTPS requests:

Dec 13 14:18:02 zaibatsutel lets-proxy[29713]: 2020-12-13T14:18:02.906-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "47e16fb2-0884-4fc9-ab22-b7f8e2dcc21a", "domain": "www.shop.lyuaunew.zaibatsutel.net (punycode:www.shop.lyuaunew.zaibatsutel.net)", "original_domain": "www.shop.lyuaunew.zaibatsutel.net"}
Dec 13 14:18:02 zaibatsutel lets-proxy[29713]: 2020-12-13T14:18:02.923-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "33bfc9e3-7f9c-4d54-a5ec-12c34892dc7f", "domain": "shop.lyuaunew.zaibatsutel.net (punycode:shop.lyuaunew.zaibatsutel.net)", "original_domain": "shop.lyuaunew.zaibatsutel.net"}
Dec 13 14:22:12 zaibatsutel lets-proxy[29713]: 2020-12-13T14:22:12.727-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "e4ff37c0-c5aa-483f-8fe4-af9a33f76bf0", "domain": "test.de.zaibatsutel.net (punycode:test.de.zaibatsutel.net)", "original_domain": "test.de.zaibatsutel.net"}
Dec 13 14:22:12 zaibatsutel lets-proxy[29713]: 2020-12-13T14:22:12.728-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "185f12d2-0d32-41bb-be56-ae44126b46b5", "domain": "test.de.zaibatsutel.net (punycode:test.de.zaibatsutel.net)", "original_domain": "test.de.zaibatsutel.net"}
Dec 13 14:22:41 zaibatsutel lets-proxy[29713]: 2020-12-13T14:22:41.611-0800        info        cert_manager/manager.go:156        Get certificate        {"connection_id": "6d752590-b0ba-4dc9-8e64-ea671bf2f4a3", "domain": "magento.paperape.zaibatsutel.net (punycode:magento.paperape.zaibatsutel.net)", "original_domain": "magento.paperape.zaibatsutel.net"}

It would be nice to limit the specific names and patterns that will receive a certificate to avoid being ratelimited.

rekby commented 3 years ago

Hello.

You can do it by deny all domains in blacklist, then allow good domains in whitelist https://github.com/rekby/lets-proxy2/blob/f97f3a999bd2e1a13f148d59dcb1960ed70c5564/cmd/static/default-config.toml#L130-L137

brandymedia commented 1 year ago

Hello.

You can do it by deny all domains in blacklist, then allow good domains in whitelist

https://github.com/rekby/lets-proxy2/blob/f97f3a999bd2e1a13f148d59dcb1960ed70c5564/cmd/static/default-config.toml#L130-L137

Hey Timofey - can you please give us an example at how you use the Whitelist config?

For example, if I only wanted the following domains to be allowed:

domainone.com www.domainone.com domaintwo.com www.domaintwo.com

Do you just use an array for example:

WhiteList = ['domainone.com', 'www.domainone.com', 'domaintwo.com', 'www.domaintwo.com']

Or, is there another way to do this.

Thanks

rekby commented 1 year ago

Hello, @brandymedia.

WhiteList/BlackList is one string with one regexp. Regexp syntax described at https://github.com/google/re2/wiki/Syntax

for your scenation you can write one of:

WhiteList = "domainone\\.com|www.domainone\\.com|domaintwo\\.com|www.domaintwo\\.com"
WhiteList = "(www\\.)?(domainone\\.com|domaintwo\\.com)"
WhiteList = "(www\\.)?(domainone|domaintwo)\\.com"

Double backslash need because toml parser use \ as backslash too.

If you will use simple dot instead of "\." - dot will match with any symbol. It will work, but will not strong correct (regexp www.test.ru will allow domains wwwbtest.ru).

brandymedia commented 8 months ago

Hey Timofeym I have tried to add a whitelist as per your example but for some reason it still allows other subdomains.

So for example, I only want to accept www and the naked domain, but not anything.domainone.com for instance.

I need to prevent multiple subdomains accessing and thus creating certificates if that makes sense.

brandymedia commented 8 months ago

For instance, I am getting lots of ones like this:

sber.gitlab.r5fg6uqvkdxvod4.auth.domainone.com.ecdsa.cer sber.gitlab.r5fg6uqvkdxvod4.auth.domainone.com.ecdsa.key sber.gitlab.r5fg6uqvkdxvod4.auth.domainone.com.ecdsa.json

Which eventually makes the service stop.

rekby commented 8 months ago

Ow, it may by confused by blacklist/whitelist rules. Do you have blacklist rules?

If you want use whitelist only - you have to add BlackList=".*" (deny all).

I add more explain to config and tests with PR https://github.com/rekby/lets-proxy2/pull/213/files

Is it solve your question?

brandymedia commented 8 months ago

Thanks Timofey, that make sense - I appreciate your fast response and solution.

I'm currently testing this to see if I can prevent all the spammy certificates.

Is there a way to remove the current certificates for a domain, so I can test whether the blacklist/whitelist is working?

I tried removing the files in the storage folder, but no new ones are added from that domain when I visit the website?

Do I need to remove them from somewhere else?

Thanks for your help.

rekby commented 8 months ago

I tried removing the files in the storage folder, but no new ones are added from that domain when I visit the website?

remove storage folder is ok way. Did you restart the proxy?

Do I need to remove them from somewhere else?

No, the proxy doesn't store anything outside from storage folder.

brandymedia commented 8 months ago

Yes, I did restart the service but then no new files are created for the domain I had removed in the storage... but it still seems to load fine over https which is confusing...

I'm getting hit by tons of requests from what I can only imagine is a wildcard subdomain which is trying to create certificates.

Looking at the logs, it is now 'denying' those requests.

However, I have removed the files for certain domains which should be blocked, but they still serve over https.

brandymedia commented 8 months ago

Okay, it looks like removing the files from storage, restarting has now worked on one of the domain.

It is still however letting in files like:

mail.www.domainone.com.ecdsa.cer

I've set Blackilist to:

BlackList=".*"

And whitelist to:

WhiteList = "domainone\.com|www.domainone\.com

As I only want to allow the www and naked versions. Not any other random subdomain.

Thanks

rekby commented 8 months ago

try to use marks of start/end string, because regexp "domain.com" mean string with domain contains "domain.com". But can has some prefix and ending.

WhiteList = "^domainone\.com|www.domainone\.com$"

optional subdomain you can make by:

WhiteList = "^(www\.)?domainone\.com$"
brandymedia commented 8 months ago

Thanks again.

So in the second example

WhiteList = "^(www\.)?domainone\.com$"

Does that work with multiple domain names but restrict to only www and naked domain?

So for instance:

WhiteList = "^(www\.)?(domainone\.com|domaintwo\.com)$"

Is that correct syntax?

brandymedia commented 8 months ago

So for example, I only want this to work with:

domainone.com
www.domainone.com
domaintwo.com
www.domaintwo.com

etc

rekby commented 8 months ago

yes, it must work

brandymedia commented 8 months ago

I think I had an erro in my syntax... it should be:

^(www\\.)?(domainone\\.com|ama-uk\\.com|domainnametwo\\.com)$

Hopefully that will work now.

Out of interest, is there a way to actually prevent the subdomains requesting certificates at all? Currently they request and get denied.

Thanks again.

rekby commented 8 months ago

^(www\.)?(domainone\.com|ama-uk\.com|domainnametwo\.com)$

It contains double-escaped dots, try use one slash:

^(www\.)?(domainone\.com|ama-uk\.com|domainnametwo\.com)$

Out of interest, is there a way to actually prevent the subdomains requesting certificates at all? Currently they request and get denied.

What mean "prevent the subdomains requesting certificates"? Prevent lets-proxy to request certificates for subdomains? (it is managed by the rules) Or prevent clients to send request with subdomains?

brandymedia commented 8 months ago

It was the one slash that was throwing the error. It seemed to work with the 2 slashes. Sorry, reg ex is not my speciality 🙈

I'll test this now and let you know the outcome.

brandymedia commented 8 months ago

Unfortunately now I can not get a certificate for a domain on the www or naked. I removed all occurrences of that domain in the storage folder and now it won't add new ones for that domain.

Do you think this could be down to limits?

brandymedia commented 8 months ago

Ho Timofey,

Still working on this and I'm getting an error for a domain I am trying to get a certificate for.

"error": "order authorization error"

This was one of the domains that was issuing lots of certificates on subdomains - I removed them all from storage and now it won't create a new one for the www or naked version.

Do you know what this error is please?

rekby commented 8 months ago

@brandymedia, try update lets-proxy up to v0.29.3, I have added authorisation details to the error.

brandymedia commented 8 months ago

@rekby cool. Can you remind me of the process?

Is it a case of removing everything and the reuploading the new binary and reconfiguring everything?

Thanks again.

rekby commented 8 months ago

You should replace binary (and restart it) only.

brandymedia commented 6 months ago

Hey @rekby. Still struggling to get the whitelist to work.

I have set in the config file:

[CheckDomains]

BlackList=".*"
WhiteList = "^(www\.)?(domainone\.com|domaintwo\.com)$"

But it's still allowing other subdomains which means the server is getting spammed on subdomains creating too many requests which then brings the service down.

I only want to accept www or blank.

Thanks for your assistance.

brandymedia commented 6 months ago

As a follow up question, does lets-proxy run before or after accessing the vhosts file in Linux Ubuntu?

Trying to find a way to stop spurious subdomains from creating hundreds/thousands of certificates.

Wondering whether by removing the default vhost which catches all requests that don't have a specific vhost would help or not..?

rekby commented 6 months ago

The rules should work fine. Similar rule checked by test.

You can write actual config and example of domain, which must be denied by rules, but allowed by real. I will check it by the test.

If it isn't help to detect problem I will need debug log for issue certificate for one of bad subdomain.

brandymedia commented 6 months ago

Thanks Timofey,

Here is an example of the config file:

[CheckDomains]

BlackList=".*"

WhiteList = "^(www\\.)?(edgarbrooks\\.co\\.uk|annandalefinancial\\.co\\.uk)$"

When I look in the storage directory there are hundred if not thousands of certificates for subdomains on these domains I do not want.

Thanks again.

rekby commented 6 months ago

Can show write one of them for check by rules by hands, by test, debug on the example, etc?

Better one of latest domain. You can grep it from a log by string "Certificate issued." or command ls -lah --sort time -r | head in the storage folder.

brandymedia commented 6 months ago

I have checked the latest files using ls -lt | less in the storage folder.

The latest one on this server is - Mar 7 08:44 yandex.avito.pay.pay.r5fg6uqvkdxvod4.www.auth.ama-uk.com.ecdsa.cer

Although on some of the servers these are getting created much quicker.

What do I need to do to test this manually please?

brandymedia commented 6 months ago

For example on another of our servers we have many records created in the same minute:

-rw------- 1 lets-proxy lets-proxy 3420 May 15 12:07 blog.eforward4.axiomfinance.co.uk.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  161 May 15 12:07 blog.eforward4.axiomfinance.co.uk.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 blog.eforward4.axiomfinance.co.uk.ecdsa.key
-rw------- 1 lets-proxy lets-proxy 3514 May 15 12:07 ozon.ozon.cdek.od0uygcil2pwhit.vmail.imsmortgages.uk.com.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  207 May 15 12:07 ozon.ozon.cdek.od0uygcil2pwhit.vmail.imsmortgages.uk.com.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 ozon.ozon.cdek.od0uygcil2pwhit.vmail.imsmortgages.uk.com.ecdsa.key
-rw------- 1 lets-proxy lets-proxy 3501 May 15 12:07 gitlab.gitlab.git.git.git.grelay1.imsmortgages.uk.com.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  201 May 15 12:07 gitlab.gitlab.git.git.git.grelay1.imsmortgages.uk.com.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 gitlab.gitlab.git.git.git.grelay1.imsmortgages.uk.com.ecdsa.key
-rw------- 1 lets-proxy lets-proxy  219 May 15 12:07 git.mwgi5cibs0ekr8c.keygqkhiyj2mu9q.booking.axiomfinance.co.uk.ecdsa.json
-rw------- 1 lets-proxy lets-proxy 3542 May 15 12:07 git.mwgi5cibs0ekr8c.keygqkhiyj2mu9q.booking.axiomfinance.co.uk.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 git.mwgi5cibs0ekr8c.keygqkhiyj2mu9q.booking.axiomfinance.co.uk.ecdsa.key
-rw------- 1 lets-proxy lets-proxy 3757 May 15 12:07 www.www.www.yxfqoqp4ql1kntzn.axiomfinance.co.uk.rsa.cer
-rw------- 1 lets-proxy lets-proxy  189 May 15 12:07 www.www.www.yxfqoqp4ql1kntzn.axiomfinance.co.uk.rsa.json
-rw------- 1 lets-proxy lets-proxy 1679 May 15 12:07 www.www.www.yxfqoqp4ql1kntzn.axiomfinance.co.uk.rsa.key
-rw------- 1 lets-proxy lets-proxy 3542 May 15 12:07 punh3lv801obdjk.4g1i8mw2mne74d1.manage.smartestdecisions.co.uk.ecdsa.cer
-rw------- 1 lets-proxy lets-proxy  219 May 15 12:07 punh3lv801obdjk.4g1i8mw2mne74d1.manage.smartestdecisions.co.uk.ecdsa.json
-rw------- 1 lets-proxy lets-proxy  227 May 15 12:07 punh3lv801obdjk.4g1i8mw2mne74d1.manage.smartestdecisions.co.uk.ecdsa.key