search

jackyzy823 / fxa-selfhosting

Selfhosting your own Firefox Accounts (FxA)!
Mozilla Public License 2.0
96 stars 13 forks source link

Sync server not starting/not staying running max retries exceeded SSL certificate varify failed #2

Closed FDrebin closed 4 years ago

FDrebin commented 4 years ago

Hello, I am trying to get myself self hosting more items and Firefox is one of the last things I want to do, I followed the instructions and made a self signed cert for *.mydomain.tld , put it into the config in my path of /opt/ssl/host.crt and /opt/ssl/host.key , all the services start but I notice the sync server fails after just a second of uptime and the error I receive is

SSLError: HTTPSConnectionPool(host='www.mydomain.tld', port=443): Max retries exceeded with url: /.well-known/fxa-client-configuration (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'),))

I have tried putting it behind my reverse proxy which has a valid SSL cert but then I get HTTPError: 502 Server Error: Bad Gateway for url: https://www.mydomain.tld/.well-known/fxa-client-configuration , but if I browse to https://myserverIP/.well-known/fxa-client-configuration I can see a file with data.

Going to https://mydomain/.well-known/fxa-client-configuration I get a bad gateway nginx which is pointing to the host itself.

Any advise?

FDrebin commented 4 years ago

Quick update, changed out the crt with cer just because, I can reach the https://www.mydomain.tld in Chrome (not firefox, firefox gives a HSTS error), and it loads firefox create account page and then I can access the ./well-known/fxa etc without issue....but the container still fails hard. I can curl it if I curl -k -O https://www.mydomain.tld/.well-known/fxa-client-configuration from the host itself and other hosts on my network.

I just can't seem to get it to function if I put it behind my reverse proxy which has valid LE SSL wildcard certs, but I also can't get it to function with my wildcard self signed cert due to the certificate verify failed....

jackyzy823 commented 4 years ago

I'm not very sure if it works. Just for your reference. you can change https://github.com/jackyzy823/fxa-selfhosting/blob/0bb65ae54efc2b37842359dc7498e96206996fe0/docker-compose.yml#L73 to

 - SYNCSERVER_IDENTITY_PROVIDER=http://fxa-content-server:3030

to make a internal http request insted of external https request (which need a valid cert)

or just comment out this line if you do not use fenix (firefox new android browser)

FDrebin commented 4 years ago

I will have to give it a go tomorrow! Appreciate it.

FDrebin commented 4 years ago

Ok awesome! That got me past the issue I was having, the only issue is now that I am at the Firefox page "Enter your email to continue to account settings" , when I enter an email it gives me Network error when attempting to fetch resources.

I looked through the fxa documentation and I didn't see a way to create a account via cli , any thoughts?

EDIT: Scratch that above, I was able to add the uri to the Firefox config, but I can't seem to get the browser to load an option for creating a profile or account using my uri.

jackyzy823 commented 4 years ago

Ok awesome! That got me past the issue I was having, the only issue is now that I am at the Firefox page "Enter your email to continue to account settings" , when I enter an email it gives me Network error when attempting to fetch resources.

Could you attach a screenshot of network request. ( Web Devtools -> Network or F12)

I looked through the fxa documentation and I didn't see a way to create a account via cli , any thoughts?

There's no cli way except edit db manually (not recommended).

EDIT: Scratch that above, I was able to add the uri to the Firefox config, but I can't seem to get the browser to load an option for creating a profile or account using my uri.

I'm not very sure if i understand what you mentioned. Do you mean after you edit fxa related settings in about:config and click Firefox Account button , the url still is accounts.firefox.com not your domain ?

FDrebin commented 4 years ago

Ok so looking at the about:config ; Some the one URL still have references to FireFox browser.newtabpage.activity-stream.fxaccounts.endpoint | https://accounts.firefox.com/

The rest appear to have been changed to my domain name, but when looking at my Firefox (I am on version 75 64bit) , and I only have a sign in to profile option, I don't see any options in my Firefox to create an account.

https://imgur.com/a/XKl3c02

Here is the network requests screenshot https://imgur.com/a/LPQAbSQ

EDIT : additional screenshot https://imgur.com/a/1wsVuxx . For whatever reason the api.mydomain shows 404, but if I ping it it does resolve and if I curl the data I get a lot of info so it appears to be working and it is properly defined according to the about:config in Firefox. I will have to tinker a bit more and see why it's being met with 404.

Going to api.mydomain.com it does load , but it's a blue circle. Looking at what file it is looking for it is looking for api.mydomain.com/v1/account/status shows not found, and then directs me to "Home" which directs to the blue circle. https://imgur.com/a/5b4WFOw

Looks like the network also has a few other errors :

`NetworkError when attempting to fetch resource. logger.js:69:29

error logger.js:69

captureError error-utils.js:63

logError base.js:781

k base.js:80

Underscore 2

_submitForm form.js:240

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.mydomain.com/v1/account/status. (Reason: CORS request did not succeed).`

Which references : https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSDidNotSucceed?utm_source=devtools&utm_medium=firefox-cors-errors&utm_campaign=default , only thing I can think of is that my api back end does have a self signed cert with my reverse proxy's Let's Encrypt in front....

jackyzy823 commented 4 years ago

Your client config seem right.

Can i have a look of detail info of OPTION api.domain.tld / status ,espically the response headers and body ( click on that request, and detailed info will display)

I think maybe the reverse proxy changed CORS headers?

jackyzy823 commented 4 years ago

I don't see any options in my Firefox to create an account.

Just try to log in , if the account do not exists , the site will guide you to register one.

FDrebin commented 4 years ago

I don't see any options in my Firefox to create an account.

Just try to log in , if the account do not exists , the site will guide you to register one.

So that was the first thing I did and it errors out "NetworkError when attempting to fetch resource."

Request URL:https://api.mydomain.come/v1/account/status Request Method:OPTIONS Remote Address:reverseproxyIP:443 Status Code: 404 Version:HTTP/2 Referrer Policy:origin

Response Headers (922 B)
Raw Headers content-encoding
gzip content-security-policy
connect-src 'self' https://api…t-src 'self'; style-src 'self' content-type
text/html; charset=utf-8 date Sun, 26 Apr 2020 14:44:00 GMT etag W/"481-2WCnnl9hGhqKNPDqYBSa+d1Axsw" server nginx strict-transport-security max-age=31536000; includeSubDomains x-content-type-options nosniff X-Firefox-Spdy h2 x-frame-options DENY x-robots-tag noindex,nofollow x-xss-protection 1; mode=block Request Headers (407 B) Raw Headers Accept / Accept-Encoding gzip, deflate, br Accept-Language en-US,en;q=0.5 Access-Control-Request-Headers content-type Access-Control-Request-Method POST Connection keep-alive Host api.mydomain.com Origin https://www.mydomain.com Referer https://www.mydomain.com User-Agent Mozilla/5.0 (Windows NT 6.3; W…) Gecko/20100101 Firefox/75.0

Is the raw headers for api.mydomain.com

jackyzy823 commented 4 years ago

I found that request missing headers : Access-Control-Request-Method and Access-Control-Request-Headers. I think that's the key problem ,but i cannot figure out why it's missing.

FDrebin commented 4 years ago

I found that request missing headers : Access-Control-Request-Method and Access-Control-Request-Headers. I think that's the key problem ,but i cannot figure out why it's missing.

That makes sense, would that be via reverse proxy or the nginx docker container?

jackyzy823 commented 4 years ago

perhaps no. it's a client side header.

Could you try this under a clean firefox env (no addons, default settings). Sorry for the inconvenience.

FDrebin commented 4 years ago

perhaps no. it's a client side header.

Could you try this under a clean firefox env (no addons, default settings). Sorry for the inconvenience.

It's all good! I apprecaite your time trying to track this down. This firefox is newly installed, only thing changed is the about:config to include my url. No extensions/addons/anything.

I did add add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; to my nginx revere proxy and now it doesn't return 404 , but still shows the network resource error.

Response headers are :

X-Firefox-Spdy h2
access-control-allow-origin *
access-control-expose-headers WWW-Authenticate,Server-Authorization,Timestamp,Accept-Language
cache-control no-cache
content-length 216
content-type application/json; charset=utf-8
date Sun, 26 Apr 2020 15:29:22 GMT
server nginx
strict-transport-security max-age=31536000; includeSubDomains
timestamp 1587914962
x-content-type-options nosniff
x-download-options noopen
x-frame-options DENY
x-xss-protection 1; mode=block

The JSON shows :

code 400
errno 108
error "Bad Request"
message "Missing parameter in request body: uid"
info "https://github.com/mozilla/fxa/blob/master/packages/fxa-auth-server/docs/api.md#response-format"
param "uid"
jackyzy823 commented 4 years ago

casue it make a wrong GET request instead of POST request

https://github.com/mozilla/fxa/blob/274ff00ba17ea80a7954a5fda9120e14aa77f15d/packages/fxa-auth-server/lib/routes/account.js#L893

and

https://github.com/mozilla/fxa/blob/0e6cf05f67e117cd16c9bce344f26374c9561d7a/packages/fxa-auth-server/lib/routes/account.js#L898

I'm total no idea about these.

Could you share your docker-compose.yml and nginx conf and all other stuffs may related. ( replace your domain to anything else) . I'll try to reproduct it tomorrow.

FDrebin commented 4 years ago

Ok so! Here is my reverse proxy config if it matters : https://pastebin.com/eX50nT3s . Here is the docker compose file modified to remove the sync servers https : https://pastebin.com/gcmFPuFd , I also had to modify it from version 3.7 in your file to version 3.3 as docker-compose complained and failed to build it.

All DNS entries point to the reverse proxy, so www , profile, oauth, api, and token (i think that's all of them) all point to my reverse proxy IP address.

My only other thought that would be rather annoying is take out the nginx proxy that's in front of all the containers, expose the containers to 0.0.0.0 and then just have my reverse proxy in front which of course causes problems with exposure....

jackyzy823 commented 4 years ago

It's so weird that your config not working. here's my proxy version based on your config. It works as usual.

  1. Nginx Frontend proxy as fxa-proxy.conf certs are Letencrypt valid certs.

    
server {

listen       80;
listen       [::]:80;
server_name  www.fxa.test.local profile.fxa.test.local token.fxa.test.local api.fxa.test.local oauth.fxa.test.local;
return 301 https://$host$request_uri;

error_page 404 /404.html;
    location = /40x.html {
}

error_page 500 502 503 504 /50x.html;
    location = /50x.html {
}
}

server { listen 443 ssl http2; server_name www.fxa.test.local;

ssl_certificate /home/ubuntu/cert/wild.fxa.test.local.cer ;

ssl_dhparam /etc/letsencrypt/dhparams.pem;

ssl_certificate_key /home/ubuntu/cert/wild.fxa.test.local.key; # managed by Certbot

ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ecdh_curve secp384r1; ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL; ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

ssl_trusted_certificate /etc/letsencrypt/live/fxa.test.local/fullchain.pem;

client_header_buffer_size 256k; large_client_header_buffers 8 1024k;

add_header Strict-Transport-Security max-age=15768000 always;

location / { proxy_pass https://172.26.5.141/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Protocol $scheme; proxy_set_header X-Forwarded-Host $http_host; } }

server { listen 443 ssl http2; server_name profile.fxa.test.local;

ssl_certificate /home/ubuntu/cert/wild.fxa.test.local.cer;

ssl_dhparam /etc/letsencrypt/dhparams.pem;

ssl_certificate_key /home/ubuntu/cert/wild.fxa.test.local.key;

ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ecdh_curve secp384r1; ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL; ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

ssl_trusted_certificate /etc/letsencrypt/live/fxa.test.local/fullchain.pem;

client_header_buffer_size 256k; large_client_header_buffers 8 1024k;

add_header Strict-Transport-Security max-age=15768000 always;

location / { proxy_pass https://172.26.5.141/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Protocol $scheme; proxy_set_header X-Forwarded-Host $http_host; } }

server { listen 443 ssl http2; server_name token.fxa.test.local;

ssl_certificate /home/ubuntu/cert/wild.fxa.test.local.cer;

ssl_dhparam /etc/letsencrypt/dhparams.pem;

ssl_certificate_key /home/ubuntu/cert/wild.fxa.test.local.key;

ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ecdh_curve secp384r1; ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL; ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

ssl_trusted_certificate /etc/letsencrypt/live/fxa.test.local/fullchain.pem;

client_header_buffer_size 256k; large_client_header_buffers 8 1024k;

add_header Strict-Transport-Security max-age=15768000 always;

location / { proxy_pass https://172.26.5.141/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Protocol $scheme; proxy_set_header X-Forwarded-Host $http_host; } }

server { listen 443 ssl http2; server_name api.fxa.test.local;

ssl_certificate /home/ubuntu/cert/wild.fxa.test.local.cer;

ssl_dhparam /etc/letsencrypt/dhparams.pem;

ssl_certificate_key /home/ubuntu/cert/wild.fxa.test.local.key;

ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ecdh_curve secp384r1; ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL; ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

ssl_trusted_certificate /etc/letsencrypt/live/fxa.test.local/fullchain.pem;

client_header_buffer_size 256k; large_client_header_buffers 8 1024k;

add_header Strict-Transport-Security max-age=15768000 always;

location / { proxy_pass https://172.26.5.141/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Protocol $scheme; proxy_set_header X-Forwarded-Host $http_host; } }

server { listen 443 ssl http2; server_name oauth.fxa.test.local;

ssl_certificate /home/ubuntu/cert/wild.fxa.test.local.cer; # managed by Certbot

ssl_dhparam /etc/letsencrypt/dhparams.pem;

ssl_certificate_key /home/ubuntu/cert/wild.fxa.test.local.key;

ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ecdh_curve secp384r1; ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL; ssl_session_timeout 1d;

ssl_stapling on;

ssl_stapling_verify on;

ssl_trusted_certificate /etc/letsencrypt/live/fxa.test.local/fullchain.pem;

client_header_buffer_size 256k; large_client_header_buffers 8 1024k;

add_header Strict-Transport-Security max-age=15768000 always;

location / { proxy_pass https://172.26.5.141/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Protocol $scheme; proxy_set_header X-Forwarded-Host $http_host; } }


2. Backend (172.26.5.141)
a clean clone of this repo with nothing changed (even synserver related things)

make self signed cert of `*.fxa.test.local`  as ./_wildcard.fxa.test.local.pem and ./_wildcard.fxa.test.local-key.pem

create a `.env` from `.env.sample`

MUST

DOMAIN_NAME=fxa.test.local

make sure you create $PERSISTENCEPATH/public and $PERSISTENCEPATH/mysql_data

and chmod a+w $PERSISTENCEPATH/public

if PERSISTENCEPATH is relative, it relate with docker-compose.yml

PERSISTENCEPATH=.

if cert and key is relative , please prepend ./ to

for profile.DOMAIN_NAME

PROFILE_CERT=./_wildcard.fxa.test.local.pem PROFILE_CERTKEY=./_wildcard.fxa.test.local-key.pem

for api.DOMAIN_NAME and oauth.DOMAIN_NAME

AUTH_CERT=./_wildcard.fxa.test.local.pem AUTH_CERTKEY=./_wildcard.fxa.test.local-key.pem

for token.DOMAIN_NAME

TOKEN_CERT=./_wildcard.fxa.test.local.pem TOKEN_CERTKEY=./_wildcard.fxa.test.local-key.pem

for www.DOMAIN_NAME

CONTENT_CERT=./_wildcard.fxa.test.local.pem CONTENT_CERTKEY=./_wildcard.fxa.test.local-key.pem



I dont think your config has anything wrong. but it's so weird. :-(
FDrebin commented 4 years ago

Mhmmm, and I assume your reverse proxy and docker host are separate devices? I might attempt to a fresh install just because I can't break anything since it isn't being fully utilized as of yet.

I might just do that, rebuild and see what happens.

jackyzy823 commented 4 years ago

Mhmmm, and I assume your reverse proxy and docker host are separate devices?

Yes

For one device with proxy and fxa just change https://github.com/jackyzy823/fxa-selfhosting/blob/0bb65ae54efc2b37842359dc7498e96206996fe0/docker-compose.yml#L397

to (in your case)

- "10.0.0.135:<notusedport>:443"

and change all proxy_pass in proxy conf to proxy_pass https://10.0.0.135:<notusedport>/;

FDrebin commented 4 years ago

Perfect. I am rebuilding the VM now and I am going to spin it back up with standard config, leave all my reverse proxy and DNS entries alone and see what happens. Hoping for success :-)

FDrebin commented 4 years ago

Well, stood it up with the reverse proxy in place, everything started didn't have to remove the https from sync server, but still met with a network error on creating an account so very annoying. I will have to see what else I can do.