search

mozilla-mobile / firefox-ios

Firefox for iOS
Mozilla Public License 2.0
12.22k stars 2.93k forks source link

Add ability to set custom identity.sync.tokenserver.uri for self-hosted Sync #5006

Closed SimonBasca closed 5 years ago

SimonBasca commented 5 years ago

User Agent: Mozilla/5.0 (X11; CrOS x86_64 11316.165.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.122 Safari/537.36

Related issue: https://github.com/mozilla-mobile/firefox-ios/issues/3150

Steps to reproduce:

There is currently no way to specify a custom URL for a Firefox Sync Server (token url) while using the publicly available, Mozilla-hosted Firefox Accounts service.

Actual results:

There is no option to do this.

Expected results:

There should be an option like on Android to specify to custom token URL for self-hosted sync server. This option is available on desktop Firefox and on Firefox for Android.

SimonBasca commented 5 years ago

Jerry Heiselman: I have attempted to create a fxa-client-configuration file located at https://thor.heiselman.com/.well-known/fxa-client-configuration that mirrors the one served by accounts.firefox.com with the one change being the URL for the sync tokenserver url to point to my own.

When configuring this URL in Firefox on iOS using the method of revealing the hidden Advance Account Settings menu by tapping repeatedly on the Version string in the Settings. I filled in the base URL for my server (https://thor.heiselman.com) and enable the "Use Custom Account Service" option. Once I enable the option, I can see the client request the fxa-client-configuration file from my server successfully, however, it then attempts to load the login page on my server despite the configuration pointing all other services back to the public Firefox service.

m.m.naseri@gmail.com: Can we bump the priority of this bug from a P3? Many people seem to be holding off moving to the FF ecosystem simply because they can’t carry their privately stored data without the additional cost of setting up an account server.

This clearly breaks the user experience and I’d have imagined that after a couple years of back and forth on Github this would’ve received more attention.

mmnaseri commented 5 years ago

Also mentioned in #3150 (which was closed in favor of the bug in the bugzilla system).

jheiselman commented 5 years ago

I was the submitter of the Bugzilla report. I am still willing to work with someone here to help resolve this.

jellium commented 5 years ago

This option (choose what Firefox Sync server to use) should definitely be available on iOS (as it appears to be available on Android). Looking forward to being able to set it up.

mwegner commented 5 years ago

I recently made the switch to Firefox as my default browser (I think a lot of developer types have it installed, but drift back to their old setups for various comfort/muscle memory reasons). I would absolutely love to have the iOS version support a custom sync server.

There are quite a few "homelab" types that run significant infrastructure at home (i.e. VM hosts), and running something like a custom sync server doesn't add any extra overhead to their tech setup. I would absolutely run it, but it doesn't make any sense unless all of my Firefox devices can also use it.

garvankeeley commented 5 years ago

When implemented, ensure URLs are https or localhost. This is a sec requirement from app services.

jheiselman commented 5 years ago

@garvankeeley if you look at my comment (copied from the original bug), all URLs are HTTPS with valid LE certs.

garvankeeley commented 5 years ago

On desktop this is set in about:config using identity.fxaccounts.autoconfig.uri Docs: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

jheiselman commented 5 years ago

@garvankeeley this issue is specifically for Firefox on iOS which has no about:config

nook24 commented 5 years ago

Is there a schedule available for this feature?

Many people seem to be holding off moving to the FF ecosystem simply because they can’t carry their privately stored data without the additional cost of setting up an account server.

This is exactly the reason why I don't use FF sync on any of my devices. Just because a self-hosted sync server it's not implemented on iOS :(

jheiselman commented 5 years ago

Same here @nook24. I already have the sync server setup for my Linux desktop. But I don’t use FF on my iPhone because of this. I would in a heartbeat though.

garvankeeley commented 5 years ago

I have confirmed this works as-intended (by the Sync team), but the intention is that the entire stack is being hosted on the custom URL where the config file is.

garvankeeley commented 5 years ago

Closing this bug as works as-intended, the docs here still apply as to how to use your own fxa and sync stack: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

jheiselman commented 5 years ago

I’m very disappointed that we (the users) simply aren’t being heard. We know it’s working as intended. We want how it works to change. I feel like plenty of people have laid out perfectly valid reasons for the behavior to change. And the lack of willingness to even acknowledge that this leaves iOS at a distinct disadvantage shows a poor attitude towards the Firefox community as a whole.

I stand by my offer to help test any changes if anyone is willing to actually listen and attempt to implement this feature request.

justindarc commented 5 years ago

You are being heard. In fact, we were willing to spend a few hours today investigating this before concluding that this is not solely an iOS issue, but an FxA issue for something that is currently not supported. With limited resources, we have to prioritize issues and feature requests and the overall number of users who want this feature is almost immeasurably small.

justindarc commented 5 years ago

You are also welcome to run your own full FxA/Sync stack and you can follow the steps outlined here to configure iOS to work with it: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-fxa.html

jheiselman commented 5 years ago

I will be available to have a discussion around 2:00 pm US/Central and for most of the day after.

As for running a full stack server, it has been stated by several users that there are specific reasons for some of us to not want to do that.

garvankeeley commented 5 years ago

This is a discussion to have with the FxA/Sync team, they implemented this feature in the product with the intention that it be consistent with Android behaviour. If more users make this request across the various products then I could see that changing their opinion.

nook24 commented 5 years ago

Thanks for your time and investigation effort. I really appreciate this.

From the docs :

Note By default, a server set up using this guide will defer authentication to the Mozilla-hosted accounts server at https://accounts.firefox.com. You can safely use the Mozilla-hosted Firefox Accounts server in combination with a self-hosted sync storage server. The authentication and encryption protocols are designed so that the account server does not know the user’s plaintext password, and therefore cannot access their stored sync data.

Alternatively, you can also Run your own Firefox Accounts Server to control all aspects of the system. The process for doing so is currently very experimental and not well documented.

I would say there should be a big fat warning, that the docs will not work for iOS devices.

I had given the /.well-known/fxa-client-configuration trick a shot but this didn't work. (As already expected).

This is a discussion to have with the FxA/Sync team, they implemented this feature in the product with the intention that it be consistent with Android behaviour.

I would really like to know why the option is not available on iOS. Is this an restriction from Apple or so? I mean, it's available on Desktop and Android. Again from the docs:

Since Firefox 33, Firefox for Android has supported custom sync servers. To configure Android Firefox 44 and later to talk to your new Sync server, just set the “identity.sync.tokenserver.uri” exactly as above before signing in to Firefox Accounts and Sync on your Android device.

Are there just not enough iOS based Firefox users?

Please don't get me wrong. I don't want to blame anyone why this isn't implemented already. I'm just wondering why there is a different behavior.

garvankeeley commented 5 years ago

I would say there should be a big fat warning, that the docs will not work for iOS devices. I had given the /.well-known/fxa-client-configuration trick a shot but this didn't work. (As already expected).

Can you indicate what part doesn't work specifically on iOS? If so, I can report this to the FxA/Sync team to investigate further. The bug report here indicates that the content server is not on the same host as the /.well-known/fxa-client-configuration, which according to the server team, is not supported (which is correct behaviour for all platforms).

I would really like to know why the option is not available on iOS. Is this an restriction from Apple or so?

Firefox iOS should behave like Desktop and Android, if I can show that it isn't behaving consistently, I can get traction on getting something fixed.

jheiselman commented 5 years ago

Firefox on iOS doesn't have the ability to use about:config to configure the syncserver (token server) URL. Therefore, it already doesn't have feature parity with Android and desktop versions. This means that iOS users cannot follow the same setup procedure as Android and desktop users.

iOS users only have the option to "Use Custom Account Service". This asks for a single URL at which it will retrieve the /.well-known/fxa-client-configuration. The structure of this file is a listing of the different components with URLs for each piece. The documentation indicates that one can set each value independently.

While Firefox on iOS does query and retrieve the fxa-client-configuration file, it doesn't obey it as far as the different URLs are listed. As stated earlier in this issue, I downloaded the file from the public Firefox Accounts service and changed only the URL for the token server and hosted that changed file on my own server (same one hosting the syncserver). The file contents are as follows:

{"auth_server_base_url":"https://api.accounts.firefox.com","oauth_server_base_url":"https://oauth.accounts.firefox.com","pairing_server_base_uri":"wss://channelserver.services.mozilla.com","profile_server_base_url":"https://profile.accounts.firefox.com","sync_tokenserver_base_url":"https://thor.heiselman.com/sync/token/1.0/sync/1.5"}

Firefox on iOS doesn't seem to use any of these other values and instead attend to authorize against the hosting server instead of any of the others listed here.

nook24 commented 5 years ago
  1. I installed my Sync Server like described in: https://moz-services-docs.readthedocs.io/en/latest/howtos/run-sync-1.5.html
  2. I created an Firefox Sync Account at https://accounts.firefox.com
  3. I created: https://sync.nook24.eu/.well-known/fxa-client-configuration
  4. On my Desktop Firefox (Mac and Windows) i set identity.sync.tokenserver.uri to https://sync.nook24.eu/token/1.0/sync/1.5 using about:config
  5. Desktop Sync works.

iOS Setup:

  1. Advance Acount Settings
  2. Set Custom Account Url to https://sync.nook24.eu and enable "Use custom Account Service"
  3. Check access.log: 
    xxx.xxx.xxx.xxx - - [13/Jun/2019:21:26:39 +0200] "GET /.well-known/fxa-client-configuration HTTP/1.1" 304 5373 "-" "Client/15323 CFNetwork/976 Darwin/18.2.0"
  4. Login to Sync ("Bei Sync anmelden") Firefox tries to load the login page from my server: 
    xxx.xxx.xxx.xxx - - [13/Jun/2019:21:27:37 +0200] "GET /signin?service=sync&context=fx_ios_v1 HTTP/1.1" 404 5542 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/17.3b15323 Mobile/16D57 Safari/605.1.15"
  5. Firefox error: Could not load page ("Seite konnte nicht geladen werden.")
garvankeeley commented 5 years ago

@vbudhram do you mind looking at the last comment, the user was able to avoid hosting the content server when setting identity.sync.tokenserver.uri on Desktop, but this appears unavoidable on iOS. Should iOS not behave the same?

jheiselman commented 5 years ago

These are the exact instructions I followed as well.

justindarc commented 5 years ago

Ok, after talking with some more FxA folks, it seems that you are correct and this should be supportable as long as we add a new pref on iOS explicitly for identity.sync.tokenserver.uri. I'll update the title to add clarity here so that this work can get scheduled.

nook24 commented 5 years ago

Sounds good! Looking forward to get this. Many thanks to all of you.

jheiselman commented 5 years ago

Thank you justindarc. This looks like it will work great for us!

justindarc commented 5 years ago

No problem. Sorry for the confusion.

justindarc commented 5 years ago

WIP pull request: https://github.com/mozilla-mobile/firefox-ios/pull/5158

fireglow commented 5 years ago

I got the update on iOS today, thanks to all involved for making this happen! Is it working for everybody as expected? All I'm seeing is a GET /mozilla/token/1.0/sync/1.5 HTTP/2.0" 200 893.
Nothing for /storage/, no actual syncing.

Edit: it's 100% working on lighttpd! Thanks again to everybody!

s-fiebig commented 5 years ago

Installed the iOS update an hour ago and everything is working as expected. Set the token server, restarted Firefox and logged in to my Firefox account. Syncing is also working in both directions. @fireglow My sync server is running under http (at least for now), maybe this makes a difference.

Thanks to all involved for making this happen!

nook24 commented 5 years ago

Works like expected for me, many thanks to all.

This is my setup: https://daniel-ziegler.com/computer/netzwerk/linux/2019/07/07/Firefox-Sync-iOS/

Balooforever commented 5 years ago

Doesn't work for me :( I use the docker ffsync image and a reverse-proxy. I add my server : https://mydomain/token/1.0/sync/1.5

jheiselman commented 5 years ago

I finally got the chance to test this out and am running into a problem.

I'm running the syncserver from Docker Hub (same as I always have). Firefox on my iPhone is able to set the token server, I'm able to login, and I see the request make it to the sync server. The device is added to my (Mozilla hosted) Firefox Account and my desktop browser pops up a notification saying that the iPhone was added to my account. I am able to send pages back and forth between my two browsers. However, my bookmarks, history, open tabs, and logins/passwords are not syncing.

All of the requests to my sync server are resulting in HTTP 200 status codes and I was using this sync server with my Android phone, so I think everything is setup properly there, but I'd be happy to do some testing/logging if someone points me in a direction.

Balooforever commented 5 years ago

Same for me @jheiselman , thank for your post, i'm not english : it's hard for me to explain ! I use Docker Hub too and it's doesn't work.

captn3m0 commented 5 years ago

Same issue here. Logs give 200:

10.8.0.1 - - [14/Jul/2019:20:59:32 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22306 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 4ms
10.8.0.1 - - [14/Jul/2019:20:59:45 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22325 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:00 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22339 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:03 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22340 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:37 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22360 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms
10.8.0.1 - - [14/Jul/2019:21:00:40 +0000] "GET /token/1.0/sync/1.5 HTTP/2.0" 200 513 "-" "Firefox-iOS-FxA/18.0b15690 (iPhone; iPhone OS 12.3.1) (Firefox)" 22364 "Host-firesync-bb8-fun-0" "http://172.22.0.22:5000" 2ms

"Send tab to device" feature works sometimes (with a lot of lag and Firefox restarts). Bookmarks and other content sync doesn't work at all.

Running the latest mozilla/syncserver image (3f9bc839727f)

BirgerK commented 5 years ago

@jheiselmann:

Same problem for me.

All other devices (PC, Android) are able to sync. I also see my iOS device on my Firefox devices list.

But the iOS device cannot sync any bookmark or "synced tabs". I can't even send a tab from another device to the iPad, because the iPad doesn't appear on the shown list. (Again: my iPad appears on the devices list on my profile!)

fireglow commented 5 years ago

Question for those that cannot get sync going: Are you using nginx?

I tried and failed with nginx, had to set up a lighttpd on a dedicated port and switch every device, iOS or not, over to that port. Only then sync worked in iOS.

BirgerK commented 5 years ago

Yes, nginx. But no chance to change the proxy! Is it missing some headers?

All other devices (PC, Android, Mac) are able to sync without any problems.

jheiselman commented 5 years ago

@fireglow I am not using nginx. I am using Caddy HTTP server.

Further, for those of us who can’t get it to work, it seems to be the same in that each of us can use it from a PC and Android. It’s only iOS based devices that don’t work. So, I don’t think there is something wrong in the web server setup.

fireglow commented 5 years ago

It has to be something, it'll be interesting to know what server people are having success with, and which servers are failing.

I'm having success with lighttpd with

$SERVER["socket"] == "10.0.112.1:8888" {
  ssl.engine  = "enable" 
  ssl.pemfile = "/usr/local/etc/ssl/acme/foo.bar/cert.pem" 
  ssl.privkey = "/usr/local/etc/ssl/acme/foo.bar/key.pem" 
  $HTTP["host"] == "sync.foo.bar" {
    ssl.pemfile = "/usr/local/etc/ssl/acme/sync.foo.bar/cert.pem" 
    ssl.privkey = "/usr/local/etc/ssl/acme/sync.foo.bar/key.pem" 
    proxy.server = ( "" =>
                   ( ( 
                       "host" => "jail_mozillasync",
                       "port" => 5000
                     ) )
                  )
    proxy.forwarded = ( "for"   => 1,
                        "proto" => 1,
                        "host"  => 1,
                      )
  }
}
jheiselman commented 5 years ago

Hmm. Well, I can try swapping out the web layer this evening to see if that makes a difference for me. If it works, then I can see about putting wireshark on it to check out the headers.

captn3m0 commented 5 years ago

I'm running Traefik in front of the syncserver to terminate TLS, it does forward all 3 X-Forwarded headers.

Config: https://github.com/captn3m0/nebula/blob/master/firefox-sync.tf#L18-L24

jheiselman commented 5 years ago

@fireglow I finally got around to trying lighttpd and it still isn't working for me. However, I did see that you had your sync server setup as the root application at a dedicated domain. I had mine mounted under a /sync context root. So I changed it up and created a dedicated DNS name and mapped it the same way that you had (full proxy from / to the sync server) and it still hasn't worked.

Symptoms as described before:

Mardiie commented 5 years ago

It has to be something, it'll be interesting to know what server people are having success with, and which servers are failing.

I'm having success with lighttpd with

$SERVER["socket"] == "10.0.112.1:8888" {
  ssl.engine  = "enable" 
  ssl.pemfile = "/usr/local/etc/ssl/acme/foo.bar/cert.pem" 
  ssl.privkey = "/usr/local/etc/ssl/acme/foo.bar/key.pem" 
  $HTTP["host"] == "sync.foo.bar" {
    ssl.pemfile = "/usr/local/etc/ssl/acme/sync.foo.bar/cert.pem" 
    ssl.privkey = "/usr/local/etc/ssl/acme/sync.foo.bar/key.pem" 
    proxy.server = ( "" =>
                   ( ( 
                       "host" => "jail_mozillasync",
                       "port" => 5000
                     ) )
                  )
    proxy.forwarded = ( "for"   => 1,
                        "proto" => 1,
                        "host"  => 1,
                      )
  }
}

are you by any chance using the account service as well? it seems the combo works if you have to believe Mozilla's own guide. https://mozilla-services.readthedocs.io/en/latest/howtos/run-fxa.html

jheiselman commented 5 years ago

@Mardiie I am not running the account server. My setup is using the Mozilla-hosted FxA while using a self-hosted sync server.

strayer commented 5 years ago

For the record, I’m having the same issue. Login works but nothing syncs. I’m using Traefik in front of syncserver too with let’s encrypt.

dstapp commented 5 years ago

Same problem here. Is it about some configuration of the sync server (e.g. that we have to use FORCE_WSGI_ENVIRON and the seen url does not match the configured application url? Or is it some missing header on the reverse proxy? I use Caddy as a reverse proxy, use LE certificates and forward all the necessary X-Forwarded- headers.

However I have the same problems, send tab to device works but sync is not started, no request to the /storage endpoints are being made. From my Mac it works perfectly.

dstapp commented 5 years ago

Figured it out! HTTP/2.0 seems to be the problem. After forcing http 1.1 in caddy using:

tls { alpn http/1.1 }

everything works just fine!

I assume this is also the reason why it does not work with nginx as a reverse proxy while it does with lighttpd (where HTTP/2.0 is not turned on). Hope this helps.

Findus23 commented 5 years ago

@dprandzioch That would make a lot of sense. Unfortunatly I can't test this as in nginx one can't disable http2 per vhost.

But that would mean that something in firefox-ios is broken to not accept http2, but also don't do a http1.1 request.