search

google / physical-web

The Physical Web: walk up and use anything
http://physical-web.org
Apache License 2.0
6k stars 665 forks source link

Google PWS drops URL fragment #852

Open DKMudrechenko opened 7 years ago

DKMudrechenko commented 7 years ago

We have an angular2 application that uses Hash location strategy, so it needs an url fragment to work. The url is then shortened via goo.gl and applied to an eddystone-url beacon.

However when the beacon is detected, both iOS and Android cannot see the fragment, so for example we had www.foo.com/#/12345 and then shortened it. Both systems will only see www.foo.com/. If I use a Physical Web app on Android and switch Physical Web Service setting to "Production Open Source PWS" it displays url with the fragment, so it pointed to Google's PWS, but is it a bug or expected behavior?

This tool also drops url fragment if I try shortened url, but if I test www.foo.com/#/12345 without shortening - everything is ok.

I found this https://github.com/google/physical-web/pull/557/files, can it be related?

cco3 commented 7 years ago

If the two URLs have the same title and the same description and only differ in the fragment, they are considered part of the same group. However, if there is only one such beacon around, it should take you to the actual fragment. Are you able to share an example URL I'd be able to test?

DKMudrechenko commented 7 years ago

Here is the example of shortened url: https://goo.gl/4KwnR6 with the actual application, and we have only one eddystone url beacon active.

cco3 commented 7 years ago

Hm, this does seem like unexpected behavior. Rui, could you comment on whether or not this is intended?

Specifically, the fragment for https://fyi.reveldigital.com/#/fbe94043-a8d8-4b9c-9209-0e11f628bd60 gets dropped when it goes through this shortened URL: https://goo.gl/4KwnR6 (But it doesn't get dropped when sent to the PWS directly.)

DKMudrechenko commented 7 years ago

Any updates on this?

scottjenson commented 7 years ago

@mmocny @oahziur ?

oahziur commented 7 years ago

Just checked with @mmocny, this is not an intentional behavior, and we are working to resolve.

mmocny commented 7 years ago

Great bug! Did a bit of investigation here, and learned something: Redirects with fragments have weird history.

Luckily, modern browsers have ironed this out and have a pretty rational approach to fragments and redirectors including fragment portion of URL:

PWS was not doing the last part. We will add it.

I created a few tests to confirm what browsers do:

First, create short links with/without a fragment: https://goo.gl/vcvYc -> https://youtube.com https://goo.gl/yVlLVD -> https://youtube.com/#foo

Next, create short links to short links, with/without fragment to with/without fragment (e.g. all permutations): http://bit.ly/2fyCh6F -> https://goo.gl/vcvYc http://bit.ly/2fLsr4u -> https://goo.gl/vcvYc#bam http://bit.ly/2gdX68P -> https://goo.gl/yVlLVD http://bit.ly/2gdX6G8 -> https://goo.gl/yVlLVD#bar

Finally, open the above links in browser, but with/without appending yet another fragment: http://bit.ly/2fyCh6F -> https://www.youtube.com/ http://bit.ly/2fyCh6F#baz -> https://www.youtube.com/#baz http://bit.ly/2fLsr4u -> https://www.youtube.com/#bam http://bit.ly/2fLsr4u#baz -> https://www.youtube.com/#bam http://bit.ly/2gdX68P -> https://www.youtube.com/#foo http://bit.ly/2gdX68P#baz -> https://www.youtube.com/#foo http://bit.ly/2gdX6G8 -> https://www.youtube.com/#foo http://bit.ly/2gdX6G8#baz -> https://www.youtube.com/#foo

(It's a bit hard to follow, but notice the final landing page always has the last fragment within a redirection chain.)

Thanks!

DKMudrechenko commented 7 years ago

Hello, any updates on this issue? I still experience it on both iOS and andorid

DKMudrechenko commented 7 years ago

Hi, it has been a while since I initially posted the issue, however I still experience the problem when using Google PWS on Android (haven't tested on iOS this time, but I suspect it will also fail). Maybe someone could point out to me a spot in the source code where PWS handles fragments?

gfwilliams commented 7 years ago

I believe I'm facing this issue too.

I'm using Puck.js to measure the amount of times a door has been opened and to then report that as an Eddystone URL. The following URL gets transmitted, which I can verify using nRF Connect: https://goo.gl/D8sjLK#12

This redirects to https://www.espruino.com/dooropen.html#12 if you open it in a browser, but when viewed via the Physical Web link the bit after the # has been stripped off.

It's a shame as this would be a great way to transfer small amounts of information without an app.

beaufortfrancois commented 7 years ago

@mmocny Can you have a look at this?

gfwilliams commented 7 years ago

If you have a Puck.js or an nRF5xDK with Espruino, it should be as simple as uploading:

require("ble_eddystone").advertise("goo.gl/D8sjLK#12");
gfwilliams commented 7 years ago

Is there any news on this? I'm just about to send a book to the publishers where I'm trying to push data in a URL, and I was kind of hoping this might be fixed by now.

beaufortfrancois commented 7 years ago

@mmocny (gentle ping)

mmocny commented 7 years ago

Thanks for the ping, but apologies, no updates on this.

We've got an open bug for it, but work hasn't been picked up. Since this issue is actually a bit more severe than I first realized, I will increase priority.

gfwilliams commented 7 years ago

That's great, thanks!

gfwilliams commented 7 years ago

@mmocny Any news on this? I just tested and it's still broken (at least on stock Android).

What other options do I have to work around this? I'm not doing final formatting edits for the book and I don't really want to send it off knowing that the example doesn't work.

gfwilliams commented 7 years ago

@mmocny @beaufortfrancois having tried this with what seems like a completely legitimate URL, the Physical Web stuff on iOS/Android won't load https://bit.ly/2rjt74F#42 - but it works with third party Physical Web apps.

So:

Is there any kind of workaround available?

mmocny commented 7 years ago

Actually, we've made a few changes to address this issue which you can expect to see with our next server push (hopefully this week).

I'll post back here once it's out and we can see which issues still aren't addressed.

mmocny commented 7 years ago

As a preview, I just tested our staging version:

"scannedUrl": "https://bit.ly/2rjt74F#42",
"resolvedUrl": "https://www.espruino.com/dooropen.html#42",

So at least the example you posted above should be resolved soon!

mmocny commented 7 years ago

Could you explain what issues you are having with bit.ly?

We have not blocked anything, though there is another issue preventing us from sometimes resolving some popular redirectors, which we are also going to improve slightly with this next release.

gfwilliams commented 7 years ago

Fantastic, thanks! Beacons broadcasting the bit.ly address shown just aren't detected by phones at all... So I don't know if it's a server thing or possibly something to do with the app?

mmocny commented 7 years ago

If you could give me a sample URL I can dig in.

Looking at aggregate PWS state, 99.985% of bit.ly URLs have worked fine over the last 7 days, which is actually exceptionally good compared to the average redirector.

mmocny commented 7 years ago

..however 20% or so of new unique bit.ly urls we see every day don't appear to work, which is consistent with historical norms.

This may be specific to your URL for some reason, and I would be happy to dig further if you shared the URL.

ferencbrachmann commented 7 years ago

@mmocny can you give that stat for b3m.it ?

mmocny commented 7 years ago

Correction, bit.ly is actually only 98.5%. b3m.it is ~95.5%.

However, clarification about what this stat means: It's just the number of times PWS succeeded to give a useful result to the client. It is entirely unrelated to notifications, usage, or anything like that. It just shows that that particular domain is: (a) working well (b) cached well by PWS (c) not many mis-configured beacons

That stat will go down if you put up http:// domains, or use 404 pages to signal downtime -- so it's really not that interesting in isolation, unless the stat shows 0% success rate ;)

ferencbrachmann commented 7 years ago

Thanks! Understood. How does http have a negative effect here? I'd like to understand. We do have an on-off switch for the beacon pages in Beeem that negatively effects this stats for sure. We make the pages not show up in PW notifications by not rendering a few required tags.

gfwilliams commented 7 years ago

Interesting - great news that it's going through.

The URL is just the https://bit.ly/2rjt74F#42 one I posted above - does that work ok? In the browser it goes straight to https://www.espruino.com/dooropen.html#42, but the app itself doesn't seem to display it when it's from a beacon.

Actually is there a way to debug the Google Physical Web app for Android? It often says 'No beacons found' when there are valid beacons (just beacons that are HTTP) - having some feedback about why it's not displaying them might help to narrow this down much more easily.

edit: the steps I used were:

mmocny commented 7 years ago

The % metric I shared is the number of times PWS returns metadata to the client. If the URL cannot return metadata, e.g. HTTP and not HTTPs, etc, then this counts against the metric I shared.

So for b3m.it URLs, 95.5% of the time clients asked, PWS had metadata to give. Of course, many clients cache locally and don't always ask PWS for every URL. And clients like Nearby decide if it's worth firing a Notification only after-the-fact, so this % doesn't include falloffs like that.

This metric is really just a way to gauge health of external domains, and I would really not read much into small percentage differences per day.

mmocny commented 7 years ago

On my phone, when I broadcast https://bit.ly/2rjt74F#42, I can see the URL fine in Nearby app.

Of course, it does not come with the fragment URL in the production version, but as I mentioned, this will hopefully change by next week!

mmocny commented 7 years ago

..also, I would use Nearby, not the Physical Web app. That app is just an old publication of the test app from this github repo (I know, it's confusing, we should probably unpublish that app from play store).

If Nearby isn't working, there are some guides for debugging in issue #909.

gfwilliams commented 7 years ago

Thanks for checking on this.

I just tried again and the bit.ly URL is now working fine (as you say without the fragment). I can only think my phone/tablet was suffering from bad mobile reception when I tried it. Sorry for wasting your time with that.

mmocny commented 7 years ago

No problem at all! Thanks for bearing with us as we work through these issues, I'll let you know when you can try this out. Cheers.

gfwilliams commented 7 years ago

Can't wait! thanks for getting it sorted.