Serving Rich Metadata / Schema.org integration

[rochforp]

I was wondering if anything had ever been decided about allowing the pws to utilize schema.org defined content. It looks like pws does get json-ld files but it doesn't seem like anything is being done with them currently. I just wanted to open a conversation about improving the content discovery experience and possible ways to do that using the existing tools people already use on their sites (RDFa, Microdata, OpenGraph, Twitter Cards...). I know there might be some possible security concerns around the use of actions but it might be a good way to present existing data. There was some talk about it in #482 but I thought it should be revisited.

[scottjenson]

We started down that road and got a bit waylaid by the energy required to ship something ;-)

The json+ld or schema.org is still an interesting direction we'd like to explore, the real question is "What's the best way to use/expose this data?" The obvious discussion would be around ranking/grouping related URLs.

Right now, we're working towards a simple distance ranking approach (we do this on iOS, it's coming to Android soon) It's not perfect but it's reasonable in that the things closest to you are highest in the list.

As the number of nearby beacons grows, we're going to have to change the UI and one of the key ideas is to 'recommend' the top X beacons but below it have 'folders' or related beacons. We haven't really started doing much thinking on that yet but if you've got some ideas (or alternative suggestions, please do feel free to chime in)

BTW, an alternative approach is to have the URL not point to a web page but a JSON blob. This would allow the PW scanner, but more importantly other scanners, to be created for very specific/targeted use cases. This becomes even more interesting when WebBluetooth offers scanning, meaning that a 'classic HTML' beacon can show you an app, that then allows you to have a targeted, local experience using JSON/Schema.org based objects. Exciting stuff as it allows the experimentation to occur outside of the Physical Web (which makes us very happy as we don't want to be the bottleneck)

[kevinahuber]

I've noticed not a lot of websites/CMS' are using JSON- LD, but more are using various forms of open graph, schema.org, or various other tags. I like the idea of being able to filter based on schema.. so show only places/places that are open, or only coupons.

[Elwell]

A +1 on the wishlist from me for this. I'd really like to see the integration schema.org (as used elsewhere such as google now and actions in the inbox) so that physical web just becomes integrated into my now cards "hey you're at location X here's the info"

[scottjenson]

Let's discuss this a bit more. Now, you're thrilled to find just a few beacons, the actual number is small enough that a sorted list by distance is very effective for up a few dozen beacons. We very much want to explore ways of adding more information but we want to do it in a way that is very transparent and open so other scanners (and more importantly PWS's) can also support it as well. The key thing is to do it in a way that is optional so we don't break any scanner/PWS that is slower to adopt.

My suggestion would be to extend the PWS to return these additional meta tags. This would allow the client to then optionally offer a UX to sort/filter these items in different ways. For example, you could image a list with folders or a completely different one with a search field (defaulting to all)

The key thing is that we hammer out what the PWS does and what the client does as. This requires letting go of a little control so the PWS and client each have a clear roll but it's possible for both of them to not support it and the system doesn't fail.

[kevinahuber]

It seems that the secret sauce and driver of a unique PW experience would usually be the hierarchy of the data. The PWS would remain agnostic to the hierarchy, keeping core to its goal of being a proxy service that sends preview information for the URL to the client. The client can then mix and mash to its heart's content.

It seems that OpenGraph would be the next move after JSON-LD, with the highest adoption percentage: http://trends.builtwith.com/docinfo/Open-Graph-Protocol (35% of the top 100k sites)

For consistency, it could be an array similar to JSON-LD:

{
  "metadata": [
    {
      "json-ld": [
        {
          "url": "https://www.southwest.com",
          "@context": "http://schema.org",
          "@type": "Organization",
          "logo": "http://www.southwest.com/assets/images/globalnav/logos/swa_logo_dark.svg"
        }
      ],
      "open-graph": [
           "audio": "http://flightsoundtrack.com",
           "video": "http://flightvideo.com",
           "image": {
                 "url":"http://flightimage.com",
                 "type":"image/jpeg",
                 "width": "400"
           }
      ],
      "description": "Official Southwest Airlines website, the only place to find Southwest Airlines fares online. Book lowest airfare deals, view flight schedules, get flight status, and book rental cars and hotels.",
      "title": "Southwest Airlines | Book Flights, Airline Tickets, Airfare",
      "url": "https://www.southwest.com/",
      "rank": 1000,
      "displayUrl": "https://www.southwest.com/",
      "groupid": "c963260bea7bb36c",
      "id": "https://www.southwest.com/",
      "icon": "https://url-caster.appspot.com/favicon?url=https%3A%2F%2Fwww.southwest.com%2Fassets%2Fimages%2Ffavicon.ico"
    }
  ]
}

Likewise with oEmbed, twitter cards, microdata, etc.

[raggi]

I have been building some things with PW recently, and those things are focused on local home iot use cases. I am adding an increasingly wide arrange of devices that may or may not have pre-existing interfaces, by way of adding beacons.

In my use cases, I observe:

• I want it to be easy and cheap to introduce actionable notifications. I'm already using serviceworker, so the API being a mirror of serviceworker makes sense here.
• In cases where I'm deploying beacons that point at pre-existing sites (e.g. a router client summary page), I can't add new standards metadata to the sites. I'd prefer to rely on existing and common metadata formats in their regular locations and formats (e.g. manifest.json).
• When I'm building new things, I'd prefer to be able to test them out "as normal" in my existing browser tooling, which makes re-using the PWA toolchain a better match than introducing new manifests that will need new tools.

Finally, I'm not necessarily sure that "35% of the top 100k sites" is the right metric for PW use cases. I don't really want CNN.com as a PW as I walk around the streets of {cityName}. I don't think PWs have so much of the knowledge graph requirements. There's implicit knowledge (locality) and a requirement for ease of integration (simple actions forwarding), rather than search type problems.

My $0.02.

[kevinahuber]

@raggi Good points. I haven't heard of many home iot use cases for physical web- I'd love to see it in action. I disagree with not including existing search engine structured data- because there are already a lot of existing sites using it, and a great value of the physical web is that one does not need to build new content to access it. I really like some of the stuff in JSON-LD- actions and richer data. Since the PWS is just scraping the data rather than running the site, I think it is a good fit. I do like the idea of a service worker-ish API- perhaps for a Web Bluetooth scanning API.

@scottjenson What are your thoughts on this- and are there any intentions for the team to move forward with any of these in the near term?

[scottjenson]

I guess I just want to back up and ask "to what end?" Are we discussing sorting/grouping of found beacons? @raggi seems to want to encorporate actions into the list. These are two very different directions.

So far, we've been trying to keep this simple and small: a list of of web pages. We experimented early on with actions but felt like we were opening up a pandora's box as how you display those actions can vary so much.

Case in point: a museum experiene. What should be done in the PW and what should be done with the website itself? Now that WebBluetooth is starting to ship, it's reasonable to say that the museum web page should be the place to experiment with how you navigate/notify the user. We feel this is the right 'web way', e.g. let each page experiment and avoid forcing a single solution into the the PW list.

Besides, we strongly hope there are multiple PW scanners out there (not just Google) If we keep adding more and more for each scanner to do, we risk fragmenting the various scanners. By pushing as much out of the scanner and into each web page, we hope to enable multiple scanners.

[raggi]

I was pointed here from a discussion on chromedev Slack. It was suggested to me that this was one possible good way to handle actions. I don't mean to cause a conflation. For my use cases, I'd just like something simple, but I'm sure there are others.

[scottjenson]

So do you want to handle actions w/in the PW scanner? (nearby or Chrome or Opera?) Curious what type of actions you're talking about. If it is a simple set of commands a simple web page would work. of course, if you want to do something fancier, it wouldn't. I guess I'm trying to figure out what brand of fancy you're after.

[raggi]

So I am wrapping up things like Nest, Sonos, various Z-Wave devices, etc.

Concrete examples pop out:

Thermostat
[ +   /    - ]

Music
[ +  |>  ||  - ]

Gate
[ OPEN ]

What I'm doing with my wrappers is enabling people who are inside my property to make comfort changes without having to open up authenticated services to everyone. The proxies perform basic actions with specific privilege and prevent abuse by rates and other such limits, sometimes by pinging me.

The actions are all basic - just buttons to post to an url.

Right now, I can pop up pages that a user has to visit, then add notifications via a serviceworker with actions, but that seems inappropriate as it's hard to detect when they go away. Users don't have a good way to control this. Proximity is the correct use case for the visibility of these cards.

[scottjenson]

Very cool stuff! I'm not against your thinking here, but I am trying to figure out how a PW scanner would be part of this. For example, you'd like to only show controls to people w/in your property. That's reasonable. However, how would the PW scanner be able to enforce this?

Another approach would be to have a "Raggi-House" web page that the user would open up and then show the commands that are appropriate for your location w/in the house. This gives the user a single web page but the actual UI would vary. This puts all of the power in your hands and keeps the PW out of it.

[raggi]

That's not really what I want, and there are many reasons for this:

• I want users who attach to my wifi / approach my beacons to get the options regardless of what software they have installed.
• I do not want to publish urls to users, or have them type anything, ever.
• I do not want to aggregate all services into a single app - I deploy different apps in different ways, some stuff I build (most of what I have right now), but other stuff I just grab and deploy behind and ssl proxy (e.g. one of the sonos web UI's I lean on). I'd like to keep it this way, not having to write a ton more code. This is more viable in the global business of IOT too, though - there will never be a single controller that can do everything and there should be no need, if we make discovery automagic.

The PW scanner is not a core part of where I'm headed. I want Chrome to do this entirely natively in future, and I believe it's already doing that with eddystone beacons. As per the above, I also don't want users to have to install anything. They should arrive and be afforded certain conveniences immediately, with 0 install, including 0 install of a service worker.

[raggi]

You might have been asking about how a PW scanner would know whether you're inside or outside my property - the "security" aspect of this? The most basic privilege I have setup is that most write-able services won't allow POSTS from outside my lan, so, you'll need to have my fairly well known wifi password. Physical security related stuff most people can't write to, the web app auths you when you try, and if you're unknown then (eventually) I'll get pinged. PW doesn't need to handle any of my security stuff, what it needs to do is present the user with 0-install capabilities and discovery.

[scottjenson]

Most of your bullet points are exactly what the Physical Web is about so I'm confused. The whole point is that you can 'walk up and use anything' with the Physical Web. This conversation started up around JSON+LD (or schema.org (or both)) to augment the PW list.

My attempt on this thread is to figure out how you'd like to augment that list. However, it appears you'd rather not use the PW at all and do everything in Chrome, that's perfectly fine, I'm just not sure what we're talking about then.

As we're on the PW github, what do you want the PW to do?

[raggi]

I'm only talking about my use case. I was pointed here by Kevin as I was talking about actions in PW notifications on the chrome dev slack.

I do not understand your distinction between "we're PW on github" vs. Chrome. When I go to chrome docs on it's physical web stuff, it points me to this repo. As a user, they are not different.

[kevinahuber]

If I understand correctly, Scott is speaking about whether it is fully within a browser (in Chrome) vs in the touchpoint of a PW scanner(where metadata has been scrapped but the user has not visited the site yet). I think with your need for something that doesn't require a lot of building out is at the heart of the physical web. Schema.org action would work well for that-https://schema.org/Action

Or you could use something like phy.net's api(https://my.phy.net/api/docs) and change the url that your beacon is pointing to after action is taken. This solution would work now- the two urls could have "pause" and "play" icons and language to match it.

Bringing it back to the bigger picture- There is a lot of data that websites provide that the physical web is currently missing. I think if we stick with standards that augment the experience rather than define the experience, then it would not be hurt by fragmentation- like adding a photo from Schema or OpenGraph

[scottjenson]

@kevinahuber thank you, I'm just trying to understand what is being asked in this issue. If were were to expose more data in the PW list, what should it be? I'm just trying to nail down a specific ask. In general, we've found that most things that people ask for are fairly specific and hard to generalize across all users. This has lead us to the conclusion that we actually shouldn't do too much at the PW scanner level and get users to web pages, where they can get a tailored experience.

If, however, this is a truly generalizable case that should be explored, I'm all for it. I just haven't heard it articulated yet.

[kevinahuber]

@scottjenson I think a picture would be a fantastic add to the PW- Opera is already pulling a picture. Based on cliche sayings, a picture is worth a lot more words than we can fit in a title and description.

Looking at the makeup of a lot of sites (including those on the physical web)- OG image is the most prevalent, intentional image that is placed in a website. We could use JSON-LD as well, but a lot less pages have JSON-LD images. I think it should not pull just any image. Many pages have social media icons first, so that could be a bit dicey.

[scottjenson]

That's an interesting idea. I'll have to take a look at how opera does it. The trick is going to be getting the image to be placed in a way that is generically useful to as many folks as possible. Do you think Opera's approach is the most useful?

[itsMattShull]

The BeaconSage beacon browser pulls the favicon and then falls back to OG image.

[scottjenson]

Our production Physical Web Server already works quite hard at getting an image, starting with the favicon and then using the the web manifest and then (I believe) the Apple Home Screen Icon as well. This hasn't been documented yet but it gives a wide range of options to present a high rez image.

What I hear you saying is that adding the OG image would be a helpful addition?

[kevinahuber]

@scottjenson Yea, I think so. But I think we should differentiate the image from an icon- the images tend to be wider and much higher res so it could be displayed larger.

My bit on the PWS- If we work to make that more robust, and pull a lot of information, this will allow "clean fragmentation" to occur. So different people can build different clients, and mix and match which structured data they use. So one could favor using just an icon, and sub in an image at an icon size if available (like BeaconSage). Others can use action- and more specific ones can perhaps show hours or detailed information JSON-LD. SBut the underlying operations among PWS('s) will be consistent.

I also think since privacy is a core concern of the physical web, we should build out the PWS to scrape this information rather than rely on individual browsers to pull them to fill in the gaps (defeating the purpose of a proxy). The hefty process in the PWS is requesting the page, rather than the parsing action, so it should get as much as it can while it has it. I would think doing stuff like pulling all og information (like all of JSON-LD is currently done) would be beneficial, then we can let individual clients decide how they present it. Likewise, we could build this out to pick up itemprop, twitter cards, etc. I'd love to be a part of building this out, if we think this is a good path.

TL;DR: og:image is a huge add, but I think the largest add to the Physical Web experience as a whole would be to build out scraping as much structured data as we can, so that more robust clients can be created in compliance with the goals of the physical web without recreating what others have done before them and potentially introducing bugs that won't display all pages on the physical web.