search

w3c / sync-media-pub

Repository of the Synchronized Multimedia for Publications Community Group
http://w3c.github.io/sync-media-pub
Other
16 stars 4 forks source link

Support for location #48

Closed gbarreiro closed 3 years ago

gbarreiro commented 3 years ago

Hello! First of all, thank you for the effort you're doing to develop this project. At the University of Vigo research department I'm working at, we're very interested in the SyncMedia standard for a project we're working on. However, for this project, we would also need to be able to trigger some multimedia contents based on the user's location. Have you ever thought about this in the SyncMedia working group? Has there any work been done on that matter?

marisademeglio commented 3 years ago

Hi @gbarreiro , that sounds like an interesting project! No, the topic of location hasn't yet come up in SyncMedia until now - could you describe your use case a bit more?

gbarreiro commented 3 years ago

As you'll understand, I can't describe you our exact use case, but of course I can bring you some similar examples. For instance, imagine you're creating a multimedia guide for a zoo, so for each area of the zoo, you would have your SyncMedia defined parallel and sequential contents –for example, for the garden where live the koalas, you might have a melodic background music, together with an HTML telling you the characteristics of this animals and showing you some photos, together with an audio narration–. The idea would be to be able to have just a single SyncMedia file for the whole zoo, so the tracks definition and some common elements like the background music can be shared, while other seq/par groups would be triggered when entering into a geofence.

We thought that this feature could be added defining a geofence in the header, as you would define a track, and then adding a reference to the geofence in the corresponding seq/par group, like in the example below:

<head>
        <sync:track 
            sync:label="Page" 
            sync:defaultSrc=“zoo.html" 
            sync:defaultFor="text"
            sync:trackType="contentDocument">
            <param name="cssClass" value="highlight"/>
        </sync:track>
        …
    <sync:location
            sync:id=“koalasGarden”
            sync:geofence=“polygon”
            <sync:coordinates>
                <sync:geofenceEdge>
                    <sync:latitude>-121.886300</sync:latitude>
                    <sync:longitude>37.338169</sync:longitude>
                </sync:geofenceEdge>
                <sync:geofenceEdge>
                    <sync:latitude>-121.885258</sync:latitude>
                    <sync:longitude>37.338697</sync:longitude>
                </sync:geofenceEdge>
                <sync:geofenceEdge>
                    <sync:latitude>-121.885692</sync:latitude>
                    <sync:longitude>37.337338</sync:longitude>
                </sync:geofenceEdge>
                <sync:geofenceEdge>
                    <sync:latitude>-121.884630</sync:latitude>
                    <sync:longitude>37.337850</sync:longitude>
                </sync:geofenceEdge>
            </sync:coordinates>
       </sync:location>
</head>
<body>
        <par>
            <audio sync:track="background-music" repeat="indefinite"/>
            <seq location=“koalasGarden”>
                <par>
                    <text src="#sn-0"/>
                    <audio clipBegin="0.91" clipEnd="3.22"/>
                </par>
          </seq>
      </par>
</body>

Of course this would add some more complexity in the SyncMedia JavaScript library, since it would need to deal with the user location, but we think that the effort could be worth it, both for our project and for the community. Let me know what you think about it.

marisademeglio commented 3 years ago

Cool idea!

Do you think it would work to keep the geolocation switching in a separate file and either:

  1. Have a separate sync media files for each location (so: zebra.sync, elephant.sync, etc), where you would get some duplication of markup for common elements, but at least you could share actual assets (just one backgroundmusic.mp3 file would be needed)

or

  1. Have one sync file that starts at an offset (so: zoo.sync#zebra) and then also add something to the sync spec that would influence the playback behavior so that it would stop after a segment had finished, instead of playing through, e.g. here I added sync:end, which may or may not be a great name for such a mechanism:
<body>
  <seq id="zebra" sync:end="stop">
   <par>
     <audio.../>
     <text .../>
   </par>
   ...
  </seq>
  <seq id="snake" sync:end="stop">
   ...
  </seq>
</body>
gbarreiro commented 3 years ago

It would be easier to opt for the first option, but this would not allow to provide a smooth experience. With this approach, the SyncMedia spec could be kept as it is, and we would only need to create a client that based on the device location can load the corresponding .sync file. However, even if you shared the assets, as you suggested, I think it would not be possible to keep a background music during the whole tour, since in the moment you move from one venue to other, another SyncMedia file would be loaded, so the playback would be restarted.

With your second approach, I think it would be more feasible to deliver a smooth experience, but besides the sync:end tags you suggested, we would need to specify somewhere the geofence coordinates for each segment, like in the example I explained in my previous message. Did you like the way I proceeded, defining first the geofences in the header, and then using them as triggers for the segments, or can you come up with a better way to do it?

marisademeglio commented 3 years ago

That's very interesting, having the background music stay continuous while the focus moves around. This sounds like a really specialized playback scenario, where you are always starting at or before the start of the background music node. And your specialized client knows not to reposition the background music playback at any point (whereas my client application would, when starting in the middle of a sync media tree, calculate the offset into any nodes that came before and are still active, although that would break musical continuity).

As for the geofence data, to me they seem like navigation points, and the closest precedent I can think of would be the chapter navigation feature in DAISY or EPUB. In these cases, the definition of navigation points are external to the text/audio sync documents.

However, if you need to put geofence data into the sync media file rather than have it live externally, could you put it as metadata? As in:

<smil xmlns:zoo="https://example.com/zoo">
  <head>
    <metadata>
       <zoo:nav>
            <zoo:location zoo:href="#polarBears">
              <zoo:geofenceEdge lat="123" lon="45"/>
              <zoo:label>Polar Bears</zoo:label> 
            </zoo:location>
            ....
       </zoo:nav>
    </metadata>
  </head>
  <body>
     <seq id="ocelots">
       ...
     </seq>
     <seq id="polarBears">
     ...
     </seq>
     ...
  </body>
</smil>
gbarreiro commented 3 years ago

The ideal case would be to have the geofence data inside the SyncMedia, like in the example you've just shared. However, you can't define a geofence with a single pair of coordinates, since geofences are polygons, hence the need of, at least, three pairs of coordinates, to define the spatial area of the geofence, so whenever the client player detects the user enters there, the corresponding track is triggered.

marisademeglio commented 3 years ago

Ok, I see - well my intention was not to really define geofence elements but rather to see if it worked for your purpose to have it in the metadata section of the document - the reason being that we do not define anything that lives under <metadata> in the spec, but rather leave it open to elements from any namespace. So you can basically put whatever you want there.

This way, the sync media spec just needs to incorporate the playback behavior modification of sync:end="stop" (or whatever it ends up being called) rather than introduce an entirely new set of concepts related to location.

gbarreiro commented 3 years ago

Thank you very much for your suggestions. We'll try to implement a location-based behavior respecting the SyncMedia specification as much as possible, following your suggestions. As soon as we have something (expect some months), we'll contact you again to share our work on this matter. Thank you very much for your help!

marisademeglio commented 3 years ago

Sounds great, can't wait to see it! In the meantime, I've filed an issue about having some syntax to indicate that a segment should play and then the presentation should stop.