Add automation support using WebDriver

[rakuco]

This was discussed during TPAC 2023. WebDriver support in the Generic Sensor API spec was fully rewritten a few months ago, and the tests in WPT are not implementation-specific anymore.

It would be good to do the same with this spec, as several existing web tests in WPT rely on Chromium Mojo JS mocks and are therefore not easily interoperable.

One idea would be to make this specification reference the automation bits in the Generic Sensor API one (in other words, implementations are not expected to implement the Generic Sensor API itself, only the WebDriver extension commands defined in https://w3c.github.io/sensors/#automation). This would make it possible to set e.g. DeviceMotionEvent data by calling test_driver.create_virtual_sensor("accelerometer") (and others) and test_driver.update_virtual_sensor("accelerometer", {x: 1, y: 2, z: 3}).

If this is a worthwhile approach, then the two opens I see at the moment are

1. What kind of virtual sensor type to use for the deviceorientation and deviceorientationabsolute events. We can clearly use "accelerometer", "linear-acceleration" and "gyroscope" for DeviceMotionEvent, but there's no clear mapping to one or more sensor types for DeviceOrientationEvent. One could require "accelerometer" and "gyroscope" for relative orientation, for example, but implementations are arguably free to obtain the Euler angles data from the platform in another manner altogether, so I'm tempted to just go with making this specification define additional virtual sensor types, "relative-orientation-data" and "absolute-orientation-data" that takes Euler angles directly.
2. At the risk of contradicting some of the statements above, using 3 separate sensors ("accelerometer", "linear-acceleration" and "gyroscope") for DeviceMotionEvent may be racy since a new event may fire while one of the sensors is still being updated. In this case, we may need to define a new virtual sensor type that takes all 3 readings at once, and leave it to the implementation to update all readings at once on the platform side.

[reillyeon]

• What kind of virtual sensor type to use for the deviceorientation and deviceorientationabsolute events. We can clearly use "accelerometer", "linear-acceleration" and "gyroscope" for DeviceMotionEvent, but there's no clear mapping to one or more sensor types for DeviceOrientationEvent. One could require "accelerometer" and "gyroscope" for relative orientation, for example, but implementations are arguably free to obtain the Euler angles data from the platform in another manner altogether, so I'm tempted to just go with making this specification define additional virtual sensor types, "relative-orientation-data" and "absolute-orientation-data" that takes Euler angles directly.

I'm leaning towards the virtual sensors defined in https://w3c.github.io/orientation-sensor/#automation, the only issue being that they use quaternions instead of Euler angles.

• At the risk of contradicting some of the statements above, using 3 separate sensors ("accelerometer", "linear-acceleration" and "gyroscope") for DeviceMotionEvent may be racy since a new event may fire while one of the sensors is still being updated. In this case, we may need to define a new virtual sensor type that takes all 3 readings at once, and leave it to the implementation to update all readings at once on the platform side.

I don't know about iOS, but on Android the underlying platform API provides accelerometer and gyroscope data independently so these two data sources don't provide readings at the same time and I think it's reasonable for the test interface to reflect that since it is something implementations must deal with.

[rakuco]

• What kind of virtual sensor type to use for the deviceorientation and deviceorientationabsolute events. We can clearly use "accelerometer", "linear-acceleration" and "gyroscope" for DeviceMotionEvent, but there's no clear mapping to one or more sensor types for DeviceOrientationEvent. One could require "accelerometer" and "gyroscope" for relative orientation, for example, but implementations are arguably free to obtain the Euler angles data from the platform in another manner altogether, so I'm tempted to just go with making this specification define additional virtual sensor types, "relative-orientation-data" and "absolute-orientation-data" that takes Euler angles directly.

I'm leaning towards the virtual sensors defined in https://w3c.github.io/orientation-sensor/#automation, the only issue being that they use quaternions instead of Euler angles.

The simplest option would then be having some extra sensor types (e.g. "absolute-orientation-euler" and "relative-orientation-euler") with a custom parsing algorithm that reads alpha, beta and gamma values and ensures they are within the ranges we allow for each ([0, 360) for alpha, for example).

I can start working on a PR if this approach (including building it upon the virtual sensor infrastructure from Generic Sensors) sounds sane enough.

[rakuco]

Switching the existing tests in WPT to this approach does seem to work, doing this from a spec perspective is a bit tricky.

The idea is to be able to say "we only depend on the Automation section", but the Automation section itself references concepts from other parts of the spec.

For example, if we were to define a per-type virtual sensor metadata entry for "relative-orientation-euler", we'd need to define a corresponding virtual sensor type, which is a sensor type. A sensor type requires one or more IDL interfaces, powerful feature names and permissions policy tokens. Similarly, if we want to use the "accelerometer" virtual sensor type for Device Motion, it means referencing https://w3c.github.io/accelerometer/#accelerometer-automation, which references the Accelerometer sensor type that requires the presence of the Accelerometer IDL interface.

In practice, this is a lot of spec legalese, but it's hard to ignore when changing this specification.

Copying the content from the Generic Sensor spec doesn't fully solve the problem either, as we'd then have conflicting behavior definitions for sensors like accelerometer and gyroscope.

[reillyeon]

What if we added an Automation section to this spec which hooked into the virtual sensor update algorithms and said "when a quaternion value is provided here, convert it to Euler angles and provide it here"?

[rakuco]

I don't think this addresses the underlying problem of virtual sensors depending on and referencing <dfn>s that are not Automation-related. In the case of using accelerometer and gyroscope types for DeviceMotionEvent, for example, wouldn't it mean this spec would also implicitly depend on the Accelerometer sensor type and IDL interface?

[rakuco]

Hmm, I think if I tweak the Automation section in the Generic Sensor spec, I can remove the explicit link between virtual sensors and sensor types in the per-type virtual sensor metadata mapping by making the sensor type itself optionally have the same string that identifies a virtual sensor type.

The Device Orientation spec side would still be a bit hand-wavy, but I guess that's to be expected.

[reillyeon]

Agreed, the challenge is that the Generic Sensors spec has "platform sensor" as a core concept which can be replaced by a virtual sensor while this specification what the source of the data is.

[rakuco]

Hmm, I think if I tweak the Automation section in the Generic Sensor spec, I can remove the explicit link between virtual sensors and sensor types in the per-type virtual sensor metadata mapping by making the sensor type itself optionally have the same string that identifies a virtual sensor type.

The Device Orientation spec side would still be a bit hand-wavy, but I guess that's to be expected.

I've made an attempt to do this in w3c/sensors#475. The Device Orientation side is in #124.