"Proper" way of catching Google Cardboard button events

[IvoJager]

Please correct me if I'm wrong, but there appears to be precious little documentation or support around detecting Carboard Button presses.

I'm aware of the gaze cursor/click paradigm, but there are many other use cases for Cardboard button presses that are cursor/gaze independent.

I'm also aware that screen presses - for example - can be handled by regular JavaScript. However, screen presses and touch events don't seem to be making it as events while in VR mode on mobile (Firefox and Chrome).

Is there are a "proper" way of catching screen presses in A-Frame? If so, would it be worthwhile adding this to the docs?

• A-Frame Version: ALL
• Platform / Device: Chrome & Firefox on Android

[dsinni]

I was pretty sure that click just handled the button press. At least in my experience.

[dsinni]

You could probably use mousedown and mouseup as well, depending on your use case.

[IvoJager]

Thank you.

My issue is; what element is the correct/prescribed element to put the listener on? E.g. I need to catch every button press originating from cardboard (preferably avoid "click" events generated by gaze control/fusing).

[IvoJager]

This is my situation; we use gazed control/fusing for navigating a UI. We'd like to use the cardboard button - if available - as a means to view different angles within the scene. (you'll just have to trust me the ergonomics of this UI are optimal... :)

[dsinni]

This sounds like a unique use case, and I don't speak for the A-Frame creators, but could you not just listen for window events like window.addEventListener('click', handlerFn) in that case? I'm not sure there is any other specified event for Cardboard other than click, mousedown, mouseup, etc.

Or is that what you're saying is not working for you?

[IvoJager]

To be honest, the most basic View-Master style app would have the same issue - as far as I can tell.

I did indeed try to attach to the document, even click events don't seem to bubble up while in mobile VR mode.

Appreciate the lighting fast response btw!

[dsinni]

No problem!

Ah, you know what? Try listening for the touchstart and touchend events on the canvas element. That should probably do it.

I'm not sure that's the "offcial" solution, but hopefully that will do what you need.

[dsinni]

You should be able to grab the canvas reference from either this.el.sceneEl.canvas from within a component or AFRAME.scenes[0].canvas from outside of a component.

[IvoJager]

Bingo! I thought I had tried everything. Thanks a bunch!

We're trying to be good citizens, and be as "progressive" with our enhancements as we can. We've just launched the VR piece of our automated "everything" builder and Cardboard remains a great gateway for getting people enthused about VR.

I hope we can get an "official" solution for the Carboard button in, before the A-Frame 1.0 freeze.

Thanks again!

[IvoJager]

Hmmm.... I spoke too soon. This works on Firefox, but not on Chrome. :(

[IvoJager]

Ok. Seems I'm not alone; https://stackoverflow.com/questions/49103088/how-to-listen-to-click-event-on-android-chrome-in-webvr-a-frame

A solution is given in that thread, but it doesn't appear to be a trivial one. FWIW, it is asserted in that thread that this is "a bug in A-Frame".

EDIT: tried that solution, but could not get that to work either.

[dmarcos]

In carboard mode the screen is active and input is taps / clicks on screen (touchstart, click...). You can listen into the window element. window.addEventlistener

[dmarcos]

In proper WebVR mode screen is not active so taps on screen won’t work. Cardboards with magnet will trigger a click event through the runtime (but that hase not been very reliable). There’s nothing A-Frame can do if screen is not active

[dmarcos]

Only generic solution is not relying on screen input and use a cursor (fuse / gaze based for maximum compatibility) https://aframe.io/docs/0.8.0/components/cursor.html#sidebar

[dsinni]

@IvoJager, @dmarcos would know best, but I can confirm that movement-controls from @donmccurdy's aframe-extras currently works on Android Chrome (Samsung Galaxy S9+) while in VR mode, and I believe it relies on touch-controls to handle the events. This code might be worth a look: https://github.com/donmccurdy/aframe-extras/blob/master/src/controls/touch-controls.js

I just tested using the latest A-Frame master build: A-Frame Version: 0.8.2 (Date 2019-01-23, Commit #a8baa43) and the latest aframe-extras: 5.0.1

[dmarcos]

@dsinni native VR (WebVR / WebXR) or Cardboard mode? Daydream devices don't emit touchstart events when in VR mode.

[dsinni]

@dmarcos Ah, maybe that's the difference? For my test, I'm simply using Chrome on Android, entering VR and then inserting in a Cardboard device. I'm not using anything Daydream related, so I can't speak to that.

[dsinni]

Looks like my phone is now Daydream compatible, though, so maybe I'll try that out.

[dsinni]

@dmarcos Doesn't Daydream use a 3dof controller much like Gear VR and Oculus Go?

[dmarcos]

@dsinni I mean. If you want a single input method that covers all platforms fuse gaze based cursor is the only option. In any case, my recommendation is to not rely on any smartphone screen input (magnet based , touchstart...) method when in VR. They're just not reliable or not available. A-Frame cannot cover for the lack of platform support.

[IvoJager]

@dmarcos @dsinni Will give movement-controls a go and report back, thanks!

That said, I must say it's a bit odd to hear an A-Frame maintainer dismiss what is likely the biggest and oldest audience for A-Frame, namely Cardboard devices, as too "unreliable". If anything, Cardboard - warts and all - is the most popular "standard" (there's even a Cardboard certification program if I'm not mistaken!) we have and should(?) target. The v2 button design is a great improvement. On top of that, some Cardboard devices have BT clickers that are very reliable indeed. I'd urge A-Frame maintainers not to dismiss Cardboard - and that includes its input mechanisms.

Fuse/gaze based interaction is great, but the progressive enhancement paradigm (unless A-Frame does not ascribe to that?) means we should be using every bit of capability available for VR experiences.

Indeed, the button definitely adds an important dimension to our not-too-crazy use case; allowing the user to switch between different, predefined angles/locations in a scene. The Google Cardboard demos show plenty of other use cases for the button.

Thanks to A-Frame, our project now generates VR experiences that work on absolutely everything and progressively enhance the user experience. If we can get the Cardboard button to work, we'd cover a whole lot of devices!

[ngokevin]

Detecting a screen touch with Cardboard / polyfill is a matter of listening to touch/click events on the window. In some browsers with official WebVR, they may disable that functionality. Cardboard is hardly the most popular or biggest audience, and few people actually use them outside of being forced to in educational environments or developers that believe it's democratizing.

We have set the focus of A-Frame to create truly engaging VR experiences and every project needs a focus. Please don't call it "dismissive", that's unfair. We've been pretty ingrained in the VR ecosystem for years and interacted with thousands of people to have formed a good idea of where we want A-Frame to go. At least it still works for now, but it's hardly the focus, and the industry has moved on sharply showing it's going to be gone and deprecated.

Anyways, refer to @dmarcos and @dsinni's comments. Should just be a click or touchstart event if it's even possible.

[dmarcos]

Screen based events are not always available on smartphone VR. I'm not being dismissive about Cardboard VR just offering advice on the input method I think will cover the largest user base with least effort and best results. Gaze / Fuse based cursor, will work regardless of phone, OS (Cardboard certification is a Google Android thing, not applicable on iOS) or cardboard holder that differ in capabilities.

When the WebVR API is available in the browser. screen gestures are disabled upon entering VR.

If the WebVR API is not available and the polyfill is used screen input is not always available:

• Some cardboards have a button that tap the screen generating touchstart, click events...
  • Some cardboards have a magnet that is often not reliable in all phones.
  • Some cardboards don't have any buttons.

The capabilities above are hard or impossible to detect in some scenarios for an application to degrade gracefully. That's why I think gaze / fuse based cursor is the most robust solution.

[IvoJager]

@dmarcos Gaze / fuse based cursor is certainly a robust common denominator. However, progressive enhancement is not the same as degrading gracefully - they are two different concepts. The Google Cardboard button is an enhancement over the common denominator as much as a separate controller is. Being hard or easy to detect for the purpose of degrading gracefully is irrelevant; the presence of a button is meant to enhance the base - gaze/fuse experience. It is by no means a prerequisite to enjoy the base experience. Notwithstanding the technical challenges you outline of course, all we're asking for is to be able to enhance the experience on headsets with a (hopefully) reliable button. If it works, great! If it doesn't or isn't available, no harm done.

@ngokevin I do not doubt for a minute you've been pretty ingrained in the VR ecosystem for a long time. I, for one, am a direct beneficiary of your contributions and expertise. However you are rather off the mark if you think Cardboard is not the biggest audience for us. Perhaps it's because we operate in a different geography (Asia, Asia Pacific, LatAm) and industry with different budgets and expectations. Or perhaps your perception of Cardboard's popularity is developer-oriented rather than coal-face, end-user oriented. Regardless, for us, Cardboard, by virtue of its low cost (it's often given out for free), commands by far the biggest VR audience. Most consumers here balk at shelling out for an Oculus Go. Most consumers here will also happily put on the foldable headset and use the cheapo BT controller they just got for free with their magazine, newspaper, phone subscription, real estate viewing or bank account. Don't take it from me either, see Emma Mankey Hidem and Peter Maddalena. From where we are sitting, Cardboard's audience and growth is vast. Other headsets, not so much for obvious reasons (cost vs need). Apologies for the wall of text, but I really wanted to point this out.

All that said, thank you for your answers. I will see what I can come up with.

Given the alluded to direction for A-Frame, is there still any interest from the A-Frame team in catering to one of the most popular Cardboard add-on controllers? (it's one of those $2.50 "Wireless Bluetooth VR-BOX Remote Control Gamepad" controllers). They send out some obscure keycodes, but once caught, they work exceedingly well for navigating an A-Frame VR experience.

[dmarcos]

@IvoJager What engagement / growth numbers do you see in the Cardboard space? What experiences in particular are you seeing growth? Any data you can share will be much appreciated.

Can you provide support for those cheap bluetooth controllers via 3rd party component? We will keep an eye on the space and can consider incorporating in core in the future if it turns into compelling experiences that users engage with.

[IvoJager]

@IvoJager What engagement / growth numbers do you see in the Cardboard space?

Firstly, this report by IDC shows screenless headsets, by far, having the largest cumulative market share. Note that IDC's stats only really count GearVR and similar offerings (only a portion of which are "upmarket" Cardboard) and that the decline occurred due to the unbundling of screenless VR headsets during the quarter.

Even from looking at these stats however, you should be aware that A-Frame's biggest potential audience, for a long time to come, is still screenless headsets.

Hard numbers are notoriously hard to come by in the VR space, let alone in "alternative" markets, but here is an analysis of the Russian market;

https://www.jsdrm.ru/jour/article/download/787/696

"By 2021 mobile devices of the augmented and virtual reality (approximately 75 % and 16 %) will be the most popular; the remaining share will be equally divided between wearable devices of virtual and augmented reality [After mixed year, 2017]."

Or this 2018 report from Perkins Coie LLP, highlighting cost concerns that producers of VR experiences have;

https://www.perkinscoie.com/images/content/1/8/v2/187785/2018-VR-AR-Survey-Digital.pdf

"Cost was the second biggest obstacle identified for VR." "Cost was viewed as a greater concern for VR (22%) than AR (14%)"

https://www.perkinscoie.com/images/content/1/8/v2/187785/2018-VR-AR-Survey-Digital.pdf

Daydream adoption is lagging in many markets because the popular (and cheaper) hardware lines, such as Xiaomi's RedMi, simply are not Daydream compliant, mostly due to cheaper screens that don't meet the low persistence requirements. Hence Xiaomi releasing a Cardboard headset as recent as mid-2017 for these markets.

There are articles like these, where we note that disposable income in many APAC countries means that Cardboard/mobile experiences are typically the only way to own VR capabilities.

However, most indicative of Cardboard's impressive reach, are the myriad branded VR headset suppliers;

https://www.google.com/search?q=branded+vr

These companies ship headsets by the thousands (quotes typically begin at 500 units or more). These headsets, even in 2019, still tend to be the very first VR experience regular consumers will have, especially in our markets. These are usually given away for free. When it comes to reach, you cannot beat free.

What experiences in particular are you seeing growth?

I am personally invested in making experiences on these devices the best I can at the lowest cost I can. Cardboard is simply the platform I work with the most, as the barrier to entry is virtually non-existent. This is far from a developer-led "starry eyed" democratization effort. It's just economic reality.

In terms of compelling experiences, brands here have been experimenting with VR on and off. First as a gimmick (the effort and cost was worth it just to been seen to be innovative), but now that we're past the hype, brands are - thankfully - looking at more practical applications.

Taking Australia as an example, these practical post-hype applications tend to focus around 360 tours and narrated experiences. Agriculture, tourism and education are big export markets for Australia.

Distances are vast and businesses see VR as a way to highlight the unique attributes of their location and environments, without the need for travel. It's a classic use case for VR that has actually survived the hype and has borne out.

We just released a project that autonomously turns website content like this; https://www.australianroses.com.au/

...into narrated (in VR mode) experiences like these; https://aife.me/australianroses-prod-vr/ (https://youtu.be/B0WPfVj-VYI?t=43)

This is a small, family run business, who would not normally invest (or be interested) in VR. It's not on their radar, nor had they considered using their location as a VR experience for their marketing efforts.
This experience was created at no extra cost (our AI does the heavy lifting, except for some 360 photography), and serves an actual purpose when engaging with new potential customers. It pairs perfectly with complementary cardboard devices and adds value to the sales process.

Can you provide support for those cheap bluetooth controllers via 3rd party component? We will keep an eye on the space and can consider incorporating in core in the future if it turns into compelling experiences that users engage with.

Definitely! Will create a component for this. If you're in the US, I believe they may have been marketed as part of the "Cynoculars" ("As seen on TV") a couple of years ago. They are currently sold alongside the VBox v2 and its clones/rebrands (Cynoculars was one), as well as the popular Bobo VR Z-series and its clones/rebrands.

[dmarcos]

Thanks. I was looking for specific examples of applications (vs surveys and analyst projections) that have shown high levels of engagement. We’ve had Cardboard and phone based holders for years and not a single application has shown significant growth / usage numbers.

To the original question: To detect Cardboard button events, listen to click and touchstart events (window.addEvenListener(‘click’ , function () {...})) but those might not be always available: Make sure there’s a fallback like gaze / fuse based cursor. Happy to answer usage questions on StackOverflow if something is unclear. I’m closing this issue for now since there’s nothing actionable on the A-Frame side. Thanks everybody for the passion and the info provided.

[IvoJager]

I’m closing this issue for now since there’s nothing actionable on the A-Frame side.

I did some more sleuthing and apparently the correct way is to listen to a Gamepad controller. For cardboard, this controller just has a single button. The fix has gone into Chromium ~ 18 months ago and should be available in the current stable version of Chrome.

Depending on whether the A-Frame team regards Cardboard support as deprecated, an official and less ambiguous statement on the latter would be most welcome so we (and others) can plan our roadmap better. Regardless, there is definitely an actionable item here.

I was looking for specific examples of applications (vs surveys and analyst projections) that have shown high levels of engagement. We’ve had Cardboard and phone based holders for years and not a single application has shown significant growth / usage numbers.

Whether Cardboard usage by one particular user for one particular application is ephemeral or prolonged is quite inconsequential to its success. Cardboard's success is exactly found in ephemeral experiences and usage. I think we can all agree that lens and tracking quality are often such that prolonged use is not advisable or desirable. However it is perfectly apt at conveying short experiences at virtually zero cost, hence its overwhelming success and reach.

People are still sinking money into Cardboard experiences in 2019 and getting an enthusiastic return from it;

https://www.engadget.com/2019/01/25/hamlet-360-virtual-reality-play/ https://www.smh.com.au/business/workplace/reality-but-not-as-you-know-it-20181108-p50evp.html https://theconversation.com/virtual-reality-tours-give-rural-students-a-glimpse-of-college-life-106667

I overheard a kid on the metro enthusiastically recounting his Cardboard experience to his dad just last week.

To the best of my abilities, I've given you numbers, surveys, articles, links to thriving suppliers, anecdotal experiences. I don't know what else you'd like me to supply?

[dmarcos]

Even if ephemeral, compeling applications would show word of mouth growth. Nothing relevant in 5 years of Cardboard I’m aware of. Some brands got attention by piggy backing on the VR hype. Once passed its peak the strategy is less and less effective.

For most cases that ephemeral content is better served by magic window mode that does not require any additional hardware besides a phone.

We saw a decline since 2016 in the amount of content produced for Cardboard and new 3rd party holders. I’m based in the US, but Spanish and A-Frame community is global. We have a good sample.

Main value of Cardboard is educational. Lowers the barrier of entry for devs to kick the tyres on WebVR development. The current Cardboard support is sufficient. Integrating the clicker in your application should be a 50 lines of code effort. A-Frame does all the heavy lifting at current level of support. I believe you won’t need to modify your roadmap. Happy to answer questions on how to support clicker if you get stuck

[kylebakerio]

Given that it's such a small (50 lines of code) effort, and not expected to change going forward and has been a standard for years (I just bought one for $14 with a well functioning button), I'm quite surprised this isn't built in to aframe. I just bought a cardboard to build a cardboard-based experience (I consider main VR headsets unaffordable and the audience too small, but it's a bonus to be able to support them at the same time for sure), and it's pretty disappointing that I have to go dig around to figure out how to implement this myself.

Even if, as you say, main benefit is educational (may be, though that's not my interest), the lack of gamepad support (no response from the bluetooth controller I have that has worked in everything I've tried until aframe and aframe-extras) and now I find not even the built-in button support is... I don't know, shouldn't aframe aspire to be a 'complete' solution? It feels like a direct rejection of cardboard to not include this support. I mean, ok, I'll write the code myself, and surely not as well as a project maintainer, but could a pull request to integrate this not be accepted at least?

FWIW, for my use case, I want to use the button to activate voice recognition, so I can use voice commands in my app. My controller isn't working with aframe at all so far, so while that'll take some more work, voice recognition was going to be lower effort to get mvp on mobile status and start toying around. Now that'll take research and digging and work too... Sure would be handy for prototyping thing, I was sure that'd be a built-in. (I'm glad this other guy mentioned the tip about it also being a gamepad event, saved me from that time lost.)

[kylebakerio]

@IvoJager before I go do it myself, did you end up making a component that did this that you have available by chance?

[IvoJager]

@kylebakerio Sorry, sadly that's a negative. We've got some code we turn on on a case-by-case basis, as reliable GC detection has been a PITA. You could do something like this;

// Google Cardboard button handler
if (AFRAME.utils.device.isMobile ())
{
    AFRAME.scenes[0].canvas.addEventListener('touchstart', gcbhandler);
    AFRAME.scenes[0].canvas.addEventListener('mousedown', gcbhandler);
}

function gcbhandler(evt)
{
    if (typeof evt.originalTarget != 'undefined')
    {
        if ($(evt.originalTarget).hasClass('a-grab-cursor'))
        {
            // register click
etc.

We have had better luck with the cheap BT controllers that often get bundled with GC devices.

// support for cheap chinese "game controller" (Cynoculars, VBOX 2)
                window.addEventListener('keydown', function(evt)
                {
                    switch(evt.key)
                    {
                        case 'MediaFastForward':
                            moveCamera(0, 0.25, false); // forward  
                            break;                          

                        case 'MediaRewind':
                            moveCamera(180, 0.5, false); // forward 
                            break;

                        case 'MediaTrackPrevious':
                            rotateCamera(22.5); // left 
                            break;

                        case 'MediaTrackNext':
                            rotateCamera(-22.5); // right   
                            break;

                        default:
                            break;                          
                    }
                });

Hope this helps!

[kylebakerio]

nice. Well, at least the code is out there now for the next desperate soul to google. Thanks for sending that.

[kylebakerio]

Just leaving this here for anyone googling in the future:

I cannot get any touch event to register in google cardboard in vr mode in android chrome. I also can't get any gamepad to show up in vr mode in android chrome. I can find where they supposedly implemented this:

Expose Gamepad API instance for Cardboard button This change exposes a gamepad with a single button, which is driven by the touch screen.

• https://bugs.chromium.org/p/chromium/issues/detail?id=718665

And I can see documentation that talks about it as a future feature:

In upcoming versions of Chrome a Google Cardboard viewer will be represented as a single button gamepad, and the state of other VR controllers will need to be read in this way anyway.

• https://developers.google.com/web/fundamentals/vr/vr-perspective/

But I see no such gamepad show up. No events fire at all, no gamepad nor mouse nor touch events. I see some indication from reading that perhaps at some point this worked on firefox, haven't experimented or targeted firefox up to this point, so I don't know about that, but perhaps that would be an option if this were a must-have feature, but that seems to basically make it a dead end for me.

If anyone sees that I'm doing something obviously wrong here or missing something, let me know. My cardboard click button works perfectly in android apps, it just seems that chrome has disabled it (from what I can tell, though I wish I could find an official chrome / google statement confirming this) in webvr.

EDIT

This library, by @DougReeder, works: https://dougreeder.github.io/aframe-button-controls/example.html

Haven't dug into it yet to see how it works, but it is perfectly reliable for my cardboard button in VR mode. yay!

[kylebakerio]

@IvoJager, if you were still interested, have you seen the link/library I posted at the bottom of my edited comment above? Has it (or its method) not worked for you on other cardboards, or have you seen it before? I'm already 80% in to implementing full controllers for my project, but I'm thinking I'll very likely make a '0 button mode', '1 button mode', and 'multi button mode' for VR.

[IvoJager]

@kylebakerio Cheers! That wasn't in existence when I was looking at this (mainstream Chrome was mid-transition to WebXR then). Nice find (and thanks Doug Reeder for implementing and updating this!).

[kylebakerio]

Just wanted to put this out there for the google results... the TLDR on getting access to touchscreen contact when in VR mode on an android device (so, cardboard mode), is listening for these events:

AFRAME.scenes[0].xrSession.addEventListener('selectstart', () => console.log("down"));        
AFRAME.scenes[0].xrSession.addEventListener('selectend', () => console.log("up"));

(you'll need to only add that listener once you're already in VR mode I imagine--you'll need an existing xrSession)

[kylebakerio]

@dmarcos I know this is an old discussion, but you had asked about data on cardboard usage--I just happened to run across this from what seems to be a relatively solid source: https://youtu.be/t-uk8InHte4?t=135

"But surprisingly, it's some of the simplest hardware and simplest experiences which have had the widest addoption. Hardware-wise, handset-based hardware is the most commonly used. So the simple cardboard headset, which is supported through the webxr API on android, are some of the commnest users of virtual reality..."