sdk and doc for vision glass

[dingsing2021]

developer guides:

1. add the the AliceIMU-1.0.0.aar into the libs dir of your project

2. Add the aar file as follows

   
   dependence{
   implementation firetree(dir:"libs",include :['*.aar'])

}

3. Add the codes

> API list

[svillar]

After reviewing the API and docs we have several doubts:

The documents describe a very minimal Java API that can basically do 3 things as far as I understand:

• check connection with glasses (and ask for USB permissions if needed)
• set the operation mode of the glasses (2D/3D)
• get a quaternion from the glasses (I guess it's the orientation?)

With this there is very little an app developer could do IMHO. There are so many things missing: controllera states, display information, texture data,graphics library initialization, event handling... that we are no longer sure about what's the scope of that API. Is it just meant to initialize the device?

For example all the other devices that we support in Wolvic provide a native (i.e. C++) API to interact with the system. As an example:

• OpenXR backend (used by all devices nowadays): https://github.com/Igalia/wolvic/blob/main/app/src/openxr/cpp/DeviceDelegateOpenXR.cpp
• OculusVR backend (deprecated, used in the past for meta devices): https://github.com/Igalia/wolvic/blob/main/app/src/oculusvr/cpp/DeviceDelegateOculusVR.cpp
• WaveVR backend (obsolete, used by FxR for HTC devices) https://github.com/Igalia/wolvic/blob/main/app/src/wavevr/cpp/DeviceDelegateWaveVR.cpp

So at this point I think we can summarize our analysis in two questions:

• What is this proposed API for? Just to get the information I wrote at the beginning of the email?
• How is the rest of the data/functionality obtained? Does it support OpenXR?

[dingsing2021]

here is the feedback

- check connection with glasses (and ask for USB permissions if needed) if the app want set the 2d/3d mode, get the imu data, the app first to check the glass connect status, and get the USB permissions.

- set the operation mode of the glasses (2D/3D) if the glass run in 2d mode, the glass will copy the frame to the left and right eye display. if the glass run in 3d mode, the glass will cut the frame in the middle (left and right) , and sentleft and right eye display. so in this mode , the source frame should be 38401080, and the glass cut the frame to 19201080.

- get a quaternion from the glasses (I guess it's the orientation?)

vision glass have a 3dof imu , so it can be support watch 3dof videos. the app can get the orientation of glass.

controllera states, display information, texture data,graphics library initialization, event handling...

• it seems that wolvic could runs like a cardboard mode , and copy the display to the vision glass.

right now ,we do not have the openxr api.

[svillar]

OK, so if I understand correctly, this device basically expects (for the 3D case) apps to render a frame in the device screen, split in half for the right eye and the left eye.

Then my next question is, how can we interact with the application? Will the phone be used as a 3DoF controller? Or how are we supposed to do clicks and other actions?

[dingsing2021]

for the interact , here is 2 options: 1, we can use the phone touch screen or the compute box button. -- here is an reference (search and install vision glass app on huawei app gallery), and you can also take experience on compute box with vision glass. 2, phone and compute box have the imu ,so it can be the 3dof controller. but it needs some works , can wolvic take some evaluate.( because we do not have any openxr runtime and sdk)

[matlu]

for the interact , here is 2 options: 1, we can use the phone touch screen or the compute box button. -- here is an reference (search and install vision glass app on huawei app gallery), and you can also take experience on compute box with vision glass. 2, phone and compute box have the imu ,so it can be the 3dof controller. but it needs some works , can wolvic take some evaluate.( because we do not have any openxr runtime and sdk)

I believe this is the app: I found a great app in AppGallery. Try it out![https://url.cloud.huawei.com/njNt1D9obK]

[matlu]

This is how to enter special characters for e.g., Wi-Fi Password. @svillar

[matlu]

Easiest way to setup a box is likely by connecting it to an external monitor. See flow below. @svillar

[matlu]

This is the SW version used. Do you have the same version? @svillar

[matlu]

Here is Video showing how to successfully type in a password using the D-pad and have it connected to an external monitor while doing it. @svillar https://github.com/open-ecosystem-development/OpenXR-SDK/assets/6307430/20aa845b-7230-4440-b1c3-8743646fd351

[svillar]

This is the SW version used. Do you have the same version? @svillar

I don't have that same menu. The most similar one is

[svillar]

Here is Video showing how to successfully type in a password using the D-pad and have it connected to an external monitor while doing it. @svillar https://github.com/open-ecosystem-development/OpenXR-SDK/assets/6307430/20aa845b-7230-4440-b1c3-8743646fd351

I'm also attaching a video where you can see how in my keyboard (which looks different) it is possible to enter the number 7 for example, but not possible to enter the < character

https://github.com/open-ecosystem-development/OpenXR-SDK/assets/1826122/0ef96eb1-3864-42a6-9d85-e4abee71408d

[dingsing2021]

Here is the solution： 1、please Switch to “拼音九键” 2、Switch to “en” （left bottom button） 3、then you can find the <> characters

[svillar]

Here is the solution： 1、please Switch to “拼音九键” 2、Switch to “en” （left bottom button） 3、then you can find the <> characters

Again the problem is not finding the characters. Please check the video I posted here. The characters are there in the keyboard but clicking on them does nothing. In that video I click 3 times in the <. Nothing happens. Then I click on the 7and it works. Then I click again in the < and nothing happens.

[dingsing2021]

hi, please show the full how-to video of https://github.com/open-ecosystem-development/OpenXR-SDK/issues/49#issuecomment-1719332504 and we think that the input worked in chinese mode，so the special charactors "><" is “《》“； so the first step that you should switch the input from chinese mode to English mode.

[dingsing2021]

and for vision glass and sdk, here is another option it can also worked in the huawei phone(mate 30,40,50 ect series) which is support typec DP out. so you can try to develop and debug @ vision glass + huawei phone. the vision glass connect to the phone via typec.

[svillar]

hi, please show the full how-to video of #49 (comment) and we think that the input worked in chinese mode，so the special charactors "><" is “《》“； so the first step that you should switch the input from chinese mode to English mode.

OK I managed to enter the proper characters now. Thanks

[svillar]

and for vision glass and sdk, here is another option it can also worked in the huawei phone(mate 30,40,50 ect series) which is support typec DP out. so you can try to develop and debug @ vision glass + huawei phone. the vision glass connect to the phone via typec.

Ah that's interesting does it mean that the Huawei phones do already have the SDK installed?

[dingsing2021]

and for vision glass and sdk, here is another option it can also worked in the huawei phone(mate 30,40,50 ect series) which is support typec DP out. so you can try to develop and debug @ vision glass + huawei phone. the vision glass connect to the phone via typec.

Ah that's interesting does it mean that the Huawei phones do already have the SDK installed?

hi svillar，you are right, and here is an app “huawei vision glass” base on the sdk,and you can try to install , and take expericence.> I believe this is the app: I found a great app in AppGallery. Try it out![https://url.cloud.huawei.com/njNt1D9obK]

[svillar]

I've tried to use the API both with a Mate 40 Pro and with the Vision Glass and I have several issues:

1. IMUManager.getManager().getImuDevice().isConnected() always return false to me (I know the glasses are properly attached to the device because I can see content in the glasses)
2. In both of them I get error -2 with the following text devices is null in the onError methods when calling switchDisplayMode() and getClassState()

[dingsing2021]

1，maybe your apps have not got the usb permission , and the system will push a window on the screen，and you should click "agree".

2，the AliceImuDemo-master.zip we sent , here is an Alice-IMU-Demo-v1.0.0.apk and you can try it. 3, and you can download and install huawei vision glass app in huawei app gallary , and try it.

[svillar]

1，maybe your apps have not got the usb permission , and the system will push a window on the screen，and you should click "agree".

I can see a dialog in the Mate Pro. But it isn't the same as the one in your screenshot. It does not say anything about Vision Glass. It's a generic dialog to allow the phone to project the screen in another device (in this case the glasses). I can see the screen contents projected in the glasses, but even after that the isConnected() call still returns false to me. Do I need anything special in the manifest maybe? BTW as instructed in the API document (or at least that's why I understand after translating it) I have to call IMUManager.init() on the Activity's onCreate method right?

2，the AliceImuDemo-master.zip we sent , here is an Alice-IMU-Demo-v1.0.0.apk and you can try it. 3, and you can download and install huawei vision glass app in huawei app gallary , and try it.

I tried the vision glass app and it works in the phone.

[svillar]

BTW I also tried to call IMUManager.getManager().getImuDevice().getUsbManager().getDeviceList() and it returns a HashMap with one device which is /dev/bus/usb/002/002 so I think glasses are properly detected. Maybe they're detected as a generic screen/monitor instead of the Vision Glass?

[dingsing2021]

1，maybe your apps have not got the usb permission , and the system will push a window on the screen，and you should click "agree".

I can see a dialog in the Mate Pro. But it isn't the same as the one in your screenshot. It does not say anything about Vision Glass. It's a generic dialog to allow the phone to project the screen in another device (in this case the glasses). I can see the screen contents projected in the glasses, but even after that the isConnected() call still returns false to me. Do I need anything special in the manifest maybe? BTW as instructed in the API document (or at least that's why I understand after translating it) I have to call IMUManager.init() on the Activity's onCreate method right?

2，the AliceImuDemo-master.zip we sent , here is an Alice-IMU-Demo-v1.0.0.apk and you can try it. 3, and you can download and install huawei vision glass app in huawei app gallary , and try it.

I tried the vision glass app and it works in the phone.

here is another dialog ， it is for the display;

[svillar]

1，maybe your apps have not got the usb permission , and the system will push a window on the screen，and you should click "agree".

I can see a dialog in the Mate Pro. But it isn't the same as the one in your screenshot. It does not say anything about Vision Glass. It's a generic dialog to allow the phone to project the screen in another device (in this case the glasses). I can see the screen contents projected in the glasses, but even after that the isConnected() call still returns false to me. Do I need anything special in the manifest maybe? BTW as instructed in the API document (or at least that's why I understand after translating it) I have to call IMUManager.init() on the Activity's onCreate method right?

2，the AliceImuDemo-master.zip we sent , here is an Alice-IMU-Demo-v1.0.0.apk and you can try it. 3, and you can download and install huawei vision glass app in huawei app gallary , and try it.

I tried the vision glass app and it works in the phone.

here is another dialog ， it is for the display;

Right, that's the one I get, but no the Vision Glass one. How can I get the other one? Should I install anything special?

[dingsing2021]

Here is two dialog ： 1, this one is for only display, any screen connect to the phone ,the OS system will push it ,and the user must click agree, if not , the phone system will not output anything , the screen will be black ,

2, the second one is for the usb permission, the apps get the imu data from vision glass hardware，send commant to switch display mode 3d etc all via the usb ，so the apps should get the usb permission first from the OS system， and the system will push this dialog，the user must click agree.

[dingsing2021]

here is the demo code :

`public class MainActivity extends AppCompatActivity implements View.OnClickListener { private TextView infoView;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    IMUManager.init(getApplication());

    setContentView(R.layout.activity_main);

    infoView = findViewById(R.id.info);
    infoView.setOnClickListener(this);

    getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
}

@Override
public void onClick(View v) {
    if (v.getId() == R.id.info) {
        if (IMUManager.getManager().getImuDevice().isConnected()) {
            if (IMUManager.getManager().getImuDevice().hasUsbPermission()) {
                Glass.getInstance().startImu(new Glass.ImuListener() {
                    @Override
                    public void onImu(double w, double x, double y, double z) {

                    }
                });
            } else {
                IMUManager.getManager().getImuDevice().requestUsbPermission();
            }
        } else {
            Toast.makeText(this, R.string.connect_vision_glass, Toast.LENGTH_LONG).show();
        }
    }
}}`

[dingsing2021]

hardware: huawei vision glass + mate40pro First of all, the wolvic runs in 3dof mode to support watching some VR 360/180 videos, 3d videos, and even playing some simple 3dof webxr applications or games. In this case, you need to use the off-screen display mode when connecting to the phone, https://developer.android.com/reference/android/app/Presentation

the phone can be simulated as a 3dof controller. Use the IMU on the mobile phone to implement 3dof. and the screen of the mobile phone is used as a touchpad, and a touch button (return, confirmation, menu, etc.) is set.

[svillar]

BTW one of the problems I have with the device is that for no apparent reason, the UI is not shown in the glasses, just a gray rectangle. I still can launch applications via adb from the laptop, but I cannot see the system UI. Is there any way to reset the device?

[dingsing2021]

BTW one of the problems I have with the device is that for no apparent reason, the UI is not shown in the glasses, just a gray rectangle. I still can launch applications via adb from the laptop, but I cannot see the system UI. Is there any way to reset the device?

the problem happend in the compute box?

[svillar]

BTW one of the problems I have with the device is that for no apparent reason, the UI is not shown in the glasses, just a gray rectangle. I still can launch applications via adb from the laptop, but I cannot see the system UI. Is there any way to reset the device?

the problem happend in the compute box?

yes only in the compute box

[dingsing2021]

BTW one of the problems I have with the device is that for no apparent reason, the UI is not shown in the glasses, just a gray rectangle. I still can launch applications via adb from the laptop, but I cannot see the system UI. Is there any way to reset the device?

the problem happend in the compute box?

yes only in the compute box

please first debug in the mate40+vision glass. and we will give you an guide to reset or reintall the compute box os system.

[dingsing2021]

BTW one of the problems I have with the device is that for no apparent reason, the UI is not shown in the glasses, just a gray rectangle. I still can launch applications via adb from the laptop, but I cannot see the system UI. Is there any way to reset the device?

the problem happend in the compute box?

yes only in the compute box

do you try to reboot the compute box?

[svillar]

BTW one of the problems I have with the device is that for no apparent reason, the UI is not shown in the glasses, just a gray rectangle. I still can launch applications via adb from the laptop, but I cannot see the system UI. Is there any way to reset the device?

the problem happend in the compute box?

yes only in the compute box

do you try to reboot the compute box?

yep, several times. It used to work, but then after some time, I could only see the gray rectangle

[svillar]

here is the demo code :

`public class MainActivity extends AppCompatActivity implements View.OnClickListener { private TextView infoView;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    IMUManager.init(getApplication());

    setContentView(R.layout.activity_main);

    infoView = findViewById(R.id.info);
    infoView.setOnClickListener(this);

    getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
}

@Override
public void onClick(View v) {
    if (v.getId() == R.id.info) {
        if (IMUManager.getManager().getImuDevice().isConnected()) {
            if (IMUManager.getManager().getImuDevice().hasUsbPermission()) {
                Glass.getInstance().startImu(new Glass.ImuListener() {
                    @Override
                    public void onImu(double w, double x, double y, double z) {

                    }
                });
            } else {
                IMUManager.getManager().getImuDevice().requestUsbPermission();
            }
        } else {
            Toast.makeText(this, R.string.connect_vision_glass, Toast.LENGTH_LONG).show();
        }
    }
}}`

I have that very same code in Wolvic (I took it from the documentation). But isConnecteddoes always return false to me. I can launch the VisionGlass app without any problem though.

[svillar]

vgsampleapp.tar.gz

So I decided to build a very simple app to try the API just to discard that we have some problem inside wolvic. I'm attaching the app here. You just need to place the AliceIMU aar file inside app/libs directory and adjust the path in local.properties. If I try this in the Mate Pro I also get device not connected error. Mind trying this on any of your devices and check whether it works?

[dingsing2021]

if (IMUManager.getManager().getImuDevice().isConnected()) {

it is not the device problem，“vision glass app” works well in your mate40pro+vision glass . here is an simple way to check：vision glass app get the SN. https://consumer.huawei.com/cn/support/content/zh-cn15928866/ 通过Vision Glass APP查询 眼镜连接手机后，打开Vision Glass App >右上角图标 （关于）> 点击SN处即可查看。

[svillar]

if (IMUManager.getManager().getImuDevice().isConnected()) {

it is not the device problem，“vision glass app” works well in your mate40pro+vision glass . here is an simple way to check：vision glass app get the SN. https://consumer.huawei.com/cn/support/content/zh-cn15928866/ 通过Vision Glass APP查询 眼镜连接手机后，打开Vision Glass App >右上角图标 （关于）> 点击SN处即可查看。

How can I send you the SN ?

[svillar]

if (IMUManager.getManager().getImuDevice().isConnected()) {

it is not the device problem，“vision glass app” works well in your mate40pro+vision glass . here is an simple way to check：vision glass app get the SN. https://consumer.huawei.com/cn/support/content/zh-cn15928866/ 通过Vision Glass APP查询 眼镜连接手机后，打开Vision Glass App >右上角图标 （关于）> 点击SN处即可查看。

How can I send you the SN ?

Also have you tried the app I sent you? Does it work?

[dingsing2021]

if (IMUManager.getManager().getImuDevice().isConnected()) {

it is not the device problem，“vision glass app” works well in your mate40pro+vision glass . here is an simple way to check：vision glass app get the SN. https://consumer.huawei.com/cn/support/content/zh-cn15928866/ 通过Vision Glass APP查询 眼镜连接手机后，打开Vision Glass App >右上角图标 （关于）> 点击SN处即可查看。

How can I send you the SN ?

it is a way to check the hardware and firmware of the vision glass，if you can get the SN，that means it's working properly. and you can share the SN here.

[dingsing2021]

vgsampleapp.tar.gz

So I decided to build a very simple app to try the API just to discard that we have some problem inside wolvic. I'm attaching the app here. You just need to place the AliceIMU aar file inside app/libs directory and adjust the path in local.properties. If I try this in the Mate Pro I also get device not connected error. Mind trying this on any of your devices and check whether it works?

we have test your demo on our side，it works. so we should check the firmware of your vision glass,here is the guide. https://consumer.huawei.com/cn/support/content/zh-cn15932056/ maybe it caused by the firmware version， it is our fault，we should upgrade the firmware to the latest， another PID VID is used. As a result, the SDK cannot identify the vision glass. so you need to upgrade the firmware，and we will send a tools update the @matlu ， please help to forward it.

[svillar]

The SN is RFDE322924000341

I tried to follow the guide but as it's in Chinese only it's a bit difficult. However I think I found the window where it displays the version and it says 1.02.929

[dingsing2021]

The SN is RFDE322924000341

I tried to follow the guide but as it's in Chinese only it's a bit difficult. However I think I found the window where it displays the version and it says 1.02.929

we find a way to fixed this bug, it does not need to upgrade the firmware of vision glass. we have sent a new sdk @matlu , please help to forward it.

[svillar]

The SN is RFDE322924000341 I tried to follow the guide but as it's in Chinese only it's a bit difficult. However I think I found the window where it displays the version and it says 1.02.929

we find a way to fixed this bug, it does not need to upgrade the firmware of vision glass. we have sent a new sdk @matlu , please help to forward it.

Thanks for the new SDK, it's working now and I can access the quaternion reported by the device.

Now with regard to the compute box do you know how to make it work again? If you recall, I mentioned previously that for me it's now showing a gray rectangle instead of the contents.

[dingsing2021]

The SN is RFDE322924000341 I tried to follow the guide but as it's in Chinese only it's a bit difficult. However I think I found the window where it displays the version and it says 1.02.929

we find a way to fixed this bug, it does not need to upgrade the firmware of vision glass. we have sent a new sdk @matlu , please help to forward it.

Thanks for the new SDK, it's working now and I can access the quaternion reported by the device.

Now with regard to the compute box do you know how to make it work again? If you recall, I mentioned previously that for me it's now showing a gray rectangle instead of the contents.

we first focus on vision glass + huawei phone. vision glass+ compute box low priority.

[svillar]

OK now I have another question. For the Vr3D case how should applications render it? Should apps split the rendered texture half for the left eye and half for the right eye? What's the recommended interpupillary distance for the glasses? Do you have any example code for that case?

[matlu]

I've some additional questions for the SDK team.

I'm still trying to get the vr3d display mode working. The problem is that when the app is executed the surface that is created by Android has the size of the phone's screen (as it can be seen in onSurfaceChanged()) instead of the glasses' size. We need to tell Android that we want to use the external screen (the VisionGlass) to render content.

The way I have found to achieve that is a bit cumbersome so that's why I'd like to share with you what I have done just in case you're doing something simpler/better in the VisionGlass app.

Basically I use the DisplayManager object to retrieve the list of active displays and then I get the id of the external one. Then I use it in ActivityOptions.setLaunchDisplayId() that I use to call startActivity(intent,options) so that the application is relaunched and displayed in a different display (at least in theory I managed to get that working with a simple app but not with wolvic). If I do that then the surface created by android matches the size of the VisionGlass.

However I find this whole process of restarting the application to be executed in the external display really suboptimal. How do you do it in the VisionGlass app? @dingsing2021

[dingsing2021]

I've some additional questions for the SDK team.

I'm still trying to get the vr3d display mode working. The problem is that when the app is executed the surface that is created by Android has the size of the phone's screen (as it can be seen in onSurfaceChanged()) instead of the glasses' size. We need to tell Android that we want to use the external screen (the VisionGlass) to render content.

The way I have found to achieve that is a bit cumbersome so that's why I'd like to share with you what I have done just in case you're doing something simpler/better in the VisionGlass app.

Basically I use the DisplayManager object to retrieve the list of active displays and then I get the id of the external one. Then I use it in ActivityOptions.setLaunchDisplayId() that I use to call startActivity(intent,options) so that the application is relaunched and displayed in a different display (at least in theory I managed to get that working with a simple app but not with wolvic). If I do that then the surface created by android matches the size of the VisionGlass.

However I find this whole process of restarting the application to be executed in the external display really suboptimal. How do you do it in the VisionGlass app? @dingsing2021

A presentation is a special kind of dialog whose purpose is to present content on a secondary display. here is an reference ： https://developer.android.google.cn/reference/android/app/Presentation#public-constructors

and vision glass app almost do it like this. https://zhuanlan.zhihu.com/p/352878364

and maybe you do not use the "Presentation"?

[dingsing2021]

do you have any update ？

[svillar]

do you have any update ？

Yes, good news actually. We're going to send you soon an APK with Wolvic rendering in the glasses and a very minimalistic UI in the phone. 3D stereoscopic rendering is still not yet working fine as we haven't managed yet to generate the correct images for each eye. I do have a question though: are there any recommended FOV (field of view) values (horizontal and vertical) to be used to generate the projection views? APIs like OpenXR provide that info but as far as I know there is no way to do that with the aliceimu API

[dingsing2021]

do you have any update ？

Yes, good news actually. We're going to send you soon an APK with Wolvic rendering in the glasses and a very minimalistic UI in the phone. 3D stereoscopic rendering is still not yet working fine as we haven't managed yet to generate the correct images for each eye. I do have a question though: are there any recommended FOV (field of view) values (horizontal and vertical) to be used to generate the projection views? APIs like OpenXR provide that info but as far as I know there is no way to do that with the aliceimu API

the fov is 36° * 21°； and there is no api to get this parameter.