search

Azure / azure-spatial-anchors-samples

Azure Spatial Anchors sample code
Other
293 stars 138 forks source link

Coarse Relocation Crash - Android #387

Closed thejoshuafazel closed 1 year ago

thejoshuafazel commented 1 year ago

Description

Trying out Geolocation for Coarse Relocation and encountering a crash when enabling the sensors. Tried it out on two different projects, the first one being the Coarse Relocation Demo Example from this repository and the second one being a similar code to the example but with only Geolocation enabled. Building only for android.

Steps to reproduce the issue

Sample Application: First Run:

  1. Start the Coarse Relocation Demo
  2. Press 'Next Step' on "Configure Sensors" step
  3. Accept the prompt to enable location. (Note: only shows notification for location, not WIFI and GPS)
  4. Hard Crash without any log errors (The last log being the message the one indicating the move from DemoStepCreateLocationProvider to the DemoStepConfigureSensors step)

Second Run: *Note: The location permission is granted for the application when checking the application settings

  1. Start the Coarse Relocation Demo
  2. Press 'Next Step' on "Configure Sensors" step
  3. Hard Crash without any logs

My Application: The code Setup is very similar but only Geolocation is enabled in code, everything else is commented out from the sample code. Only one scene in the application i.e. the Coarse Relocation Scene along with the main menu.

First Run:

  1. Start the App
  2. Configure Sensors
  3. Accept the prompt to enable location. (Note: only shows notification for location, not WIFI and Bluetooth)
  4. Hard Crash without any logs

*Note: The location permission is granted for the application when checking the application settings First Run:

  1. Start the App
  2. Configure Sensors
  3. Hard Crash with the error given in the screenshots below.

Expected behavior

GPS Sensor should be enabled and can be used for setting up and viewing the anchor.

Screenshots

My Appplication:

image

The message 'True' is a log from the SensorPermissionHelper.HasGeoLocationPermission() function call.

The application throws the following error:

Autoconnected Player InvalidOperationException: CoarseReloc: Location access has been declined. at Microsoft.Azure.SpatialAnchors.NativeLibraryHelpers.CheckStatus (System.IntPtr handle, Microsoft.Azure.SpatialAnchors.status value) [0x00000] in <00000000000000000000000000000000>:0 at Microsoft.Azure.SpatialAnchors.PlatformLocationProvider.get_GeoLocationStatus () [0x00000] in <00000000000000000000000000000000>:0 at SetupAnchorScript.Update () [0x00000] in <00000000000000000000000000000000>:0

Development information

AR Device information (please complete the following information):**

Additional context

All logs have been enabled in the 'Logging Level' option. Bluetooth, GPS and WIFI are enabled at all times. All the other example scenes seem to be working fine so it does not look like a config issue.

Additionally, have tried my application with Android Version 12 (One UI-Version is 4.1) on a Galaxy A3 5G. It seems to be working fine on it, this seems to me like a device specific problem. Is there anything I can do to narrow down the issue?

Varnauld commented 1 year ago

Assigning to @darax (tracking n#42358353)

sergeiromanov commented 1 year ago

check if this is related https://github.com/Azure/azure-spatial-anchors-samples/issues/363 I don't now about unity but in android ndk you have to be careful with how you configure location provider sensors.

thejoshuafazel commented 1 year ago

@sergeiromanov Went through that issue before logging this one. My wifi, bluetooth and gps are ON before running the application.

darax commented 1 year ago

@thejoshuafazel Looking through https://forum.unity.com/threads/gps-not-work-well-on-android-12.1212534/ this sounds like a known issue that was fixed in a later unity build.

Can you try Unity >= 2020.3.31f1?

thejoshuafazel commented 1 year ago

@darax Tried the same project with unity 2020.3.42f1 and the crash still occurs.

Additional Information: While messing around in code, I tried asking for the location access before starting the ASA session and it works fine but as soon as the ASA session starts the app crashes.

darax commented 1 year ago

In logcat I see this: 2022-11-29 11:42:30.275 7061-7088/com.microsoft.azure.spatialanchors.examples.unity.helloar A/s.unity.helloa: java_vm_ext.cc:579] JNI DETECTED ERROR IN APPLICATION: JNI AllocObject called with pending exception java.lang.SecurityException: Need android.permission.BLUETOOTH_SCAN permission for AttributionSource { uid = 10412, packageName = com.microsoft.azure.spatialanchors.examples.unity.helloar, attributionTag = null, token = android.os.BinderProxy@db9381, next = null }: GattService registerScanner

I'm getting my lastest unity install setup with android so I can try myself, but I wonder if it works if you either don't ask for bluetooth location or the bluetooth_scan is added to the android manifest.

thejoshuafazel commented 1 year ago

Do you mean that either we dont ask for bluetooth location permission or the bluetooth_scan should be added to the android manifest?

darax commented 1 year ago

Yes. I'm halfway through building without a change now to make sure I'm still reproducing this. After I confirm I'll try the workarounds. Hopefully adding the bluetooth_scan permission to the manifest works.

darax commented 1 year ago

@thejoshuafazel I didn't see a crash when I rebuilt, but I was using the unity 2022.2 beta when I did this ( most of my time was spent futzing with input. :)). It was failing to complete the demo. to Assets/Plugins/Android/AndroidManifest.xml I added
<uses-permission android:name="android.permission.BLUETOOTH_SCAN"/> to Assets/AzureSpatialAnchors.Examples/Scripts/SensorPermissionsHelper I added: const string androidBluetoothScanPermission = "android.permission.BLUETOOTH_SCAN"; and in RequestSensorPermissions() is added a call: RequestPermissionIfNotGiven(androidBluetoothScanPermission);

after that the coarse reloc demo worked again. LMK if this works for you. If it doesn't, maybe there is a hint in logcat.

thejoshuafazel commented 1 year ago

@darax

Can you tell me the package versions you used for unity 2022.2 beta?

Using the following: XR Core v2.1.0 AR Foundation v5.0.3 Azure Spatial Anchors SDK v2.13.3

Getting the following errors when opening the project with unity 2022.2 beta:

Library\PackageCache\com.microsoft.azure.spatial-anchors-sdk.core@fc109db5a78b\Runtime\Scripts\SpatialAnchorManager.cs(404,17): error CS0012: The type 'XROrigin' is defined in an assembly that is not referenced. You must add a reference to assembly 'Unity.XR.CoreUtils, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.

Library\PackageCache\com.microsoft.azure.spatial-anchors-sdk.core@fc109db5a78b\Runtime\Scripts\SpatialAnchorManager.cs(404,61): error CS0012: The type 'XROrigin' is defined in an assembly that is not referenced. You must add a reference to assembly 'Unity.XR.CoreUtils, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.

Library\PackageCache\com.microsoft.azure.spatial-anchors-sdk.core@fc109db5a78b\Runtime\Scripts\SpatialAnchorManager.cs(404,61): error CS1061: 'ARSessionOrigin' does not contain a definition for 'enabled' and no accessible extension method 'enabled' accepting a first argument of type 'ARSessionOrigin' could be found (are you missing a using directive or an assembly reference?)

Library\PackageCache\com.microsoft.azure.spatial-anchors-sdk.core@fc109db5a78b\Runtime\Scripts\SpatialAnchorManager.cs(797,31): error CS0311: The type 'UnityEngine.XR.ARFoundation.ARSessionOrigin' cannot be used as type parameter 'T' in the generic type or method 'Object.FindObjectOfType<T>()'. There is no implicit reference conversion from 'UnityEngine.XR.ARFoundation.ARSessionOrigin' to 'UnityEngine.Object'.

Library\PackageCache\com.microsoft.azure.spatial-anchors-sdk.core@fc109db5a78b\Runtime\Scripts\SpatialAnchorManager.cs(797,31): error CS0012: The type 'XROrigin' is defined in an assembly that is not referenced. You must add a reference to assembly 'Unity.XR.CoreUtils, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.

image

thejoshuafazel commented 1 year ago

In the meantime, I looked through the logcat errors

The fatal error is 

2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594] JNI DETECTED ERROR IN APPLICATION: JNI NewString called with pending exception java.lang.IllegalArgumentException: provider "gps" does not exist
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at java.lang.Exception android.os.Parcel.createExceptionOrNull(int, java.lang.String) (Parcel.java:2460)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at java.lang.Exception android.os.Parcel.createException(int, java.lang.String) (Parcel.java:2440)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Parcel.readException(int, java.lang.String) (Parcel.java:2423)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Parcel.readException() (Parcel.java:2365)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.ILocationManager$Stub$Proxy.registerLocationListener(java.lang.String, android.location.LocationRequest, android.location.ILocationListener, java.lang.String, java.lang.String, java.lang.String) (ILocationManager.java:1445)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.LocationManager.requestLocationUpdates(java.lang.String, android.location.LocationRequest, java.util.concurrent.Executor, android.location.LocationListener) (LocationManager.java:1498)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.LocationManager.requestLocationUpdates(java.lang.String, long, float, java.util.concurrent.Executor, android.location.LocationListener) (LocationManager.java:1186)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.LocationManager.requestLocationUpdates(java.lang.String, long, float, android.location.LocationListener, android.os.Looper) (LocationManager.java:1151)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean com.microsoft.mixedreality.spatialservices.coarserelocalization.AndroidGeoLocationWatcher.resume() (AndroidGeoLocationWatcher.java:61)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean com.unity3d.player.UnityPlayer.nativeRender() ((null):-2)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean com.unity3d.player.UnityPlayer.access$300(com.unity3d.player.UnityPlayer) ((null):-1)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean com.unity3d.player.UnityPlayer$e$1.handleMessage(android.os.Message) ((null):-1)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Handler.dispatchMessage(android.os.Message) (Handler.java:102)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean android.os.Looper.loopOnce(android.os.Looper, long, int) (Looper.java:201)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Looper.loop() (Looper.java:288)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]   at void com.unity3d.player.UnityPlayer$e.run() ((null):-1)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594] Caused by: android.os.RemoteException: Remote stack trace:
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]    at com.android.internal.util.Preconditions.checkArgument(Preconditions.java:59)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]    at com.android.server.location.LocationManagerService.registerLocationListener(LocationManagerService.java:746)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]    at android.location.ILocationManager$Stub.onTransact(ILocationManager.java:585)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]    at android.os.Binder.execTransactInternal(Binder.java:1184)
2022-11-30 19:49:51.488 15378 15398 Fatal s.unity.helloa java_vm_ext.cc:594]    at android.os.Binder.execTransact(Binder.java:1143)
darax commented 1 year ago

@thejoshuafazel
For 2022.2 I'm using "com.unity.xr.arcore": "5.0.0-pre.13", "com.unity.xr.arfoundation": "5.0.0-pre.13", "com.unity.xr.arkit": "5.0.0-pre.13", "com.unity.xr.arsubsystems": "5.0.0-pre.13", "com.unity.xr.management": "4.2.0", "com.unity.xr.openxr": "1.5.1",

I would recommend first trying the unity version you were using (before 2022.2) while making the following modifications to the android files. to Assets/Plugins/Android/AndroidManifest.xml I added <uses-permission android:name="android.permission.BLUETOOTH_SCAN"/> to Assets/AzureSpatialAnchors.Examples/Scripts/SensorPermissionsHelper I added: const string androidBluetoothScanPermission = "android.permission.BLUETOOTH_SCAN"; and in RequestSensorPermissions() is added a call: RequestPermissionIfNotGiven(androidBluetoothScanPermission);

if you really want to use 2022.2, I posted instructions here: https://github.com/Azure/azure-spatial-anchors-samples/issues/389. The above modifications will still be necessary.

thejoshuafazel commented 1 year ago

@darax

I would rather stick to the version I was using before. Made the changes you suggested and still getting the same error as before. Pasting it below as well. You can find the full logs here : com.microsoft.azure.spatialanchors.examples.unity.helloar-logcat.txt

2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594] JNI DETECTED ERROR IN APPLICATION: JNI NewString called with pending exception java.lang.IllegalArgumentException: provider "gps" does not exist
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at java.lang.Exception android.os.Parcel.createExceptionOrNull(int, java.lang.String) (Parcel.java:2460)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at java.lang.Exception android.os.Parcel.createException(int, java.lang.String) (Parcel.java:2440)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Parcel.readException(int, java.lang.String) (Parcel.java:2423)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Parcel.readException() (Parcel.java:2365)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.ILocationManager$Stub$Proxy.registerLocationListener(java.lang.String, android.location.LocationRequest, android.location.ILocationListener, java.lang.String, java.lang.String, java.lang.String) (ILocationManager.java:1445)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.LocationManager.requestLocationUpdates(java.lang.String, android.location.LocationRequest, java.util.concurrent.Executor, android.location.LocationListener) (LocationManager.java:1498)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.LocationManager.requestLocationUpdates(java.lang.String, long, float, java.util.concurrent.Executor, android.location.LocationListener) (LocationManager.java:1186)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.location.LocationManager.requestLocationUpdates(java.lang.String, long, float, android.location.LocationListener, android.os.Looper) (LocationManager.java:1151)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean com.microsoft.mixedreality.spatialservices.coarserelocalization.AndroidGeoLocationWatcher.resume() (AndroidGeoLocationWatcher.java:61)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void com.unity3d.player.UnityPlayer.nativeFocusChanged(boolean) ((null):-2)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void com.unity3d.player.UnityPlayer.access$000(com.unity3d.player.UnityPlayer, boolean) ((null):-1)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void com.unity3d.player.UnityPlayer$e$1.a() ((null):-1)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean com.unity3d.player.UnityPlayer$e$1.handleMessage(android.os.Message) ((null):-1)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Handler.dispatchMessage(android.os.Message) (Handler.java:102)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at boolean android.os.Looper.loopOnce(android.os.Looper, long, int) (Looper.java:201)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void android.os.Looper.loop() (Looper.java:288)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]   at void com.unity3d.player.UnityPlayer$e.run() ((null):-1)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594] Caused by: android.os.RemoteException: Remote stack trace:
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]    at com.android.internal.util.Preconditions.checkArgument(Preconditions.java:59)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]    at com.android.server.location.LocationManagerService.registerLocationListener(LocationManagerService.java:746)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]    at android.location.ILocationManager$Stub.onTransact(ILocationManager.java:585)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]    at android.os.Binder.execTransactInternal(Binder.java:1179)
2022-12-06 19:34:26.378 13754 13775 Fatal s.unity.helloa java_vm_ext.cc:594]    at android.os.Binder.execTransact(Binder.java:1143)
thejoshuafazel commented 1 year ago

Hey @darax,

I think I may have found the problem.

Before going to the problem here is what I did to debug the issue:

  1. Created a "Hello World" App in android studio

  2. Wrote the following code to see the output:

    fun getAllProviders() {

    val context: Context = this.baseContext
    // Get a reference to the location manager
    val locationManager = context.getSystemService(Context.LOCATION_SERVICE) as LocationManager

    // Get a list of all providers
    val providers = locationManager.getAllProviders()

    Log.e(null,locationManager.isLocationEnabled.toString())
    // Print the list of providers
    for (provider in providers) {
        Log.e(null,provider)
    }
}
  3. The output for the providers were: 'passive', 'fused' and 'network'

After this went digging through the ASA sdk code.

  1. Opened up com.microsoft.azure.spatial-anchors-sdk.android-2.13.0.tgz
  2. Got the spatialanchors_ndk file inside plugins
  3. Opened the spatialanchors_ndk file, from inside of it I got the coarserelocalization-3.0.0-release0104 file.
  4. Inside the AndroidGeoLocationWatcher class, I see request location updates using the 'gps' and 'network' providers. I dont think there is a case to handle the FusedLocationProvider which is what my device is using.

Let me know if there is a work around for this, or when this will be supported in ASA. We really want to use ASA for our project, hope this is fixed soon.

darax commented 1 year ago

Nice find! Out of curiosity can you modify the java to request the FUSED_PROVIDER instead of the GPS_PROVIDER?

thejoshuafazel commented 1 year ago

@darax

I'm not able to recompile the coarserelocalization-3.0.0-release0104 after de-compiling it. Can you try it and let me know?

thejoshuafazel commented 1 year ago

@darax

Could you perhaps try the change you suggested and recompile the com.microsoft.azure.spatial-anchors-sdk.android-2.13.0.tgz and send it to me? I can test it out

darax commented 1 year ago

@thejoshuafazel I don't have a way to share an internal build (policy, not technology). I spoke with the team, and the fused location provider was used previously and the results from the network provider was providing better results than the fused location provider. Clearly asking for the "gps" provider when it is not on the system is a bug, and I'm going to work on a fix. Hard to anticipate when the next release will be, but I'd expect January or February. Once I have the fix in an internal build I'll also try to get this out as a 'hotfix'. This might take me some time logistically. The fix is straightforward. The coarse reloc packages are owned by a sister team, and this will be my first time contributing code to their repository.

thejoshuafazel commented 1 year ago

@darax

I understand that sharing the build might not be possible at this moment but I do have a few questions based on your response.

  1. From what I understand the fused provider automatically selects which provider is best and supplies the location data. Ideally, this should mean that the fused provider would provide the best possible information but that is not the case according to the team you spoke with. Do you know the reason why? How many devices was this tested on and was it in an indoor or outdoor environment? We are planning to use it in an outdoor environment where there might not be that many wifi networks.
  2. Estimating when this will be a part of the release is hard, but when can you say for sure when this will be released? The bug already pushed back our project by 2 weeks, would be good to get a finalised date.
  3. Related to the second point, I dont see a change-log page. When are new versions released and how do developers get to know when it is released? The only way I found was to compare the latest download version available from the web download page with the version I have locally but still dont know what were the changes between the two
darax commented 1 year ago

fixed in 2.14.1