search

m4xw / rosettadrone_mini2

The Rosetta Drone 2 Project. Enable Mavlink and H.264 on DJI drones. Supports Mini 2 with Virt Stick Litchi CSV and DJI Waypoint class interpreters
BSD 3-Clause "New" or "Revised" License
28 stars 7 forks source link

app crashes after pressing "OPEN" #14

Open patogit opened 2 years ago

patogit commented 2 years ago

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. turn on the drone, the controller, and open Rosetta Drone on Sony Xperia XZ2 with Android 10 with WiFi connected to internet
  2. little messages appear in this order: "registering, pls wait..." then "Disconnected" then "Register Success" and OPEN turns blue.
  3. Tap "OPEN".
  4. App switches to landscape mode and instantly crashes.

This happens every single time.

patogit commented 2 years ago

I put the sdk.js file in the root of the SD card, since libgadget.config.so says /sdcard/sdk.js. Maybe I'm missing a detail. The rear LED still works.

patogit commented 2 years ago

Restested after restarting the phone, with and without WiFi, with and without DJI Fly and QGroundCtl. Rosetta Drone failed every time.

m4xw commented 2 years ago

Try to manually accept all app permissions on the Info page before starting it

patogit commented 2 years ago

I went to app info > Permissions, and the only two missing were camera and microphone. I granted both. The app still crashes just like before.

patogit commented 2 years ago

Could this be a problem with the sdk.js file name or placement? Or could it be a problem with using Drone Hacks to remove limits? Are there logs I can look for that would give more explanation of what's happening?

m4xw commented 2 years ago

Could this be a problem with the sdk.js file name or placement? Or could it be a problem with using Drone Hacks to remove limits? Are there logs I can look for that would give more explanation of what's happening?

It should not crash in either case. is your phone x64?

patogit commented 2 years ago

The phone is 64-bit ARM. Just to be double sure, I installed a benchmark app which confirmed this. It's a Sony Xperia XZ2, with the same chipset as the Google Pixel 3.

patogit commented 2 years ago

The readme says that Android 5.1 and up are supported. I'm building for Android 11, and that's the only SDK version installed in Android Studio. Could that be the problem?

patogit commented 2 years ago

I've now tried the app on the Sony XZ2 flashed with iodéOS (a derivative of LineageOS based on Android 11) and on Pixel 2 with stock ROM (Android 11 from October 2020). They all fail the same.

patogit commented 2 years ago

On the Pixel 2 I just made a logcat of the failure. You can see the filtered version of it with only the lines that include rogue here: https://pastebin.com/raw/vB9zKHV3 . There's the WIN DEATH at 23:50:12.816. The complete version of that time period is at https://pastebin.com/raw/jSN4aXgF. Please let me know if there's anything else I can do to find the cause of this error.

m4xw commented 2 years ago

Did u update the sdk or smth?

patogit commented 2 years ago

Did u update the sdk or smth?

I don't know. This is the first time I've built/compiled an Android app. I followed the instructions here and in Android Studio as well as I could. If there's a requirement to use a specific version of the SDK, or downgrade something, how about we put that in the readme? If you tell me, I can add it.

I'm willing to put some money and/or time into getting this working. If you can tell me where my time or money could go in order to get this working, I'd appreciate it.

linus131313 commented 2 years ago

Any updates on this? I'm having the same issue with my Mini2 and Samsung Galaxy S20...

m4xw commented 2 years ago

Any updates on this? I'm having the same issue with my Mini2 and Samsung Galaxy S20...

Will need more changes than I have time for rn for Android 12..

luandro commented 2 years ago

Here's some more errors from logcat:

05-31 18:03:41.166 13605 13605 W SnapshotHandler: java.io.FileNotFoundException: /data/user/0/sq.rogue.rosettadrone/files/phenotype/shared/com.google.android.gms.maps#sq.rogue.rosettadrone.pb: open failed: ENOENT (No such file or directory)
05-31 18:03:41.166 13605 13605 W SnapshotHandler:   at sq.rogue.rosettadrone.MainActivity.onCreate(MainActivity.java:562)
05-31 18:03:41.870 13605 13605 E AndroidRuntime: Process: sq.rogue.rosettadrone, PID: 13605
05-31 18:03:41.870 13605 13605 E AndroidRuntime: java.lang.RuntimeException: Unable to start activity ComponentInfo{sq.rogue.rosettadrone/sq.rogue.rosettadrone.MainActivity}: java.lang.NullPointerException: Attempt to invoke interface method 'java.lang.Object java.util.List.get(int)' on a null object reference
05-31 18:03:41.870 13605 13605 E AndroidRuntime:    at sq.rogue.rosettadrone.DroneModel.initFlightController(DroneModel.java:279)
05-31 18:03:41.870 13605 13605 E AndroidRuntime:    at sq.rogue.rosettadrone.DroneModel.<init>(DroneModel.java:234)
05-31 18:03:41.870 13605 13605 E AndroidRuntime:    at sq.rogue.rosettadrone.MainActivity.onCreate(MainActivity.java:589)
05-31 18:03:42.106  1330  9305 W ActivityTaskManager:   Force finishing activity sq.rogue.rosettadrone/.MainActivity
05-31 18:03:42.113  1330  9305 W ActivityTaskManager:   Force finishing activity sq.rogue.rosettadrone/.ConnectionActivity
luandro commented 2 years ago

More detailed:

05-31 18:26:56.160 17275 17275 E AndroidRuntime: java.lang.RuntimeException: Unable to start activity ComponentInfo{sq.rogue.rosettadrone/sq.rogue.rosettadrone.MainActivity}: java.lang.NullPointerException: Attempt to invoke interface method 'java.lang.Object java.util.List.get(int)' on a null object reference
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3449)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3601)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:85)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2066)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.os.Handler.dispatchMessage(Handler.java:106)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.os.Looper.loop(Looper.java:223)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.ActivityThread.main(ActivityThread.java:7656)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at java.lang.reflect.Method.invoke(Native Method)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
05-31 18:26:56.160 17275 17275 E AndroidRuntime: Caused by: java.lang.NullPointerException: Attempt to invoke interface method 'java.lang.Object java.util.List.get(int)' on a null object reference
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at sq.rogue.rosettadrone.DroneModel.initFlightController(DroneModel.java:279)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at sq.rogue.rosettadrone.DroneModel.<init>(DroneModel.java:234)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at sq.rogue.rosettadrone.MainActivity.onCreate(MainActivity.java:589)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.Activity.performCreate(Activity.java:8000)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.Activity.performCreate(Activity.java:7984)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1309)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3422)
05-31 18:26:56.160 17275 17275 E AndroidRuntime:    ... 11 more
05-31 18:26:56.184 17275 17275 E DJICrashHandler: uncaughtException 3
05-31 18:26:56.184 17275 17454 E UsbAccessoryService: final destroy() 71
05-31 18:26:56.285 17275 17454 E UsbAccessoryService: final destroy() 72
05-31 18:26:56.285 17275 17454 E wm220   : ****resetSwitchFromWifiFlag
05-31 18:26:56.286 17275 17454 E DJIUsbAccessoryReceiver: receiver destroySession
05-31 18:26:56.287 17275 17454 E UsbAccessoryService: final destroy() 73
05-31 18:26:56.287 17275 17454 E UsbAccessoryService: final destroy() 75
05-31 18:26:56.287 17275 17454 E UsbAccessoryService: final destroy() 76
05-31 18:26:56.387  1330  2648 W ActivityTaskManager:   Force finishing activity sq.rogue.rosettadrone/.MainActivity
m4xw commented 2 years ago

aircraft.getGimbals fails?

What if u remove the .get(0) here https://github.com/m4xw/rosettadrone_mini2/blob/develop/app/src/main/java/sq/rogue/rosettadrone/DroneModel.java#L279

and here https://github.com/m4xw/rosettadrone_mini2/blob/develop/app/src/main/java/sq/rogue/rosettadrone/DroneModel.java#L1953 mGimbal.rotate -> mGimbal.get(0).rotate

m4xw commented 2 years ago

fwiw i sometimes had it randomly crash on start too so i cant exclude some init race condition... that would be easy fix tho. happens so rare on my end tho, usually if i hurry too much pressing open while dev :P

patogit commented 2 years ago

The master branch of the upstream repo is now working for me, at least to some degree. https://github.com/The1only/rosettadrone/issues/99#issuecomment-1142953215

patogit commented 2 years ago

I'm here working on it with @luandro

patogit commented 2 years ago

This is in Android 11 on a Pixel 2.

m4xw commented 2 years ago

waypoint mission arent supported at all upstream (mini 2). fwiw i stubbed the automated wp generation so i feed it data via corridor scan proper way would be to generate optimized intermediates, existing algo just didnt work nicely in my tests Anyway given the crash report, suggested changes above should work Will fix it tomorrow Note that gimbal rotation on my fork might not work without a lazy init of it

m4xw commented 2 years ago

(unless dji sdk has proper missions for mini 2? that would be news to me, i still didnt bother checking it)

m4xw commented 2 years ago

If u test now i can give some feedback fwiw, gonna nope out in 5min, 3am here

patogit commented 2 years ago

remove the .get(0) here https://github.com/m4xw/rosettadrone_mini2/blob/develop/app/src/main/java/sq/rogue/rosettadrone/DroneModel.java#L279

and here https://github.com/m4xw/rosettadrone_mini2/blob/develop/app/src/main/java/sq/rogue/rosettadrone/DroneModel.java#L1953 mGimbal.rotate -> mGimbal.get(0).rotate

I made those changes, so now:

279: mGimbal = aircraft.getGimbals();
1953: mGimbal.get(0).rotate(m_ServoSet, djiError -> {

I just tried to build that. Build fails. The result is:

m4xw commented 2 years ago

just fix the type

patogit commented 2 years ago

just fix the type

What would the fixed line be?

m4xw commented 2 years ago

dji.sdk.gimbal.Gimbal where mGimbal member variable is declared. Cant check rn

patogit commented 2 years ago

image I see that I'm using DJI SDK v4.14-trial1. Not sure where that's defined, or how to update to v4.16. I pushed Alt-Enter, and got this: image Then I clicked on Cast, then build, and now that type error is gone but the get error from L1953 remains.

m4xw commented 2 years ago

pushed untested to misc branch, bumped sdk for u too. not sure if functional but compiles!

patogit commented 2 years ago

pushed untested to misc branch, bumped sdk for u too. not sure if functional but compiles!

Thanks! Are the following steps correct for me to follow:

patogit commented 2 years ago

I'm on the build step of that sequence, and Android Studio is downloading SDK v4.15.

m4xw commented 2 years ago

*uxsdk u mean

pushed untested to misc branch, bumped sdk for u too. not sure if functional but compiles!

Thanks! Are the following steps correct for me to follow:

* git fetch

* git checkout misc

* go to Android studio and build

* test with drone

pretty much pull, sync gradle and build. assuming u built it before, which u did.

patogit commented 2 years ago

I'm not sure how to sync Gradle, so I found a context menu to "Reload Gradle Project" and then "Refresh Gradle Dependencies".

patogit commented 2 years ago

Now I found a button for "Sync Project with Gradle Files" and clicked that.

patogit commented 2 years ago

Ok, it worked more than before!!

The DJI SDK has been updated to 4.16.1, confirmed by the tooltip in Android Studio and in the app running on the phone.

The app builds, and runs on the phone. Sometimes it shows the firmware version of the drone, sometimes it says "Model not available". The difference seems to be that having RD open and connected to the controller before turning on the drone, I get the firmware version. If I turn on the drone first and then open RD, I get "Model not available".

It gives a few little disappearing messages about connecting, and then shows connected to DJI MINI 2. I tap OPEN when it turns blue, and nothing happens. No fail, no nothing. (On the upstream version I used yesterday, tapping OPEN launched QGC.) I open QGC myself. It shows disconnected, and then after doing the whole sequence of apps and turning the drone off and on a few times, QGC shows connected!! And there's a button to plan the mission and fly it, which is missing in the upstream RD I used yesterday. So I plan a survey mission, push the button to upload it to the drone, it appears to upload. I use the DJI hardware controller to take off and go up to 20m. The mission is set for 50m. I push a button and slide a slider to start flying the mission, but the drone doesn't move. (At least I've got the mission lines drawn on the map, with an accurate live position of the phone and the drone, and I know what height I planned the mission for, so at least I could use the lines as a guide to fly manually. A step closer to achieving https://github.com/m4xw/rosettadrone_mini2/issues/17 ). I fly the drone 20 meters horizontally, and then in QGC push the "return home" button, and it works! The drone comes back, and comes all the way down to 3m AGL.

Now I try all of this with WiFi on the cell phone turned off, airplane mode activated, because when I want to use this in the field, I'm not going to have internet. This time QGC starts to connect, and then says it's downloading (bar across the top of the screen, no explanation of what's downloading). Then a dialog pops up, " Vehicle 1 did not respond to the request for parameters. This will cause QGC to be unable to display its full user interface." I tap ok, and now QGC says Communication Lost. (EDIT: Why does this need internet to fly? With DJI Fly, an internet connection is not necessary. Also, what's QGC downloading?)

Compared to upstream, QGC is showing a different screen. The place that said DJI and GCS with room for green dots, that's not present this time.

Notes:

So, both this misc branch and upstream work, but give different interfaces in QGC. Both have elements that the other doesn't have. Neither can fly a survey mission (yet), but at least this misc version can bring the drone home.

patogit commented 2 years ago

I'm really excited with so much progress!!

stefansommer commented 1 year ago

@m4xw @patogit Did you get further with this? I'm on android 9 (galaxy note 8) with mini 2. On the upstream branch, I get the crash right after pressing open (the main screen displays briefly). On the misc branch with the latest updates from June, it doesn't get past the main screen (pressing open switches to landscape mode and then immediately back to main screen) I can get a logcat but so far haven't been able to identify the cause of it crashing.

m4xw commented 1 year ago

@m4xw @patogit Did you get further with this? I'm on android 9 (galaxy note 8) with mini 2. On the upstream branch, I get the crash right after pressing open (the main screen displays briefly). On the misc branch with the latest updates from June, it doesn't get past the main screen (pressing open switches to landscape mode and then immediately back to main screen) I can get a logcat but so far haven't been able to identify the cause of it crashing.

Log will be required. Last issue I debugged here were some races.

On Samsung Android forks you can also try to disable power saving and what not, that caused troubles for me before

stefansommer commented 1 year ago

Thank you. Before I post the full (rather long) log, could it be this exception? I.e. related to the google maps api?

java.util.concurrent.ExecutionException: java.lang.SecurityException: GoogleCertificatesRslt: not allowed: pkg=sq.rogue.rosettadrone, sha256=[5a0a7116118b241372ffd4037ab7e9d08b7dc772a008189ff77d69195eba1b03], atk=false, ver=224417028.true (go/gsrlt)
    at aji.s(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):3)
    at aji.get(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):2)
    at aks.g(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):2)
    at yr.d(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):1)
    at yt.run(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):0)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:459)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:764)
 Caused by: java.lang.SecurityException: GoogleCertificatesRslt: not allowed: pkg=sq.rogue.rosettadrone, sha256=[5a0a7116118b241372ffd4037ab7e9d08b7dc772a008189ff77d69195eba1b03], atk=false, ver=224417028.true (go/gsrlt)
    at android.os.Parcel.createException(Parcel.java:1966)
    at android.os.Parcel.readException(Parcel.java:1934)
    at android.os.Parcel.readException(Parcel.java:1884)
    at fd.c(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):2)
    at ru.a(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):4)
    at jj.e(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):2)
    at ki.t(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):2)
    at ki.u(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):3)
    at ki.e(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):2)
    at km.handleMessage(:com.google.android.gms.dynamite_mapsdynamite@224417028@22.44.17 (100400-0):69)
    at android.os.Handler.dispatchMessage(Handler.java:102)
    at android.os.Looper.loop(Looper.java:214)
    at android.os.HandlerThread.run(HandlerThread.java:65)
m4xw commented 1 year ago

Depends if its handled. Just post the full thing or send it to m4x at m4xw (.) net I obviously have the api key filled here

stefansommer commented 1 year ago

I got it to work by switching the rotation setting from 'portrait' to 'auto rotate': The app crashes with portrait, it does not with auto rotate.

m4xw commented 1 year ago

I got it to work by switching the rotation setting from 'portrait' to 'auto rotate': The app crashes with portrait, it does not with auto rotate.

Oh boy. I could swear i tried it before and it still would force rotation (and work)

m4xw commented 1 year ago

Also misc branch changes and sdk update was merged to develop now. I stubbed the frida integration for now. Need to update the gadget shared library for android 13. Fwiw (with android 13) there is now a new issue that i need to set video rate to like 5 and then disable video feed, return out of the menu, enable it again, to get video output in rosetta. I wonder if this is due to a sdk change now... Since they fixed their video decoder tho, I might just gonna bring back ther old video decoding for lower latency and ffmpeg decoding for compat and cross-streaming to QGC

stefansommer commented 1 year ago

That's great. Nice with the sdk update.

Removing this.setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE); in MainActivity.java and changing <activity android:name="sq.rogue.rosettadrone.MainActivity"> to <activity android:name="sq.rogue.rosettadrone.MainActivity" android:screenOrientation="landscape"> in AndroidManifest.xml fixed the screen rotation issue for me.

I didn't manage to get video to work (I tried following your workaround steps). But if you anyway plan to change the decoding, perhaps it is not worth it looking too much into it now?

m4xw commented 1 year ago

That's great. Nice with the sdk update.

Removing this.setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE); in MainActivity.java and changing <activity android:name="sq.rogue.rosettadrone.MainActivity"> to <activity android:name="sq.rogue.rosettadrone.MainActivity" android:screenOrientation="landscape"> in AndroidManifest.xml fixed the screen rotation issue for me.

I didn't manage to get video to work (I tried following your workaround steps). But if you anyway plan to change the decoding, perhaps it is not worth it looking too much into it now?

Thanks I will take a look.

Regarding video, thats weird. You can try unstubbing this part https://github.com/m4xw/rosettadrone_mini2/blob/develop/app/src/main/java/sq/rogue/rosettadrone/DroneModel.java#L827-L830

That had the same effect (but it tries to change it too often which causes video reinits which causes it to be very flakey.)

Weirdly enough this only happens on the first time, later app opens stay fixed. Another person reported it would work after a phone reboot as well... Didnt get to debug it since i made the changes.

May I ask what you use case is gonna be?

stefansommer commented 1 year ago

I couldn't make it work on my android 13 phone, neither changing the settings, rebooting or uncommenting the lines in DroneModel.java helped. However, it works on my old phone. The video quality is quite low and delayed though.

I aim to train and run deep learning models on the video + telemetry streams with some of my students, either offline by saving the streams (both video and telemetry) and use it for training. If possible, then subsequently running the models real time if the video stream quality is good enough.

If you do bring back the old decoding as you sketched and if it has lower latency, I would be extremely thankful. I'm not so versed in video decoding, so it is a bit hard for me to debug the problems directly.

m4xw commented 1 year ago

I couldn't make it work on my android 13 phone, neither changing the settings, rebooting or uncommenting the lines in DroneModel.java helped. However, it works on my old phone. The video quality is quite low and delayed though.

I aim to train and run deep learning models on the video + telemetry streams with some of my students, either offline by saving the streams (both video and telemetry) and use it for training. If possible, then subsequently running the models real time if the video stream quality is good enough.

If you do bring back the old decoding as you sketched and if it has lower latency, I would be extremely thankful. I'm not so versed in video decoding, so it is a bit hard for me to debug the problems directly.

Nice, I planned to do simliar stuff myself, so lmk if u need help. Planning to hande it from the GC side? Fwiw gonna whip up a branch for you real quick

m4xw commented 1 year ago

Quick and dirty https://github.com/m4xw/rosettadrone_mini2/commit/aefee606bfb88130740e67fcc279460633d283ef

m4xw commented 1 year ago

Note: this wont stream to QGC now tho, but just to give u a idea