steve1316 / granblue-automation-android

Educational application written in Kotlin aimed at automating user-defined workflows for the mobile game, "Granblue Fantasy", using MediaProjection, AccessibilityService, and OpenCV.
63 stars 11 forks source link

Button not found #5

Closed kevin01523 closed 2 years ago

kevin01523 commented 3 years ago

It seems it has troubled finding some buttons like full auto but it works with semi on normal special quest like deluge trial

Screenshot_20210826-131510 Screenshot_20210826-132537

I'm on android 10 I'm using skyleap browser I tried both standard and lite graphical settings

I'll try on chrome later

steve1316 commented 3 years ago

I also am on Android 10 on a Galaxy S20+ phone. I primarily play on Chrome on standard graphics but I just tested with Skyleap on both standard and lite and everything works fine.

User experience will differ as I did not account for all possible screen sizes but I try to where I can. I recommend using the Python version if the bot continues to not work smoothly on your Android device. Or you can tell me what Android device you are using and I will see if I can support your screen size.

kevin01523 commented 3 years ago

2280x1080 or 2192x1080 density is 392 I'm on redmi note 6 pro tulip Well you can check the image I posted for its dimensions depth etc since its a full screenshot of my phone screen Screenshot_20210826-164822

Hmm it seems that xeno also fails it cannot even get the right banner.I'm testing it to do the special quest but gets. Stuck in home or to the preview if the next event

steve1316 commented 3 years ago

We have the same width of 1080 pixels and height does not factor into the bot's performance. We do have differing densities but my Tab S7 with density of 300 works as well. Unfortunately there is not much I can do to accommodate your device.

kevin01523 commented 3 years ago

Hmm sometimes it can work though if its simple quest and its on semi.just asking it works on tab s7 on both orientation?vertical and horizontal

The only fix I can think of is an update into the images to match also lower or higher the accuracy of the OCR/image recognition like what i did on sikulix before when I'm using it on kancolle

steve1316 commented 3 years ago

For the Tab S7, it would only work in a split screen configuration with the browser on the left side, and Desktop Mode turned off for portrait and turned on for landscape. And even then, it would often get stuck so Tab S7 support remains experimental for now.

I will get around to creating a setting soon that will allow you to adjust the confidence that the OCR determines to be a match. Currently it is set to be at least 80% or more.

kevin01523 commented 3 years ago

i guess an option to add own images would be nice where the app would read from which coild be enabled as a debug feature eith a toggle also a log export would be nice i experience lots of java errors like outof bounds but i cant see the error details as the apps do crash.

npnp take your time

steve1316 commented 3 years ago

Just updated the bot to v2.5.0. Added 2 new options for setting the confidence levels for both single and multi match CV operations. Could not implement the option to replace images with your own without a significant overhaul of the bot.

I might come back to this later so hopefully setting your own confidence levels will solve your incompatibility issue. Let me know how it goes!

steve1316 commented 3 years ago

Forgot to mention, logs are saved in the /logs/ folder in the application folder inside your internal storage's /Android/data/.

However, logs do not print the segfault errors. These ones causes the app to crash completely and leaves no stacktrace, which makes it so that I cannot find what is the cause. They typically happen when you stop the bot while it is in the middle of doing something like searching for summons and even then, it segfaults at random.

kevin01523 commented 3 years ago

OK I'll test it for a few hours to few days I'll update you on my findings

kevin01523 commented 3 years ago

Seems that it manages to find the full auto button now if I set both confidence to 70. I haven't tested it on other quest except deluge trial.I also noticed it has no recovery in the event it got stuck during the quest I guess doing a reload for recovery is helpful

steve1316 commented 3 years ago

By recovery, do you mean AP/EP recovery? If so, turn on all 3 of these settings in the Misc section:

chrome_XzMgrD9m87

and then enable this in the Settings page of the bot:

studio64_HMuF4p3PIQ

steve1316 commented 3 years ago

Just updated the bot to v2.5.1 and included a new setting that will allow you to set your own image scale. You can find out your scale by taking a screenshot directly on your device and then pick a easily recognizable image asset from the /images/folder in this repository and scaling it until it relatively matches the size of the asset in your screenshot. Whatever percentage you scaled the image asset by will be the number that will go into this new setting.

Let me know how it goes!

kevin01523 commented 3 years ago

Ah what I meant for recovery is when the game itself get stuck while loading due to connection issues the bot can click the reload button to recover(well that's a diff issue altogether) ill test the new version it later

Edit:where is the images folder?i know about the webp files but I guess normal users can be confused where to look for it

steve1316 commented 3 years ago

Ah, sorry about that. It is at app/src/main/assets/buttons/ in this GitHub repository.

I may consider adding an option to have the bot reload the page if it doesn't arrive at the correct location but I will need to think more on it.

kevin01523 commented 3 years ago

Screenshot_20210907-235235 Screenshot_20210908-000518 Screenshot_20210908-000533 i guess still can't find full auto at confidence 80 also its stops if the test for home screen is enabled?

steve1316 commented 3 years ago

It does not conduct the next text if the current scale/confidence settings already found the Home screen. It needs to fail that first before doing the test to find a scale that works for you. You can force it to do that by setting your scale something low like 0.20. It will then start the next test by starting from 0.10 scale and incrementing by 0.01 until it finds a match for the HOME button.

When it can't find the full auto button, is it already at the Combat screen?

In addition, you can find the latest logs with all of these logging messages in the /logs/ folder under the /Android/data/ folder in your internal storage.

kevin01523 commented 3 years ago

it fails to find the full auto if confidence is 77-100 but finds it on 70 i guess ill test again

steve1316 commented 3 years ago

Oh my bad, I did not realize you had 1080p screen. The bot supports 1080p by default but it accidentally sees your device as not having 1080p screen. This will be fixed in next update later today in v2.5.3.

kevin01523 commented 3 years ago

Oh my bad, I did not realize you had 1080p screen. The bot supports 1080p by default but it accidentally sees your device as not having 1080p screen. This will be fixed in next update later today in v2.5.3.

i did mention it on the previous comments oh well i hope it does get fixed

kevin01523 commented 3 years ago

Screenshot_20210908-220140 Screenshot_20210908-220314

I guess at scale 1.0 and confidence 80 it still cannot find full auto atleast less crashes now and can find quest buttons etc

log @ 2021-09-08 22:01:44.txt log @ 2021-09-08 21:59:29.txt log @ 2021-09-08 21:55:37.txt

kevin01523 commented 2 years ago

More logs It didn't push the attack button dunno why log @ 2021-09-08 22:36:31.txt log @ 2021-09-08 22:31:42.txt

log @ 2021-09-08 22:45:25.txt