Add checking to make sure cards and/or skills are properly selected

Preparation

[X] No one has asked for this feature before in other issues (even the closed ones)

Describe the feature

FGO JP is very laggy on many devices, especially Bluestacks. 90++ nodes are very unforgiving if any piece of the rotation is missed, often leading to death and/or 40 missed AP (as I just experienced). FGA does not handle lag very gracefully. I'm requesting that this program waits for visual feedback that a skill had been used or a card had been selected before moving on to the next piece of the rotation.

I do not think this should be complicated. Currently, you already know what areas skill icons and card icons are at, via your image scans. If you poll a pixel's color (not even image recognition, just plain color) at basically any location within those areas before the button press, and then make sure it's basically the same color but darker after the press, that serves as confirmation that a skill was properly used. Cards are a bit more complicated since they float up and down during selection, but I feel that there should be a way to scan for the (1st) (2nd) and (3rd) after selection, though I understand if this is hard to implement. Thanks.

There's a fine tune menu where you can increase and decrease wait times for when skills are use. You can use that if the game is laggy for you

The game is not consistently laggy, though. Sometimes the game isn't lagging at all. I have watched it fail to use Oberon skill 3 on someone when there was no visible lag, fail the kill miserably, and then the next cycle it worked just as intended. I've come back to it mid-card selection, and it's just frozen because one of its card clicks did not register. Adding a large delay to every click would just meaninglessly delay the script. It also might not even solve the issue, because when lag spikes do happen, they might be longer than most of the delays that I could put in to begin with, or the lag itself might happen during the interaction.

When making a state machine like this, I think it's pretty standard to look for feedback that something has happened, rather than just assuming that it did.

I also know that I am not the only one having this issue; browsing around forums there are reports of people sometimes just telling the program to do every action twice, just in case. If this is difficult to implement, I understand.

One thing I can think of that makes it difficult is that currently, iirc FGA's image recognition matches in greyscale. We don't know the pixel color in the first place so polling pixel color can't be done with the current image recognition.

However, what if we:

Test skills are properly selected: before clicking on a skill, we can take a snapshot of the icon area. If after a press and delay, the icon area changes, then it means the skill is properly clicked. If the icon image matching did not change, then either it is not properly selected, or it has already been clicked and in cooldown.

Test cards are properly selected: unless you need to know if each of the card is selected (which given the current implementation of spamming, you wouldn't know anyways), there's no need to try to look for the (1st), (2nd), (3rd) icon - just looking for the absence of the "go back" button on the bottom right cornet will do. No back button = cards are selected, yes back button = cards are not properly selected.

Just a thought. I'm still not sure how to test if skills are properly selected but this might be a starting point.

Could there at least be checking to make sure that NPs go off when they should, and then abort and wait for user input when it fails? Four runs went fine today, and then the fifth one failed and I lost another 40 AP because I was at round 3, my NP clearly didn't go off at some point in the past, and then my DPS was asleep in front of an 800k HP saber after essentially having done nothing, at turn 5. I think, at least, it is obvious where there SHOULD be an NP attempt, but there is not one. If it's having to dial in an extra face card on a rotation, something is wrong. Unfortunately "just sit there and card it down" is not an acceptable course of action with these sorts of HP bars, and my AP is just getting wasted too regularly...

This is a pretty complicated topic when you look outside of your own FGA usage.

Let's split it into the 2 topics of NP vs Skill usage.

For NP usage, the main issue of not hitting NP cards is usually that the NP was not fully charged or because FGO lagged and the NP card wasn't clickable yet. The first case is usually caused either by RNG or missed skills. The second issue can be mitigated by increasing the "Wait before cards" value in the Fine Tune settings.

But there are also users who use NP spam or manually input NP spam like behavior and write command lists ending with ,456,456,456,456. In those cases, the absence of NP cards is normal. In those cases, FGA tries to select 3 NPs and afterwards selects 3 normal cards in case none of the NPs are available.

Therefore it's not possible to implement a "check if the selected NP was actually used". Even if we ignore those cases, the best way to start the card selection again would be to press the Back button, then the Attack button again and then the lag may occur again. And if the NP bar isn't full yet, this would result in an endless loop because FGA cannot click a non-existent NP card.

Regarding Skill usage, FGA already keeps track of skill usage for the skill spam feature. What you're saying is basically "press the skill and then immediately check if the icon changed".

The most probable situations for missed skills are probably FGO lagging when selecting a target or incorrect command lists.

The first case can be dealt with by increasing the "Skill Delay" wait value in Fine-Tune.

In the second case, it could be that the user selected a skill which is still in cooldown or not specifying a target for targetable skills. FGA would enter a loop of picking the skill, closing the "skill in cooldown" or target selection window and then detect that the skill was not used, therefore clicking it again.

Before making any changes, I would like to see videos of FGA messing up after "Wait before cards" and "Skill Delay" were increased. Otherwise, it's very probable that users complain about their special use cases not working anymore.

This is a pretty complicated topic when you look outside of your own FGA usage.

Let's split it into the 2 topics of NP vs Skill usage.

For NP usage, the main issue of not hitting NP cards is usually that the NP was not fully charged or because FGO lagged and the NP card wasn't clickable yet. The first case is usually caused either by RNG or missed skills. The second issue can be mitigated by increasing the "Wait before cards" value in the Fine Tune settings.

Even if one did increase all sorts of wait times and timers, they're just slowing down the all farming of an event with what can at best be considered a bandaid. And if the lag lasts long enough, it won't even solve the issue.

But there are also users who use NP spam or manually input NP spam like behavior and write command lists ending with ,456,456,456,456. In those cases, the absence of NP cards is normal. In those cases, FGA tries to select 3 NPs and afterwards selects 3 normal cards in case none of the NPs are available.

Do you consider this to be most common use, while my usage, which is just do one NP per wave, and then card down if the enemy is still alive, to be a niche use of it? Because I believe mine is easily the simplest and most common use case of your program. The vast majority of farmers just want to do one NP per wave, or even more simply, just do one NP per turn, regardless of wave. In a farming node, NP spamming and card spamming should generally not happen until at best the very end, especially in current 90++ content, and it is going to be executed on an enemy with barely any HP left. If it isn't, the frontline is more or less likely to be dead anyway, because the enemies do too much damage in these 90++ nodes to try to do blind NP and/or card spam in order to save a critically failed rotation.

Therefore it's not possible to implement a "check if the selected NP was actually used". Even if we ignore those cases, the best way to start the card selection again would be to press the Back button, then the Attack button again and then the lag may occur again. And if the NP bar isn't full yet, this would result in an endless loop because FGA cannot click a non-existent NP card.

You currently have logic that can check and prioritize normal cards. I know this because there is an entire card prioritization menu in your program, and it does work because I've used it. Why is it not possible to detect whether a certain servant's card is simply in the top half of the screen? That should constitute an NP being ready on demand for a certain character. Or, barring that, whether a certain NP slot is filled in at all

Regarding Skill usage, FGA already keeps track of skill usage for the skill spam feature. What you're saying is basically "press the skill and then immediately check if the icon changed".

The most probable situations for missed skills are probably FGO lagging when selecting a target or incorrect command lists.

The first case can be dealt with by increasing the "Skill Delay" wait value in Fine-Tune.

In the second case, it could be that the user selected a skill which is still in cooldown or not specifying a target for targetable skills. FGA would enter a loop of picking the skill, closing the "skill in cooldown" or target selection window and then detect that the skill was not used, therefore clicking it again.

I'm asking if the program can keep VISUAL track of its usage, yes. Just hitting the correct area(s) and assuming a skill has been used, without feedback, is a bit rudimentary, imho.

You can (considering you click on any swapouts via plugsuit already, likely for card prioritization purposes) record all of the skill icon conditions when a servant is first issued onto the field, rather than mid-selection. Barring strange cases like a user erroneously using Arash or Habetrot to die into a character that was swapped out into slot 4 and then expecting their skills to be up again (which should be handled much like any other edge case: tell the user they messed up and abort execution), any servant coming in should have all skills ready. Those icons can be recorded during their first load. That means if a skill was used successfully, there should be a change in the icon for that servant.

Unlike Granblue Fantasy, FGO does not have any cases where there is a visual skill lockout until a certain turn number. All icons simply show up as available on turn 1, every time.

All other cases can (and imo should) be dealt with by detecting that there is a loop/error/etc and telling the user that there might be something wrong with their command list. Which would be preferable to what it currently does... which is just rolling with erroneous logic and causing the frontline to die and waste AP and/or CS. At the very least, perhaps a toggle in the program to tell it to immediately abort, should anything unexpected happen (unexpected meaning... are we on the next wave when they said we would be? Etc).

Before making any changes, I would like to see videos of FGA messing up after "Wait before cards" and "Skill Delay" were increased. Otherwise, it's very probable that users complain about their special use cases not working anymore.

This isn't really a special use case. I don't think I'm doing anything special by using one NP per turn. These are just 90++ farming nodes. They're unforgiving. They're in the JP version now. The JP version also is running on an engine that does not play well with Android 12. The English JP farming community may be a minority at the moment, but you might encounter more of these complaints later anyway, after the EN/Global community encounters these nodes and/or advances down a certain number of patch notes themselves.

I can try to record it, but the action of trying to record it might cause other issues, because recording on top of your program and then FGO also running, is a bit taxing for many devices... that are already having lag spikes. If you know of any forums you frequent, you could just try going to Google and then putting in "site:\<your forum> FGA lag" or something similar. Most users are likely just dealing with these issues, but it is becoming harder when these nodes get more unforgiving. If it's too much trouble and/or you don't feel you have the time for it, as a fellow developer I understand. I will see if I can muck around in your code sometime, if I get time, and fix my issues myself, although I have never done this language. Thank you for responding.

I would very much like if FGA was more freeze/lagspike-resistant (even if it's optional). On my device the game runs fine most of the time, but can randomly lag/freeze for any time from 1-2 seconds up to 10-15 seconds. So increasing the delays is not an acceptable solution (it'd slow down 90% of farming by x5 or more while still not guaranteeing the resilience to a bad freeze).

Finding a way for FGA to figure out the actual state the game is in (i.e. we opened a skill menu and pressed on OK - easy to check, then verify that we are not still on that menu and that the skill is grayed out before proceeding with further steps) would be a huge improvement. At the moment I have to babysit FGA due to the above issues which defeats it's purpose somewhat (it still helps a lot to automate repetitive actions and for many lower level farming nodes even if it messes up it doesn't matter much but not for 40 ap event ones).

Fate-Grand-Automata / FGA

Add checking to make sure cards and/or skills are properly selected #1415

Preparation

Describe the feature