Stuck in serverside conflict loop until google play services app data is cleared

[miracle7]

We are having a strange issue where a serverside conflict is always returned when trying to open the metadata, no matter how many times we resolve it.

It seems to be local to the device as other devices still work as expected when this starts happening.

Cloud save works as expected again if we clear the google play services app data (not just the cache). We are passing the 'ReadNetworkOnly' parameter to the open meta data function, but despite this google play games appears to be doing something offline.

In addition to this strange serverside conflict loop it gets stuck in, if the device has no internet connection and tries to open the metadata, instead of coming back with an error code, it waits a long time before returning the success code and behaving as if it is connected.

Can anyone think of anything I could possibly be doing to cause this to happen? We can not get a reliable repro for this issue and it is causing a lot of problems for our users.

[miracle7]

After repeatedly resolving a conflict from the open metadata callback (for over 15 minutes), google play services crashed, and we found the following in the logs: http://pastebin.com/0WnsVDRk

The lines of most interest are from lines 266-326. They don't really mean too much to us, but maybe it will help someone else to point us in the right direction?

Getting quite desperate on our end, lots of users being unable to play our game and leaving bad reviews as a result, so would appreciate a prompt response :(

[JohnyGQD]

Does this fit your experience, or is it somehow different issue? https://github.com/playgameservices/play-games-plugin-for-unity/issues/1204

We are going for an opt-out button in next update (conflict will give them choice of keeping server copy, which usually ends up in loop, local copy that ends up in either loop or wipe, and finally disabling the cloud completely so the game can at least start) as the customers are pretty furious about this. Knowing it before, we would just make cloud save opt-in in a first place, so it wouldn't affect unexperienced users by any means. Now it's too late and we don't wanna risk data wipe for existing cloud users by turning it off by default.

[LeagueofMonkeys]

Hi JohnyGQD. Yes that does look like the same issue we're having, I work with miracle7. I wonder how widespread this issue is. We'll continue looking into it. Please let us know if you find a cause/solution.

[prodigga]

I'm having this issue too, bump.

[miracle7]

@JohnyGQD For us it doesn't seem to matter whether we choose to resolve using the base or conflicting version of the save, it can get stuck in a loop either way (though not consistently).

Also as far as I know we haven't ever had a save wiped during the conflict resolution process.

For some reason it seems that even if only playing on one device you can end up with conflicts too which doesn't make sense. Was thinking perhaps we were opening the metadata twice or something so google's server treats the one phone as two devices, but investigated that and we definitely only open it once, either when the app opens or after a save (which supposedly closes the metadata).

[JohnyGQD]

During my debugging session, I end up with having ReadNetworkOnly everywhere, saving with automatic resolution as UseUnmerged and loading with manual resolution. I wasn't able to reproduce it with this configuration, but kept the opt-out button anyway. Just in case..

[miracle7]

Okay, we still get stuck in this conflict loop regardless of whether we use ReadNetworkOnly or ReadCacheOrNetwork, so musn't be that.

Not sure what you mean by saving with UseUnmerged as you cannot save with a conflict resolution policy or anything like that, only when you open metadata.

We were able to view our games app data directory on Google Drive, and found there were only 2 conflicting saves rather than a large list of them, which implies that the conflict resolution isn't being applied rather than us generating multiple conflicts and trying to get through them all.

Upon uninstalling and reinstalling our app, the loop also seems to disappear, but not when clearing the apps data on the phone, which makes it seem like google play services is hanging onto some corrupt conflict data that it associates with that install.

We keep the metadata open wherever possible so we can commit an update without having to wait for the metadata to open first, which is impossible to do in the one frame that we are given on application pause/quit.

We noticed that sometimes when you open the metadata as soon as the save callback comes back, the save the server gives us is one behind the one that we just uploaded. Not sure if we should be waiting some amount of time before re-opening or something like that, but we assumed that all serverside operations would be complete if we are receiving that callback.

[MobileMonsters]

It seems like you're having the same problem as we do.

The following happens:

• We call the manual conflict resolution so we can download the saved game.

this.playGamesClient.OpenWithManualConflictResolution( filename, // always the same since we only have one single progress DataSource.ReadNetworkOnly, true, stateListener.OnSavedGameConflict, stateListener.OnSavedGameOpenedForRead);

• Then the plugin calls our OnSavedGameConflict method. We mainly return the original one.

public void OnSavedGameConflict(IConflictResolver resolver, ISavedGameMetadata .....) { resolver.ChooseMetadata(original); // On game startup this is always the original }

And this OnSavedGameConflict method gets called about every seven seconds and the final OnSavedGameOpenedForRead method never gets called.

So the following happens inside the C# plugin code.

• NativeSavedGameClient calls its InternalManualOpen method which calls SnapshotManager.Open
• This calls the parameter delegate with the Status.SnapshotOpenStatus.VALID_WITH_CONFLICT response
• A NativeConflictResolver is created which uses the InternalManualOpen method as callback.
• Now when my OnSavedGameConflict method gets called and inside that method I call resolver.ChooseMetadata(original)
• The NativeConflictResolver again calls the InternalManualOpen method which then again runs the same procedure as above

Now I could guess that it's not very clever to always return the original meta but that's exactly what OpenWithAutomaticConflictResolution with the option UseOriginal does. This method internally calls the InternalManualOpen method. The only difference is, that it sets the prefatchDataOnConflict parameter to false.

[zelon]

I'm having this issue too. As I traced, the plugin resolved the conflict and tried to reopen saved file, but there is still conflict. It seems that the conflict cannot be resolved.

[MobileMonsters]

Btw. The savegame gets loaded after several minutes and we talk about like 10 or 15 minutes.

Here's another user (that actually doesn't use the Unity SDK but I guess it's not the SDKs fault):

http://stackoverflow.com/questions/37467990/play-games-snapshot-conflict-resolution-gives-conflict

[miracle7]

Has anyone made any progress on this?

Just wondering which version of the plugin everyone is running to see if this is something that might be related to a specific version.

We are currently on 0.9.27, and in the progress of trying to upgrade to latest. The 0.9.34 'current build' unitypackage we downloaded from github was always corrupt, so we had to clone the repo then open the package from there.

Once we got the package opening, we ran into this problem, so we are trying to upgrade android sdk as suggested there and see how that goes.

[miracle7]

@claywilkinson any chance you could point us in the right direction? Our users are extremely upset that they are unable to play our game and we are feeling pretty stuck :(

[miracle7]

@MobileMonsters When the save is finally loaded after 10-15 minutes, does google give you a successful status?

Also when opening the metadata again after that point, does it go back into the loop or work from that point onwards?

[gwiazdorrr]

We have experienced the very same issue. Conflicts out of nowhere, resolution taking more than half an hour only to get STATUS_SNAPSHOT_CONTENTS_UNAVAILABLE. After restarting the game -- same story.

Unfortunately, clearing Google Play Services data does not fix this permanently, so we are now going to pursue other cloud save options.

[miracle7]

@gwiazdorrr I feel your pain :(

Just wanted to confirm that clearing Google Play Services data does not permanently fix the issue for us either, it just gets past the current loop it gets stuck in, and after that there is a chance for it to happen again.

Reinstalling our game also temporarily fixes this problem, but not clearing our app data, so we know it's not something our app is holding on to. It's almost as if Google Play Services is identifying our app by an installation id or something.

[MobileMonsters]

@miracle7 I don't have an answer to these questions since we don't have this behaviour on our own but our customers have.

Has anybody an idea of how this behaviour could be reproduced?

[zelon]

In my case, it seems that it happen just after update apk.

And below log is recorded when trying to resolve. It seems that google play games has uid problem.

07-16 21:53:25.606 800 3829 W AppOps : Bad call: specified package com.google.android.play.games under uid 10193 but it is really 10060 07-16 21:53:25.607 800 4143 W AppOps : Bad call: specified package com.google.android.play.games under uid 10193 but it is really 10060

[miracle7]

@zelon, @claywilkinson said in this thread that that issue is supposedly not a problem.

@MobileMonsters the closest we have come to reproducing it our end is by having two devices running the game simultaneously and continually opening the metadata, modifying it and saving it on both devices.

Most of the time the conflicts appear and are resolved as you would expect, but eventually we get stuck in the described conflict loop on one of the devices after we attempt a resolution.

There doesn't appear to be any specific steps leading up to this that causes the loop, and looking at the reviews on the google play store it happens across all android versions and devices too, although we usually see it on our Galaxy s6 edge here when we can get it to occur.

[AdriaanDeVos]

@miracle7 Currently we are getting infinite conflicts without the savegames showing up in the Google Play UI (PlayGamesPlatform.Instance.SavedGame.ShowSelectSavedGameUI())

We have not found a solution yet and are wondering were the problem is... Google servers? Google play games service? Play Games Plugin for Unity?

[miracle7]

I have been in contact with someone at google, and they recommended that I "experiment with altering the device's clock to a future date, saving a game, and then setting the clock back"

This is what happened when I tried to save in the future, then return to regular time:

Time is 5:45pm Can save and load without issues Set time to 6:00pm Play game a bit, can save and load without issues (save A) Set time back to 5:45pm Play game a bit then save (save B) Open metadata returns save B

I then get stuck in the following cycle. For the sake of simplicity and ability to reproduce I always resolve using save B, and when saving upload save B, even if the server gives me save A back. I want A to disappear when I do this.

1. Save (upload B)
2. Open meta
3. Server returns save A with no conflicts
4. Save (reuploading B)
5. Server returns conflict with A as original, and B as unmerged
6. Resolve using B
7. Complete callback comes back with save B
8. Back to step 1

The first time I did this I encountered the looping problem at step 6, and the moment the phones time matched up with the time I jumped forward to, Google Play Services 'stopped working' and I saw this in the logs:

07-19 04:36:41.600: E/Volley(29334): [6982] BasicNetwork.performRequest: Unexpected response code 404 for https://www.googleapis.com/drive/v2internal/files/1gl2YlZ7TdYrJdgwTW20B1c_VTv8qXcKHG8m4YmeK4pU?prettyPrint=false&fields=owners(isAuthenticatedUser,picture(url),displayName,permissionId,emailAddress),webContentLink,spaces,lastModifyingUser(isAuthenticatedUser,picture(url),displayName,permissionId,emailAddress),originalFilename,headRevisionId,fileExtension,recencyReason,id,folderColorRgb,version,webViewLink,indexableText(text),editable,gplusMedia,quotaBytesUsed,properties(key,value,appId,visibility),writersCanShare,sharedWithMeDate,shared,explicitlyTrashed,authorizedAppIds,parents(isRoot,id),thumbnailLink,modifiedByMeDate,creatorAppId,labels(restricted,trashed,starred,viewed),appDataContents,md5Checksum,thumbnail(mimeType,image),localId(value,space,version),ownerNames,sharingUser(isAuthenticatedUser,picture(url),displayName,permissionId,emailAddress),copyable,modifiedDate,userPermission(id,role,withLink,name,domain,additionalRoles,value,type,photoLink,emailAddress),etag,recency,createdDate,mimeType,alternateLink,lastViewedByMeDate,folderFeatures,title,description,fileSize,downloadUrl,embedLink,subscribed&acknowledgeAbuse=false&allProperties=true&fileScopeAppIds=68227629593&updateViewedDate=false
07-19 04:36:41.610: D/AudioPolicyManager(2971): getDeviceForStrategy mLastPolicySelection device = 0
07-19 04:36:41.610: W/NetworkResponseErrorDecoder(29334): StatusCode:404 Reason:[notFound] Location:[file]
07-19 04:36:41.690: E/DriveAsyncService(29334): Drive item not found, or you are not authorized to access it.
07-19 04:36:41.690: E/DriveAsyncService(29334): OperationException[Status{statusCode=Drive item not found, or you are not authorized to access it., resolution=null}]
07-19 04:36:41.690: E/DriveAsyncService(29334):     at jsk.v(:com.google.android.gms:1806)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at jsk.a(:com.google.android.gms:381)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at jsk.c(:com.google.android.gms:348)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at jsk.a(:com.google.android.gms:486)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at jur.a(:com.google.android.gms:51)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at jtn.a(:com.google.android.gms:1108)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at gpq.run(:com.google.android.gms:188)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at itt.run(:com.google.android.gms:453)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at iyg.run(:com.google.android.gms:17)
07-19 04:36:41.690: E/DriveAsyncService(29334):     at java.lang.Thread.run(Thread.java:818)
07-19 04:36:41.700: E/SnapshotAgent(29334): Conflict not found when resolving conflict DriveId:CAESLDFnbDJZbFo3VGRZckpkZ3dUVzIwQjFjX1ZUdjhxWGNLSEc4bTRZbWVLNHBVGPQCILSHlZzAVSgA
07-19 04:36:41.700: W/AppOps(3491): Bad call: specified package com.google.android.play.games under uid 10297 but it is really 10143
07-19 04:36:41.700: W/AppOps(3491): Bad call: specified package com.google.android.play.games under uid 10297 but it is really 10143
07-19 04:36:41.700: W/GamesNativeSDK(3078): Unexpected GamesStatusCode 4006
07-19 04:36:41.700: V/GamesNativeSDK(3078): Encountered GmsCore error with status code: 4006
07-19 04:36:41.700: E/PopupManager(29334): No content view usable to display popups. Popups will not be displayed in response to this client's calls. Use setViewForPopups() to set your content view.
07-19 04:36:41.710: W/GamesServiceBroker(29334): Client connected with SDK 8487000, Services 9256240, and Games 37230040
07-19 04:36:41.720: D/AudioPolicyManager(2971): getDeviceForStrategy mLastPolicySelection device = 0
07-19 04:36:41.720: D/ActivityManager(3491): retrieveServiceLocked(): component = null; callingUser = 0; userId(target) = 0
07-19 04:36:41.730: I/Unity(3078): Open meta failed: InternalError

The next time I tried to open the metadata, I got the same conflict that it was stuck on, but this time it was resolved successfully and I was able to save and load as per usual. This is the same behavior as when I clear the google play services app data.

I am not sure whether this is exactly the same problem as we are having, since a lot of our users (maybe 10-15% based on reviews) are experiencing getting stuck in this conflict loop, and the saving in the future thing should be pretty rare as far as I can tell.

That being said, this is the closest we have come to reproducing it, would be interested to know how this correlates with the problems others are experiencing.

[miracle7]

@claywilkinson It has been almost a month since I opened this issue and multiple people are having their live apps devastated by this bug, we are getting totally swamped with bad reviews and angry support tickets so was wondering whether you have had a chance to take a look at this thread yet?

[MobileMonsters]

We finally decided to develop our own solution. It's pretty much fact that Google gave up on us on this one. At least for now. But we can't risk losing more players.

We'll use the Google sign-in as identification still and the user will have the experience as if it's the typical Google cloud save he is using. So for them nothing changes.

For the time being we developed a workaround. Once we recognize the loop starts, we show an error message that the Google Games - Cloud Save service is not available and skip the attempt to check the cloud save. Users get an info that their game is not backuped online (no cross device).

So at least they can play although it's not making them very happy.

Am 22.07.2016 02:08 schrieb "Roman Maksymyschyn" notifications@github.com:

@claywilkinson https://github.com/claywilkinson It has been almost a month since I opened this issue and multiple people are having their live apps devastated by this bug, we are getting totally swamped with bad reviews and angry support tickets so was wondering whether you have had a chance to take a look at this thread yet?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/playgameservices/play-games-plugin-for-unity/issues/1262#issuecomment-234420657, or mute the thread https://github.com/notifications/unsubscribe-auth/ATSeSGoz4vbvqKsiYe4XVO2T3G4-TIgQks5qYAn8gaJpZM4I_n5k .

[miracle7]

@MobileMonsters We just had a discussion yesterday in our studio with a very similar resolution, except we will be opting to just disable cloud save entirely rather than wait until they get stuck in a loop or encounter single device conflicts.

We will likely just run our own server/system in future to handle syncing saves between devices, using either Facebook or GPG as the unique identifier.

Hopefully Google are able to help us resolve this before it gets to that point, but we are starting to lose hope as well and looking into damage control / alternatives.

[MrXee]

We have been facing this issue for weeks now and I would like to add some more information on this subject.

We have released our Skateboard Party 3 game on June 21st to the Google Play Store, and back then, this issue did not happen. We downloaded our game today and tested it and we could reproduce this issue 100%. This tells us that this is probably a Google Play Services server issue. So, since you've reported this issue 25 days ago, I would guess something happened between June 21st and June 28th that broke on the servers.

I will also add that when we have the "looping state" and leave it, the Google Play Services process crashes on our devices. These are the logs when it's looping:

I/Unity (15498): [Play Games Plugin DLL] 07/22/16 12:55:33 -02:30 DEBUG: Entering internal callback for SnapshotManager#OpenCallback I/Unity (15498): [Play Games Plugin DLL] 07/22/16 12:55:33 -02:30 DEBUG: Invoking conflict callback

[Froghut]

I have no more info to add to this problem, just wanted to say it's happening for our game as well. This seems like a serious issue that is affecting a lot of games, I really hope this will be fixed soon.

[MobileMonsters]

Ok, now how do we get attention of some Google engineers so these problems get solved or so we get any information about what's going on?

[Zamaroht]

I've been fighting with this problem for a whole week and there seems to be no way to even workaround it. We really need some support from the server side :/

[MobileMonsters]

We lost 20% of our DAU since this started ... that correlates with the amount of people that use the Google Games sign in. And its starting to costs us the same amount in revenues!

[miracle7]

That really sucks, feeling your pain @MobileMonsters

[MobileMonsters]

I've posted the issue in the Unity forum, maybe we'll get some help there,

http://forum.unity3d.com/threads/google-saved-games-stuck-in-conflict-loop.418430/

[PSyton]

We have same issue when use C++ GPG SDK.

We came up a workaround for this.

Resolve conflict manually. When conflict happend, check if original and merged metadata are same and retry count for conflict resolution > 1. Delete both original and merged files and stop conflict resolving process.

Code example:

if (originalPlayTime == umnergedPlayTime)
{
  // Prevent infinity loop with resolve same snapshot
  if (m_conflictRetry > 1)
  {
    logError("resolveConflict: Detected infinity loop!");
    manager.Delete(openResponse.conflict_original);
    manager.Delete(openResponse.conflict_unmerged);
    return;
  }
...

[prodigga]

Thanks for the tip though it is a bit of a flakey solution. Gpg (for whatever reason) takes a long time to process any requests that we send it (couple of seconds, where as it should really be instant considering the size of the tiny save files) so the user could possibly exit the game or have their game crash before reuploading their save after deletion. Seems really risky, no? :(

Can't believe we haven't heard from Google on this yet. Have you posted a bug on the c++ sdk Github page?

On 30 Jul. 2016 3:01 am, "Pavel V. Sysolyatin" notifications@github.com wrote:

We have same issue when use C++ GPG SDK.

We came up a workaround for this.

Resolve conflict manually. When conflict happend, check if original and merged metadata are same and retry count for conflict resolution > 1. Delete both original and merged files and stop conflict resolving process.

Code ecample:

`

if (originalPlayTime == umnergedPlayTime) { // Prevent infinity loop with resolve same snapshot if (m_conflictRetry > 1) { logError("resolveConflict: Detected infinity loop!");

manager.Delete(openResponse.conflict_original); manager.Delete(openResponse.conflict_unmerged);

return; }

`

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/playgameservices/play-games-plugin-for-unity/issues/1262#issuecomment-236235477, or mute the thread https://github.com/notifications/unsubscribe-auth/AALThDrqwDbGCgA8BKgCzaDAm2E2uqP8ks5qajHlgaJpZM4I_n5k .

[PSyton]

@prodigga

Yes it is really risky, but for this time it's better for us than infinity loop for users.

We use some mechanism for prevent loosing user data when application closed or crashed while google API calls.

We wrote to Google support, no response yet. And, we think that this not c++ SDK prblem. It's a Gogle Play Services bug.

[neoshamangames]

Here is one more person having this issue. Fortunately, we haven't launched yet, but this is definitely holding up our release.

PSyton, I'm attempting to implement a workaround like yours. What does "manager" refer to in your script? Thank you.

[PSyton]

@fractalfrenzy pgp::SnapshotManager instance from C++ GPG SDK. I think in Unity you have simular entity.

[claywilkinson]

@PSyton - where did you contact Google Support?

From reading other posts and looking around it looks like at the beginning of July there was some sort of timestamp issue with Google Drive (which is the underlying storage for Saved Games) which caused this looping.

Are you still experiencing it?

Can you share logs of the problem which I can pass along?

[stephanedupont]

Still experiencing it, and I'm not using this plugin and not using C++, so it's definitely a Google Play Services bug.

[claywilkinson]

OK, thanks - logs would be helpful. When you say you are not using Java and not using C++ - are you using this Unity plugin? The plugin actually wraps the C++ SDK - so most any issue that affects C++ will affect this plugin.

[stephanedupont]

Oups sorry, message edited, what I meant was I'm not using this plugin and not using C++

[stephanedupont]

It was just a feedback to confirm that the problem is also for users not using Unity or C++. That's how I landed on this thread.

[claywilkinson]

Thanks for the clarification! If you have logs, I'd still like them to see if it will help get a resolution.

[stephanedupont]

Logs incoming, but in the same time, as someone said here: https://github.com/playgameservices/android-basic-samples/issues/210

"The behaviour I see seems to be that, when resolving a conflict, this snapshot passed to the resolve function will itself create more conflicts, judging by the last modified time of each snapshot in the open callback."

That's what I experienced also, each time I resolve a conflict a new, it generates a new conflict, and it never ends.

[stephanedupont]

Well, it's hard to get you logs because the account on which I experience this bug has now hundreds of conflicts in it, so I'm not sure it will be of any help. Here they are anyway: https://cloud.eluna.org/s/LTVZep9ZkGgEjhe

It's probably a hard one to solve, but please, if you can do anything quickly, it would be awesome, because end-users are not happy with this at all.

[claywilkinson]

Thanks - I'll pass it and your sense of urgency along.

[stephanedupont]

Thank you

[MobileMonsters]

Greetings, any updates on this one?

[claywilkinson]

Not yet, I'll update this thread when I know more.

[c-rizz]

same problem, using java

[seed-jack]

Hello,

Developer from Doctor Who: Legacy as having the same problems around the same time this all started. Thought I'd add some information about what we are seeing.

• using Unity 4.7.1
• PlayGames Plugin DLL reports itself as "[Play Games Plugin DLL] 08/15/16" from logcat
• seeing "google-breakpad" dump after performing a save on our game end but only on first launch after install
• see the same "bad call: specified package com.google.android.play.games" with different uid's (probably harmless?)
• PlayGames Plugin outputs "Entering internal callback for SnapshotManager#CommitCallback"
• SnapshotAgent prints out "Found conflict for snapshot 'our_filename'"
• loop repeats

If I leave the game running long enough, Android will say that the Google Play Services stopped responding. After leaving it there for about 5 minutes, the dialog will disappear and continue with the game.

Hope that the fix is Google side but if it does require a library patch I hope it takes into account the older Unity 4.7.1 as we can't move to the newest Unity 5 version.

Let me know if there is any more information that I can provide.

[claywilkinson]

Thanks for the details @seed-jack.

• Based on the findings from other developers the problem affects Java apps also, so it is pretty safe to say it is Unity version independent.
• The "bad call" messages in the log are harmless.

I'll update the thread when I know more about the fix. Since the issue is not Unity related it should also address 4.7.1.

Just out of curiosity, why can't you move to Unity version 5+?