Possible to fix automatic rollbacks that occur?

[huangkaiw3n]

Description

A small number of users get automatic rollback sometimes. When that happens, they lose that update. I've experienced it before and a reinstall of the app allowed the code push to update the app successfully. Is there anything on the dev side to trigger something like this or some way to code the codePush syncing to attempt to sync again even after a rollback? So that an update is not lost to a user until a reinstall.

Additional Information

• react-native-code-push version: 2.0.3-beta
• react-native version: 0.44.0
• iOS/Android/Windows version: On iOS and Android (not sure what are the user's versions)
• Does this reproduce on a debug build or release build: Release (As I've never ran codepush on debug mode)
• Does this reproduce on a simulator, or only on a physical device: Doesn't reproduce consistently and never tried codepushing when running on simulator (debug mode)

Eg. when I check the deployment stats:

My CodePush code in App.js (imported by index.ios.js and index.android.js)

let codePushOptions = {
  checkFrequency: codePush.CheckFrequency.ON_APP_RESUME,
  installMode: codePush.InstallMode.IMMEDIATE
};

export default codePush(codePushOptions)(App)

[iageoghe]

Hi @huangkaiw3n - Thanks for submitting this issue. I have a couple of questions:

1. Do you mean that such users can not update until the next release you distribute through CodePush?
2. Do you want a behavior where the app tries X number of times to install a CodePush release?

[huangkaiw3n]

@iageoghe Thanks for looking at this issue.

Regarding your questions:

1. I'm not 100% sure if this is even the case. Eg. When a user gets an automatic rollback presumably due to a crash on the first update attempt, whether he will be able to receive the next CodePush release. (So far, I've only encountered an automatic rollback perhaps once or twice on my staging CodePush releases. And when they happen, closing and reopening the app did not trigger another attempt to update. Reinstalling the app did trigger the CodePush update though)
2. Yeah, this could be a way to combat the issue where the update is clean but the app crashed at the update attempt due to some arbitrary problem that could happen at the time of update. I was also wondering if my CodePush syncing code was generally the correct way to call sync or if there is any existing way to address this.

Do you know if this is a first or were there any others experiencing lost updates due to an auto rollback at CodePush update attempt?

[Minishlink]

FYI, we seem to have this issue on an app using 2.1.0-beta, but not on another app that uses 2.0.3-beta, though this might be a coincidence.

EDIT: issue's author is on 2.0.3-beta so it's definitely a coincidence...

[jbcullis]

I've been experiencing this too. It seems like the app thinks it has already updated. It's very sporadic and I cannot reproduce this at will but when users contact me to say the app isn't updating a reinstall or rolling a new version will typically resolve the issue.

[huangkaiw3n]

@jbcullis Hey I think we're facing the same issue. Could you share your react native version and react native code push version? Also how you are calling the sync in your react native code? This could help us find some clues.

[jbcullis]

@huangkaiw3n our versions are:

"react": "~15.4.0-rc.4",
"react-native": "0.38.0",
"react-native-code-push": "^1.17.0-beta",

Calling sync when the app starts: CodePush.sync({ updateDialog: false, installMode: CodePush.InstallMode.IMMEDIATE }).then(...)

The then statements hides the loading box if an update is not found.

[jbcullis]

I had never thought to look for rollbacks, I have been watching this since our most recent update 8 days ago and now have 1 rollback on iOS.

[JasonHenson]

+1

[jeremyong]

We just want an option to disable it at this point. We have observed in vitro that the rollback is happening on its own accord without a crash happening. To be frank, this functionality is strictly undesirable and I don't know of any use case why you'd want to rollback to the bundled code ever. The most I would expect is a simple checksum to ensure that what was deployed is in fact what was shipped.

Note that objections to this proposal would probably take the form of "but what if the user is stuck and can't ever update." This is true regardless of if the rollback takes place or not as the developer is always potentially able to introduce a bug that prevents an update/sync from happening.

[jbcullis]

Yep, I second this @jeremyong - Put something in the API that allows us to FORCE an update to the latest bundle. We put in a long press button so we can ask users to do a manual check but it doesnt help at all because code-push thinks it has the latest version. The issue has always been that once an update faile, code-push still thinks it was a successful update therefore refuses to update.

[ghuh]

We've seen this issue when the user manually kills the app on Android, there was no crash involved.

RN: 0.47.2 Codepush: 5.0.0-beta Android: 7.0 (Galaxy S7)

[yaronlevi]

We always get 10% rollbacks, mostly Android.

Configuration is::

AppRegistry.registerComponent('myapp', () =>
  codePush({
    checkFrequency: codePush.CheckFrequency.ON_APP_RESUME,
    installMode: codePush.InstallMode.ON_NEXT_RESUME,
  })(Root),
);

react-native-code-push@5.1.3-beta react-native@0.49.0

[huangkaiw3n]

To be fair, the rollback is useful if it is really rolling back for a case where the JS bundle has a problematic line that causes the app to crash.

BUT that isn't working good and instead, good bundles are getting rolled back and these users lose that push for good until they reinstall. So yeah, @jeremyong's suggestion sounds good. A flag to disable rollbacks for a push would fix this issue.

[yaronlevi]

@huangkaiw3n +1

It could be great if had a way to diagnose the reason for a specific rollback, or as suggested disable this feature altogether. We get rollbacks on JS bundles that are 100% good. My guess is that the process of installing the new bundle gets interrupted in some way: flaky internet connection, some wired user interaction regarding resume/open. But we can only guess, as there is no way of knowing what happened in the client's end.

[ruslan-bikkinin]

@jeremyong @jbcullis @ghuh @yaronlevi @huangkaiw3n Hi all of you guys and let me clarify how rollback is working. CodePush does not trace any errors or something like that, the condition to determine whether the update was succeeded or not is if codePush.notifyAppReady (or codePush.notifyApplicationReady) was called. This method mark update as installed successfully on native side so if on next app start there will be pending update with status differs than installed successfully then it will be rolled back. If you are using sync command, this will be done automatically as part of update mechanism:

https://github.com/Microsoft/react-native-code-push/blob/8c5bf4ba72a63de6a58e87e1276345656a1d3147/CodePush.js#L343

This part of code couldn't be called when there was error before reaching this code which in general could be caused by incorrect bundle execution which, in turn, caused by bug in bundle initialization. As you can see this is very straight-forward and harmless approach, so if you want to handle app rollback in other conditions you should install update manually, no need to provide additional option for this.

Please let us know if this information was helpful for you or you need additional help from our side.

[yaronlevi]

@ruslan-bikkinin

could be caused by incorrect bundle execution which

Regarding "incorrect bundle execution", do you mean some error in our javascript code? Like in app.js near AppRegistry.registerComponent()? (maybe in the ctor)

And if not, where is this "bug in bundle initialization" could be found. The only code we control (write) is JS and some native modules.

[ruslan-bikkinin]

@yaronlevi

If CodePush.notifyApplicationReady(); will not be called until next app restart app will be rolled back. This method call could be unreached due to several errors including developer code bugs.

Please refer to this documentation section for more info.

[huangkaiw3n]

@ruslan-bikkinin Thanks for the clarification. However, the issue here is that this mechanism of notifyApplicationReady apparently causes a handful of users hitting rollbacks on js bundles that were fine.

I only started realising these 'false rollbacks' when users sent in feedback and follow-ups with them discovered that they were not on the latest code push version which should have fixed some bugs. Instructing them to reinstall allowed their app to download the actual latest version.

Any arbitrary crash even when it's not due to a faulty js bundle, but due to the OS or certain conditions can potentially cause notifyApplicationReady to fail, resulting in a 'false rollback' on a good js bundle.

As mentioned by @iageoghe , even a try X times I believe will mitigate this problem, especially on production systems with large user base. Otherwise, a flag to disable rollbacks for a certain code push version will help as well.

[jbcullis]

@ruslan-bikkinin we have all the same issues as @huangkaiw3n - I don't know 100% if it's the false rollback since we have not yet been able to reproduce it locally. There is no consistency for reproduction as far as I can tell. What I can tell you with 100% certainty is, the app THINKS it's the latest version and refuses to update. Maybe give us an option to submit our own version number and leave the version control to us?

[jbcullis]

@ruslan-bikkinin - hypothetically, if the app experienced a hard exit in the same session where we updated the bundle, would it be possible for the app to still think it's up to date but roll back to the previous bundle?

[huangkaiw3n]

@ruslan-bikkinin Yeah, there's no way to reproduce consistently as they're 'false roll-backs', presumably an arbitrary condition between update and notifyApplicationReady.

Even perfectly good apps do crash very occasionally. When it does crash between the update and notifyApplicationReady, the code push is lost to the user till an install.

I only encountered it when the users started to really go up, and it was from user feedback. Was able to further confirm my suspicions when I actually encountered it once or twice when testing on Staging.

Low percentage, but strictly undesirable:

[jeremyong]

@ruslan-bikkinin exactly as @huangkaiw3n has said.

Also I've read the code and understand the general flow (and have commented out relevant sections locally). I still think that no rollback whatsoever is the best policy. Our code leverages sync as suggested and we are certain that notifyApplicationReady is called but the rollback occurs anyways sporadically and we have observed this first hand as well. I don't trust the usage of user preferences to store metadata about whether a bundle succeeded or failed.

[JasonHenson]

Not really sure what the debate is. I would never want to have "ghost" rollbacks that happen without my knowledge and leave users stuck on an old version. If I don't initiate the rollback then I personally don't want it to happen. If an update fails, it should try it again the next time code push checks for an update. If for some reason the app starts to crash because of the latest version code pushed, users could put in a support ticket and we would look into the issue and just code push a fix or release a new binary as a last resort.

[ruslan-bikkinin]

@ghuh

We've seen this issue when the user manually kills the app on Android, there was no crash involved.

Please submit new issue for it with detailed information about your application so we could help you in this particular case.

@ghuh @jbcullis

hypothetically, if the app experienced a hard exit in the same session where we updated the bundle, would it be possible for the app to still think it's up to date but roll back to the previous bundle?

If for some reason the app starts to crash because of the latest version code pushed, users could put in a support ticket and we would look into the issue and just code push a fix or release a new binary as a last resort.

If you release a broken update (that causes your app crash) and disable automatic client rollbacks, your application couldn't be updated via codepush anymore because client code couldn't be executed. Besides might be situations when broken application will be crashing over and over again - that will lead to bad user experience. This is not what we expect for our customers in general. As a solution for possible rare false-positive rollbacks we could implement retry installation mechanism in future: if for some reason update couldn't be installed we rollback it but doesn't send report to the server. Instead we will try to install it again later and if will not succeeded then we will rollback it and send report to the server. I believe that would be a reasonable compromise in this case.

[JasonHenson]

I think rolling back and retrying to update again would be best.

I was thinking last night whether it is possible to check for rollbacks and then log them with sentry.io, so at least I would get an email when this happens.

[huangkaiw3n]

@ruslan-bikkinin Building in retries sound good and I believe should be built as a default behaviour. The whole issue of these false-positives though rare is that the user loses the update entirely. A retry mechanism perhaps like !latestVersion && minimumTimeSinceRollback should eliminate these entirely since they are rare.

So far this is the only nagging issue I have with using code-push for close to a year and rest of it has been smooth so thumbs up and a big thanks to the code-push team.

[yaronlevi]

+1 for a retry mechanism.

Also, getting some kind of log info about the device that experienced the rollback could be very helpful.

[ruslan-bikkinin]

Got it, guys!

Also, getting some kind of log info about the device that experienced the rollback could be very helpful.

How do you think, @yaronlevi, what kind of information will be useful?

[yaronlevi]

@ruslan-bikkinin

We use react-native-device-info (which is quite popular).

I would try to follow the implementation of:

import DeviceInfo from 'react-native-device-info';
const uniqueid = DeviceInfo.getUniqueID();

And make this unique id, which is a unique device id, available in a log when a crash happens.

Looking at the source code of DeviceInfo.getUniqueID() could be a good reference, as this library is pretty mature and well tested, and generates this unique id for both iOS and Android.

[jbcullis]

@ruslan-bikkinin I'm interested in a perpetual retry, or even a result on the sync that tells me it's a rollback user and an option for us to tell it to retry would be ideal....that way we could log it and possibly identify cause for rollback. Our users are primarily rural so my feeling is it's a dropped connection but don't know for sure. I was trying to hard reset the phone during an update (as per suggested forcing app close above) as the other day but couldn't replicate it. I know it's not a bad push because the same user will re-install and get the latest package no problem.

[jeremyong]

@jbcullis I don't think this is a good idea. Based on statistics and logs I've collected. The rollbacks may be occurring to the same people even after a follow-on release is happening. Again, I'm not sure exactly what might be causing it, but with your approach, an affected user may continuously download and apply the update, only to roll it back on the next application start. I think it should be disabled until the root cause is determined.

[jbcullis]

@jeremyong in our experience, it's luck of the draw with any given push. We have users who have been successfully accepting pushes for the last 18 months and out of the blue, one of the releases will present this issue. Push a new release or ask the user to re-install the app and the problem is solved. This means, the very same push is working just fine with the same user on the same device which is why I believe it's more likely an environmental issue. The randomness is also what is making it so hard to debug because we are yet to be able to re-produce it in house.

My preference would be on sync, a response indicating the user has rolled back but failing that, the next best solution for us is a perpetual retry...meaning each time sync is called the app tries to update. At bare minimum, I'd like to see an option to catch a rollback and add it to our bug tracker.

Quite frankly, I'd take an option where we manage the version codes and handle update decisions.

[yaronlevi]

Does one of the suggested solution mentioned above is scheduled on the roadmap or is considered?

This is especially important since Code Push is now a part of App Center which is a paid service. This could come as a real deal breaker for some.

[monotkate]

@ruslan-bikkinin We've been trying to implement retries in our flow that uses the API commands instead of sync due to state issues, where we want to try a bad install again with a cleared user state but have been running into issues, particularly on ios.

Can code-push handle rollback more than once from the same failed upload? On ios I'm seeing issues where the pushed build never calls notifyAppReady, and yet still manages to get stuck on the failed build after the first rollback.

[ruslan-bikkinin]

@monotkate I believe there is no way to implement retry logic using API because this feature requires SDK changes.

@yaronlevi Yes, we are considering to implement this as highly requested feature. I'll keep you posted.

[monotkate]

@ruslan-bikkinin Yeah, we came to that conclusion ourselves. It would be awesome if, when you guys implement retry, we've got the ability to run code based on how many times it's retried. That way we can try to do something like wipe state between retries in case it's causing failures.

[ryanvanderpol]

When rollbacks like this happen is there anyway to see a log somewhere? We're using Sentry in our app but not seeing anything relevant in there and we're seeing really high rollback rates for our last five or six OTA updates. I have no idea why they're rolling back and I can't seem to find logs anywhere.

[ruslan-bikkinin]

@ryanvanderpol Unfortunately, codepush doesn't provide any logs for rollbacks for now. You can create separate issue to continue discussion about your specific case.

[yaronlevi]

Crash reporting on Code Push errors should be a top priority in the roadmap. Especially now that App Center is a paid service.

[patniko]

Hi @yaronlevi, please see my response in #1099 for more information. You'll be happy to know we have our eyes already on the problem.

[jbcullis]

We really need resolution on this, I've had 2 nasty emails and lost both customers over this issue this month. It scares me to think how many folks are seeing the issue and just bailing on the app rather than reaching out.

Is there a way to mock a rollback in dev so we can put some handling in place? And please, please, please let us handle the retry and associated logging.

[atticoos]

I've observed installations that fail at the download level appear to result in a failure to install and get marked as a rollback. Our logs seem to indicate intermittent interruptions on the code push server side, which one should expect of any remote resource.

However, I'd imagine we can be more resilient about download failures. I may be misguided that my small number of rollbacks are represented by download failures, but I just encountered a higher volume and the logs I have on file show:

• An SSL error has occurred and a secure connection to the server cannot be made
• Could not connect to the server
• The request timed out

The number of log entries seems to match the number of rollbacks reported.

---

What can be done to gracefully handle transport related issues, where the installation can become reattempted w/o ignoreFailedUpdates being used, as that can have unintended consequences of actual rollbacks being reattempted.

[jbcullis]

Good work @ajwhite - most of our customers are rural so consistent connectivity is not guaranteed.....this would explain a lot for us because it seems luck of the draw who experiences a rollback on any given release.

[atticoos]

Indeed a failure can be marked if part of the download & unpacking process failed.

However, it's not a catch-all scenario. This is only if it's a "code push error". A code push error can happen when:

• The download is missing a bundle file
• Fails the hash check when signature verification is enabled
• Fails the signature check when signature verification is enabled
• Missing JWT when signature verification is enabled
• Fails the hash check when signature verification is not enabled

All of these are valid reasons to reject the build. I'm not otherwise seeing download failures leading to rollbacks, so this might be a red herring. I can't pin down why only a subset of devices are rolling back..

[jbcullis]

@ajwhite My assumption was some sort of interruption but maybe it's more likely the initial connection to code push as you suggested previously. What I can tell you is, same person, same device....will be fine for all previous updates and then randomly runs into this issue....then a new release will get that same user back on course.

[yaronlevi]

@ajwhite You said "in our logs"... What logs to do you mean, JS or Native? Something like Sentry.io?

[atticoos]

@yaronlevi these would be JS logs on a separate service like Sentry

[jariwalabhavesh]

I was also face this issue. Now i am able to identify why it is happening in my case.

1. I had codepush V2 already publish for android app
2. Then i had release V3 and facing rollback in this version

I found that by mistake i had commented below code block, which is causing rollback in V3, in V2 it is working fine without any rollback.

// codePush.sync({
//   updateDialog: true,
//   installMode: codePush.InstallMode.IMMEDIATE
// });

May this can help other to identify issue.

[hobzcalvin]

I was pointed to this issue via App Center support staff. Lots of React mentions but this is definitely an issue for us too in Cordova.

The worst part in our experience is new users who download our app for the first time: Let's say we've released a dozen CodePush releases since the app was released to the app store. All those releases worked fine (except maybe the latest) and target the given app-store version. The user downloads the app from the app store and it attempts to get the latest CodePush release, but (clearly very mysterious thing!) happens and the app rolls back. But instead of attempting to install the previous / perfectly-fine CodePush release, the CodePush plugin reverts to the version of the code that was packaged with the app, which is now very out of date.

So I would propose falling back to installing previous CodePush updates, in addition to retrying install of the latest one, instead of simply giving up.

As a quasi-workaround, we can manually roll back the latest release, which creates a new one, which the app will hopefully successfully install. Then we can try re-releasing the problematic update, etc. But this results in lots of release churn that interrupts users while they're in the app. Really not a solution.

[hobzcalvin]

Out of curiosity, has anyone tried using the notifyFailed handler in notifyApplicationReady to document when notifying the plugin fails? Obviously if there's such an error handler, there must be some theoretical way for notifyApplicationReady to fail, which almost guarantees a rollback.