Viewing error logs for rollbacks

[ryanvanderpol]

We are disabling CodePush in our production iOS and Android apps because we have seen continuous instability with our OTA updates and no way to view logs or diagnose the reasons for these rollbacks. We will consider coming back if logs are made available so that we can accurately determine the cause of rollbacks and address them.

Here is the history table for the last couple weeks for one of our iOS apps:

I've brought this up across several different threads and every time the issues are closed and I am told that "it's being tracked elsewhere" but there no one can give me any indication of where "elsewhere" is or when we can expect to get these logs. I am DEFINITELY not going to use AppCenter until this is resolved.

When can we expect logs for rollbacks to be available?

[ruslan-bikkinin]

Hi @ryanvanderpol and thanks for submitting the issue. We understand your problem but CodePush itself not intended to be a bug/crash tracker, you should use AppCenter Crashes or any other related tools for it. We could in future add possibility to gather extra information about device wherein rollback been occurred but in terms of Appcenter as we are about to move current RN SDK to AppCenter SDK and ship all new features there.

Anyway, of course we will try to help you with your current issue. Could you please, update your issue with additional information about your application, environment as specified in our issue template? As your case is very specific, please, try to provide as much details as possible including Sentry logs.

[sergey-akhalkov]

Hi @ryanvanderpol, please also pay attention to the codePush.notifyAppReady() function, from the official docs:

Notifies the CodePush runtime that a freshly installed update should be considered successful, and therefore, an automatic client-side rollback isn't necessary. It is mandatory to call this function somewhere in the code of the updated bundle. Otherwise, when the app next restarts, the CodePush runtime will assume that the installed update has failed and roll back to the previous version. This behavior exists to help ensure that your end users aren't blocked by a broken update.

When can we expect logs for rollbacks to be available?

CodePush is not crashes tracker. You need to use one of the related tools to collect crashes info for analysis.

Could you also please clarify - do I get it right from this https://github.com/Microsoft/react-native-code-push/issues/1044 issue that you have Sentry installed/configured and it does not track any errors even if you have rollbacks? If so, please make sure you are using codePush.notifyAppReady() method.

[ryanvanderpol]

You guys are missing the point and this is why I am leaving CodePush. The attitude I consistently see around here is "it's not our fault". I understand that you think CodePush is "not intended to be a bug/crash tracker" but if crashes happen while CodePush is applying an update I'd imagine the average person would want to know what's going on and have some sort of report that they could look at to investigate the problem. I fully believe that is something CodePush should provide if you guys intend for this to be a polished, production-ready product.

For what it's worth, our production apps already use Sentry and nothing is being logged related to rollbacks.

[patniko]

Hi @ryanvanderpol, better crash and rollback support is very high on our priority list! In fact we just moved our entire backend into App Center so we can set the stage for better crash and analytics support in the near future which should accomplish what you are looking for. The way rollbacks presently work within CodePush prevents us from gathering the detailed lower level information you are after and requires a rewrite, but we are changing that in the near future. After we finish moving the CodePush SDK's into the App Center SDK's as a new module around February we'll be focusing heavily on better rollback, crash and patching support for React Native and Cordova solutions.

[yaronlevi]

@pniko

focusing heavily on better rollback, crash and patching support

Excited to hear that!

[adbl]

I would like to add that we typically see a couple of rollbacks on each update that reaches Production. We have extensive logging (also crash reporting) elsewhere and in those cases I looked into I have never found any related log/crash in our other tools.

We log stuff immediately when the JS code has loaded and starts executing. When a device upgraded from version A -> B but got rolled back, we had zero JS log entries at version B (for this device). Thus I assume something went wrong internally within codepush, since 99% of devices made this upgrade.

[jbcullis]

Same as @adbl - we only get around 1% rollback according to the code push stats but never see any code-push errors appear in our logs.

We're also contemplating just going back to standard app store rollouts, we run LOB software so have to eliminate user inconveniences wherever possible.

[ryanvanderpol]

Just wanted to leave this here as a data point for the CodePush team: we dropped CodePush about a month ago and are WAY happier now. It has increased our apps' stability tremendously. CodePush seemed like an awesome idea, but we continually had customers reporting crashes and would get tons of rollbacks with every CodePush update we sent out with no error logs available to look at.

[AgDude]

@ryanvanderpol are you using another similar solution, or did you go back to using binary updates?

[ryanvanderpol]

@AgDude We went back to binary updates. For Android, that's basically as easy as a CodePush update. For Apple, we obviously have to wait for the review cycle, which has been about 24-48 hours for the last 10 updates we've done.

This is obviously a bit more overhead, but the improvement we've had in overall stability is well worth it.

EDIT: and FWIW we find that 95% of our users adopt the updated binary within 5 days, which is even better than we were seeing with CodePush.

[patniko]

@ryanvanderpol We are still heads down on rewrites for the SDK's right now that will bring detailed rollback and JS crash information to the surface in a visual and actionable manner, but have finished building out our new backend and releasing Cordova support for App Center. Small team, but we are absolutely still dedicated to mitigating your concerns. I will follow up on this thread in the February timeframe with an update as we get closer to shipping the updates you've been looking for.

[yaronlevi]

@pniko Maybe we should merge those two issues:

https://github.com/Microsoft/react-native-code-push/issues/880

They practically repeat the same pain points, and suggest the same solutions.

[patniko]

Good point @yaronlevi ! I will go through the board with the team and do a bit of house cleaning.

[hobzcalvin]

Hi @pniko, February came and went. What's the timeline for seeing this resolved? (As I mentioned in #880 this is definitely a Cordova issue too; not sure why both of these are under the React Native project.)

[Zakeelm]

We had a few internal changes that have caused a delay on the SDK work and a reprioritization of our team's backlog. While we don't have an ETA for this feature item at the moment, I can assure you this is one of the highest priority items we have on our roadmap. We're going to be investigating ways to implement this without the SDK rewrite, so that we can potentially get this to you sooner than later.

[jun1st]

Guys, What’s the status?

I’m facing serials rollbacks when testing code push. And would like to know how is this going on?

[twelve17]

Hi @Zakeelm , first of all, thanks to you and the team for listening and responding to this issue. I am curious if there was a chance to work on the ability to log rollback issues?

[jpike88]

I stopped using Codepush over a year ago because of mysterious rollbacks, that provide no context as why they occur. Finding this issue as still unresolved makes giving Codepush a go a very hard ask. I just don't get it though. Why provide a service (free or paid), that fails to perform in such a fundamental way?

[haswalt]

This is still very much an issue. Our exception logging is not able to catch any errors causing rollbacks because CodePush seems to silently drop these.

At the moment, only a small subset of devices (1 out of 4 internal test devices) fails but with no idea why or how to debug we're concerned about the impact on real users when we release.

[teovillanueva]

This is still an issue, has anyone found a way to get a reason for rollbacks?

[adbl]

Hey I got a notification about this old thread. I stopped using CodePush back 3-4 years ago for my project at the time and since then I've recommended on all other projects to avoid it, referencing this issue.

Unbelievable how it is still not addressed.

[CyberMew]

@adbl did you manage to find a viable alternative?

[jbcullis]

@CyberMew Honestly, just roll out through standard app store processes. It's worth the extra effort to eliminate untraceable bugs that ultimately end in lost users or support tickets.

[adbl]

Yeah, I would never use it for anything except to roll out some emergency fix where the app is really broken and the uncertainty is worth the speed benefit. OTOH in that rare moment you have forgotten your CodePush routines causing even more stress.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[adbl]

Bump 😂

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[adbl]

Bump for good laughs

[shllg]

When looking at the thread, it's sad that after all the time there still no way to figure reasons of rollbacks out. We're not able to finish the integration of CodePush due to rollbacks and it's not clear why.

https://github.com/microsoft/react-native-code-push/issues/2020#issuecomment-1048186521

[williamgurzoni]

I think the most sensible thing to do is to add an option to disable rollbacks.

[jun1st]

just get rid of react native code push,

Feng Qijun

On Wed, Feb 23, 2022 at 4:33 AM Sascha @.***> wrote:

When looking at the thread, it's sad that after all the time there still no way to figure reasons of rollbacks out. We're not able to finish the integration of CodePush due to rollbacks and it's not clear why.

2020 (comment)

https://github.com/microsoft/react-native-code-push/issues/2020#issuecomment-1048186521

— Reply to this email directly, view it on GitHub https://github.com/microsoft/react-native-code-push/issues/1099#issuecomment-1048188199, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABCJOTUR4KJCONRVCUGAADU4PXKJANCNFSM4EG2WB4Q . You are receiving this because you commented.Message ID: @.***>

[shllg]

I think the most sensible thing to do is to add an option to disable rollbacks.

@williamgurzoni Here is a good (and long) discussion where is explained why it's not a good idea to disable automatic rollbacks: https://github.com/Microsoft/react-native-code-push/issues/880#issuecomment-342797994

---

The question is, why CodePush marks an update download as done but after restart, it is still marked with updateIsLoading. My ObjC skills are not good enough to figure it out by code and - as this ticket states - there are no further logs present to get any hints.

FYI in our case, the log message is generated here:

https://github.com/microsoft/react-native-code-push/blob/master/ios/CodePush/CodePush.m#L404

But it's obviously not the point of failure. Somewhere the update is loaded and marked as updateIsLoading.

 NSDictionary *pendingUpdate = [preferences objectForKey:PendingUpdateKey];
    if (pendingUpdate) {
        _isFirstRunAfterUpdate = YES;
        BOOL updateIsLoading = [pendingUpdate[PendingUpdateIsLoadingKey] boolValue];
        if (updateIsLoading) {
            // Pending update was initialized, but notifyApplicationReady was not called.
            // Therefore, deduce that it is a broken update and rollback.
            CPLog(@"Update did not finish loading the last time, rolling back to a previous version.");
            needToReportRollback = YES;
            [self rollbackPackage];
    ....

[shllg]

FYI: I patched the iOS code. At least for our mentioned issue, we have a working code push cycle now:

https://github.com/microsoft/react-native-code-push/issues/2020#issuecomment-1048696343

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[SYoder1]

Bump

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 15 days after stale. Please feel free to open a new issue if you still have a question/issue or suggestion.

[abhishekgoel101]

bump

[Acetyld]

bump

[vjerryv]

bump

[NNHungLong]

still facing mysterious rollback without any trace/ way to debug.

[baixjian]

bump

[funbistraitpay]

it's the end of the year... and I'm facing some heat at work because codepush has some mysterious rollbacks on iOS which I'm unable to figure out. Is there a better alternative for OTA updates?

[lklepner]

Expo Updates

https://docs.expo.dev/versions/latest/sdk/updates/