Replicator error - can't push to SG

[arribbar]

This bug appears since the last update of this plugin (or I haven't seen it before). The first problem I have it's I can't reproduce it every times but only some times.

The bug: My device doesn't make any push to the Sync Gateway but can pull any docs. It seems to come from at the connection with the sync gateway when I start the application. I got this error :

06-04 11:11:45.516: V/Listener(18097): Authorized via basic auth
06-04 11:11:45.596: E/Router(18097): Replicator error: java.lang.Exception: com.couchbase.lite.support.RemoteRequest@421ab9c0: Request org.apache.http.client.methods.HttpGet@42178ee8 has been aborted.  Repl: Pusher@421a5958[http://---:---@192.168.43.61:12345/customers].  Source: customersregistered, Target: http://amel:amadeus@192.168.43.61:12345/customers

As I said before, I don't know how to reproduce it correctly as it happens randomly for me. I'm using the last version of this plugin, the 1.0.0-72 and the same goes for the sync gateway and the server. To connect to the sync gateway, I'm using the basic http authentication putting the username and password to the url (http://username:password@192.168.43.61:12345/customers), adding before the user with a curl request to the SG.

[tleyden]

Can you reproduce this and capture all of the logs and paste them?

There are a few cases which would cause requests to be aborted like that -- if either replication.goOffline() is called, or if replication.stop() is called. I'd like to try to figure out which (and why) to try to further debug it.

[arribbar]

The logs from logcat are available on pastebin : http://pastebin.com/FfzzDr0w

[tleyden]

@arribbar

This error looks like it is the problem:

06-06 16:28:31.381: W/Sync(12812): com.couchbase.lite.replicator.Replication$8@421aad38: error getting remote checkpoint

Because of this, the replication is stopping. Normally the response to the request to get the remote checkpoint should either be a 200 OK or a 404 NOT FOUND. For some reason, your sync gateway is throwing a completely different error.

This should be visible in two places:

• Sync Gateway logs (you may have to enable more verbose logging)
• Sniffing the network traffic between couchbase lite and sync gateway.

Can you reproduce again and upload the sync gateway logs to a gist or pastebin?

If that's not helping, then try HTTPScoop, ngrep, or wireshark to capture the network traffic.

[arribbar]

It's weird because I have the 404 NOT FOUND

In the gist https://gist.github.com/5b32cca2c3d256dd6a6a, you have the logcat of the device, the cache, CRUD and changes logs of Sync Gateway and the ngrep.

To help understanding the ngrep and SG logs, SG IP address : 192.168.43.61 Device with the push problem: 192.168.43.1 Other device: 192.168.43.82

Also, i'm using the same username and password for both devices ...

[tleyden]

It looks like the reason for the error getting remote checkpoint is that the get checkpoint request was was aborted for some reason:

06-24 11:06:02.516: E/Sync(20564): Pusher@421a54b8[http://---:---@192.168.43.61:12345/customers]: Progress: set error = java.lang.Exception: com.couchbase.lite.support.RemoteRequest@422722a8: Request org.apache.http.client.methods.HttpGet@42272368 has been aborted

So we need to trace why that request is being aborted.

Can you reproduce this again, and upload the full logcat logs which should start as soon as the app is opened for the first time? There should be the output Starting Manager version: somewhere in the logs.

Keep your eye out for "stopRemoteRequests()" or "aborting request" in the logs.

[arribbar]

I don't see anything like that on the logs from the device

Here the logs when the replication is working : https://gist.github.com/arribbar/0d6973d97a35f945db07

And when is not working : https://gist.github.com/arribbar/18ed705fb72d30753aed

I don't know if the device is well initialized when we call the first GET _changes. Also, at the end of the logcat when is not working, we can see :

06-24 18:03:57.434: V/Sync(26716): com.couchbase.lite.support.Batcher@421b13a0: processNow() called
06-24 18:03:57.434: E/RemoteRequest(26716): com.couchbase.lite.support.RemoteMultipartDownloaderRequest@4213e708: executeRequest() finally

What do you think about that ??

[tleyden]

I think I'll still need to see more logs to determine what aborted these requests.

For example, for the log line here: https://gist.github.com/arribbar/18ed705fb72d30753aed#file-logcat-L154

If I search for the hash code (42203300) of the com.couchbase.lite.support.RemoteRequest@42203300 object, I'm not able to find any place earlier in the logs where it was aborted.

What about restarting the device / emulator completely, and capturing logs from the time it was started and reproducing the issue again?

If you are capturing the adb logcat output in a terminal, you may want to make sure the scrollback buffer is increased so the logs don't "roll off the edge". Alternatively, you can use adb logcat -f filename.txt to capture logs to a file.

[tleyden]

What do you think about that ??

I want to get to the bottom of why this is happening before looking at anything else: https://gist.github.com/arribbar/18ed705fb72d30753aed#file-logcat-L154

[arribbar]

Actually, you have the logs from when I start again the application, this one deleted previously. I'd like to give you more logs but there is nothing else ...

[tleyden]

OK I just discovered that for the phonegap plugin, the default logging is not high enough. In https://github.com/couchbaselabs/Couchbase-Lite-PhoneGap-Plugin/blob/master/src/android/CBLite.java we'll need to add calls such as https://github.com/couchbaselabs/ToDoLite-Android/blob/master/ToDoLite/src/main/java/com/couchbase/todolite/Application.java#L53-L57

[arribbar]

Ok, I'll do it tomorrow then, when I am able to do it

[tleyden]

@arribbar yes, if it's possible for you to rebuild the plugin on your end, that would be the fastest way. I've pushed the code to github (https://github.com/couchbaselabs/couchbase-lite-phonegap-plugin-builder/commit/78d841b80bd0186d5e2cc0de4817cb51ee3926ab), and @jchris is looking at rebuilding the plugin on our jenkins CI, but unfortunately that build job is running into snags.

[jchris]

@arribbar mind testing with this zip download, which contains Traun's logging enhancements? You can use the download with the phonegap plugin install command by unzipping it and then giving the full path to the directory to the phonegap plugin install command, instead of the URL.

You may also be able to make the same changes in your project's CBLite.java file but it can be tricky to figure out which copy of the file to modify. All of them! :)

[arribbar]

Sorry, as I am working from London, it was a bit hard to work on it yesterday night ^^

I've done the changes in the CBLite.java as @tleyden has done, thanks to him. So here, in this gist https://gist.github.com/arribbar/00c672226a94a45dce78, you can find all the logs from the start of the application. I added the SG and nregp logs as well in case of.

Just after the error getting remote checkpoint, https://gist.github.com/arribbar/00c672226a94a45dce78#file-logcat-L147, you can see a failure but I can't say anything about that ...

[tleyden]

I think the problem is related to this line:

https://gist.github.com/arribbar/00c672226a94a45dce78#file-logcat-L136

I have an idea to further isolate it, but do you have an easy way to modify CBLite.java?

@jchris can you write up some instructions on how to do the "monkey patching" for the couchbase lite java plugin?

[arribbar]

Yes, I can modify easily the CBLite.java easily.

Just tell me what to do exactly :)

[tleyden]

@arribbar Here are the modifications (+ imports)

https://github.com/couchbaselabs/couchbase-lite-phonegap-plugin-builder/blob/feature/issue_22_replicator_dies/src/android/CBLite.java#L105-L119

[arribbar]

It seems to work fine like this. I tried four times without any problem...

The gist : https://gist.github.com/arribbar/16bd71cc39ad91b04d76

[arribbar]

I have updated the gist https://gist.github.com/arribbar/16bd71cc39ad91b04d76, adding the CBLite.java. I had to solve some bugs into the code you gave me, @tleyden. So i don't know if the code is working as you wanted or not.

[tleyden]

I had to solve some bugs into the code you gave me

Can you be more specific?

[arribbar]

You can see it on the gist I gave you on my last comment. I changed how to get the context of the activity. You can see it looking at the startCBLite function into the CBLite.Java file

[tleyden]

@arribbar ok I see what you mean, thanks for the heads up. Yes the changes you made look correct, since it's still short-circuiting the default NetworkReachabilityManager.

[tleyden]

I'm closing this issue in favor of https://github.com/couchbase/couchbase-lite-java-core/issues/248 since it does not appear to be phonegap-specific.

[tleyden]

@arribbar I closed https://github.com/couchbase/couchbase-lite-java-core/issues/248. I've added a unit test that proves that it works, so it would be great if you could remove the workaround from https://github.com/couchbaselabs/couchbase-lite-phonegap-plugin-builder/commit/e2f2d99db4c1854b1ff2f74f3a6bbc1eda6a7374 on a development version and re-test.

[arribbar]

@tleyden, sure, I could try but do I need to update the phonegap plugin, don't I? Because If I don't change anything except that, It shouldn't work, no ?

[tleyden]

@arribbar you'll need to run the latest master branch version of cbl android. I guess you need an updated zip file in order to rebuild the phonegap plugin, and unfortunately there is a build failure on Jenkins I need to fix before I can give that to you.

[tleyden]

@arribbar here is a recent zipfile:

http://factory.couchbase.com/view/build/view/mobile_dev/view/android/view/master/job/build_cblite_android_master-community/9/artifact/couchbase-lite-android-community_0.0.0-422.zip

The build is still marked as failing, but I think it's just an issue w/ our Jenkins configuration.

[arribbar]

@tleyden Sorry, I can't try it anymore as I miss one device to try it ...