OutOfMemoryError _bulk_docs

cesine commented 11 years ago

05-01 22:59:21.379: E/AndroidRuntime(15157): FATAL EXCEPTION: RemoteRequest-http://username:password@server.com:5984/usersdb/_bulk_docs 05-01 22:59:21.379: E/AndroidRuntime(15157): java.lang.OutOfMemoryError 05-01 22:59:21.379: E/AndroidRuntime(15157): at org.codehaus.jackson.util.ByteArrayBuilder.toByteArray(ByteArrayBuilder.java:154) 05-01 22:59:21.379: E/AndroidRuntime(15157): at org.codehaus.jackson.map.ObjectMapper.writeValueAsBytes(ObjectMapper.java:2106) 05-01 22:59:21.379: E/AndroidRuntime(15157): at com.couchbase.touchdb.support.TDRemoteRequest.run(TDRemoteRequest.java:121) 05-01 22:59:21.379: E/AndroidRuntime(15157): at java.lang.Thread.run(Thread.java:856)

cesine commented 11 years ago

05-01 23:06:54.149: E/dalvikvm-heap(15514): Out of memory on a 9828619-byte allocation. 05-01 23:06:54.149: I/dalvikvm(15514): "RemoteRequest-http://username:password@server.com:5984/usersdb/_bulk_docs" prio=5 tid=32 RUNNABLE 05-01 23:06:54.149: I/dalvikvm(15514): | group="main" sCount=0 dsCount=0 obj=0x423bdc20 self=0x956320 05-01 23:06:54.149: I/dalvikvm(15514): | sysTid=15606 nice=0 sched=0/0 cgrp=default handle=9198400 05-01 23:06:54.149: I/dalvikvm(15514): | schedstat=( 653644000 23730000 274 ) utm=65 stm=0 core=1 05-01 23:06:54.149: I/dalvikvm(15514): at org.codehaus.jackson.util.ByteArrayBuilder.toByteArray(ByteArrayBuilder.java:~154) 05-01 23:06:54.149: I/dalvikvm(15514): at org.codehaus.jackson.map.ObjectMapper.writeValueAsBytes(ObjectMapper.java:2106) 05-01 23:06:54.149: I/dalvikvm(15514): at com.couchbase.touchdb.support.TDRemoteRequest.run(TDRemoteRequest.java:121) 05-01 23:06:54.149: I/dalvikvm(15514): at java.lang.Thread.run(Thread.java:856) 05-01 23:06:54.149: W/dalvikvm(15514): threadid=32: thread exiting with uncaught exception (group=0x40c471f8) 05-01 23:06:54.149: E/AndroidRuntime(15514): FATAL EXCEPTION: RemoteRequest-http://username:password@server.com:5984/usersdb/_bulk_docs 05-01 23:06:54.149: E/AndroidRuntime(15514): java.lang.OutOfMemoryError 05-01 23:06:54.149: E/AndroidRuntime(15514): at org.codehaus.jackson.util.ByteArrayBuilder.toByteArray(ByteArrayBuilder.java:154) 05-01 23:06:54.149: E/AndroidRuntime(15514): at org.codehaus.jackson.map.ObjectMapper.writeValueAsBytes(ObjectMapper.java:2106) 05-01 23:06:54.149: E/AndroidRuntime(15514): at com.couchbase.touchdb.support.TDRemoteRequest.run(TDRemoteRequest.java:121) 05-01 23:06:54.149: E/AndroidRuntime(15514): at java.lang.Thread.run(Thread.java:856) 05-01 23:06:54.159: V/Pivot88(15514): HTML5 Replicating onPause

cesine commented 11 years ago

in the replicator-fixes branch:

                bodyBytes = TDServer.getObjectMapper().writeValueAsBytes(body);

05-01 23:15:32.409: E/dalvikvm-heap(15719): Out of memory on a 4188384-byte allocation.
05-01 23:15:32.409: I/dalvikvm(15719): "pool-4-thread-8" prio=5 tid=38 RUNNABLE
05-01 23:15:32.409: I/dalvikvm(15719):   | group="main" sCount=0 dsCount=0 obj=0x42f8e310 self=0x8934c0
05-01 23:15:32.409: I/dalvikvm(15719):   | sysTid=15848 nice=0 sched=0/0 cgrp=default handle=6048168
05-01 23:15:32.409: I/dalvikvm(15719):   | schedstat=( 1185927000 1678580000 23807 ) utm=67 stm=51 core=0
05-01 23:15:32.409: I/dalvikvm(15719):   at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:~82)
05-01 23:15:32.409: I/dalvikvm(15719):   at java.lang.StringBuilder.<init>(StringBuilder.java:67)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.util.TextBuffer.contentsAsString(TextBuffer.java:352)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:278)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:59)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2723)
05-01 23:15:32.409: I/dalvikvm(15719):   at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1900)
05-01 23:15:32.409: I/dalvikvm(15719):   at com.couchbase.touchdb.support.TDRemoteRequest.run(TDRemoteRequest.java:150)
05-01 23:15:32.409: I/dalvikvm(15719):   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
05-01 23:15:32.409: I/dalvikvm(15719):   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
05-01 23:15:32.409: I/dalvikvm(15719):   at java.lang.Thread.run(Thread.java:856)
05-01 23:15:32.929: W/dalvikvm(15719): threadid=38: thread exiting with uncaught exception (group=0x40c471f8)
05-01 23:15:32.949: E/AndroidRuntime(15719): FATAL EXCEPTION: pool-4-thread-8
05-01 23:15:32.949: E/AndroidRuntime(15719): java.lang.OutOfMemoryError
05-01 23:15:32.949: E/AndroidRuntime(15719):    at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:82)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at java.lang.StringBuilder.<init>(StringBuilder.java:67)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.util.TextBuffer.contentsAsString(TextBuffer.java:352)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:278)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:59)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:218)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:47)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2723)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1900)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at com.couchbase.touchdb.support.TDRemoteRequest.run(TDRemoteRequest.java:150)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
05-01 23:15:32.949: E/AndroidRuntime(15719):    at java.lang.Thread.run(Thread.java:856)

cesine commented 11 years ago

Choices

use smaller json objects, or find out why our objects are larger than 3MB
* I compacted the DB from 28MB down to 5, and now it appears to replicate without error.
see if GSON gives better performance, or iff the streaming in GSON could be used...
http://stackoverflow.com/questions/11385645/changing-from-gson-to-jackson-for-json-parsing
http://stackoverflow.com/questions/14323727/out-of-memory-error-while-parsing-a-large-json-using-jackson-library-on-android

cesine / TouchDB-Android

OutOfMemoryError _bulk_docs #2