search

avinashjoshi / walkaround

Automatically exported from code.google.com/p/walkaround
Apache License 2.0
0 stars 1 forks source link

Exception during re-index map reuce job #91

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
There's following exception in the wavereactor-hr log related to the re-index 
map reduce job.

2012-04-02 00:51:47.942
com.google.walkaround.slob.server.MutationLog reconstruct: Reconstructed 
requested version null from snapshot at 7 followed by 1 deltas
I 2012-04-02 00:51:47.942
com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler$1 run: 
Re-indexing SlobId(lmTba_6WnUYL7epj) at version 8
I 2012-04-02 00:51:47.942
com.google.walkaround.wave.server.wavemanager.WaveIndex update: Updating index 
for SlobId(lmTba_6WnUYL7epj): IndexEntry(SlobId(lmTba_6WnUYL7epj), 
wim@fixnum.org, ,     

, 1332402896247, [wim@fixnum.org])
I 2012-04-02 00:51:47.959
com.google.walkaround.util.server.RetryHelper runBodyOnce: Body exited 
normally, run time: 57ms
I 2012-04-02 00:51:47.959
com.google.walkaround.wave.server.wavemanager.ReIndexMapper map: Re-indexing 
Wavelet("lm_zZygomxSqKFYD")
I 2012-04-02 00:51:48.294
com.google.walkaround.util.server.RetryHelper runBodyOnce: Running body 
VoidBodyWrapper(com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Hand
ler$1@5476bf)
I 2012-04-02 00:51:48.299
com.google.walkaround.util.server.appengine.CheckedDatastore$1 run: Begun 
transaction CheckedTransactionImpl(Txn [s~wavereactor-hr.-7913504055520821756, 
BEGUN] (class com.google.appengine.api.datastore.TransactionImpl@1c4c801))
I 2012-04-02 00:51:48.316
com.google.walkaround.slob.server.MutationLog getMetadata: Got {"1":0}
I 2012-04-02 00:51:48.324
com.google.walkaround.slob.server.MutationLog getSnapshotEntryAtOrBefore: query 
SELECT * FROM WaveletSnapshot WHERE __ancestor__ is Wavelet("lm_zZygomxSqKFYD") 
ORDER BY __key__ DESC returned first result null
W 2012-04-02 00:51:51.004
com.google.appengine.api.datastore.QueryResultsSourceImpl logChunkSizeWarning: 
This query does not have a chunk size set in FetchOptions and has returned over 
1000 results.  If result sets of this size are common for this query, consider 
setting a chunk size to improve performance.
  To disable this warning set the following system property in appengine-web.xml (the value of the property doesn't matter): 'appengine.datastore.disableChunkSizeWarning'
I 2012-04-02 00:51:53.459
com.google.walkaround.util.server.RetryHelper runBodyOnce: Body exited 
abnormally, run time: 5165ms
W 2012-04-02 00:51:53.459
com.google.walkaround.util.server.RetryHelper run: Problem on retry 0, millis 
elapsed so far: 5165
com.google.walkaround.util.server.RetryHelper$RetryableFailure: 
com.google.appengine.api.datastore.DatastoreTimeoutException: The datastore 
operation timed out, or the data was temporarily unavailable.
    at com.google.walkaround.util.server.appengine.CheckedDatastore.safeRun(CheckedDatastore.java:458)
    at com.google.walkaround.util.server.appengine.CheckedDatastore.access$100(CheckedDatastore.java:67)
    at com.google.walkaround.util.server.appengine.CheckedDatastore$CheckedIteratorImpl.hasNext(CheckedDatastore.java:109)
    at com.google.walkaround.slob.server.MutationLog$DeltaIterator.hasNext(MutationLog.java:321)
    at com.google.walkaround.slob.server.MutationLog.reconstruct(MutationLog.java:858)
    at com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler$1.run(ReIndexMapper.java:77)
    at com.google.walkaround.util.server.RetryHelper$3.run(RetryHelper.java:182)
    at com.google.walkaround.util.server.RetryHelper$3.run(RetryHelper.java:180)
    at com.google.walkaround.util.server.RetryHelper.runBodyOnce(RetryHelper.java:142)
    at com.google.walkaround.util.server.RetryHelper.run(RetryHelper.java:156)
    at com.google.walkaround.util.server.RetryHelper.run(RetryHelper.java:180)
    at com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler.process(ReIndexMapper.java:62)
    at com.google.walkaround.wave.server.wavemanager.ReIndexMapper.map(ReIndexMapper.java:94)
    at com.google.walkaround.wave.server.wavemanager.ReIndexMapper.map(ReIndexMapper.java:51)
    at com.google.appengine.tools.mapreduce.v2.impl.handlers.Worker.processMapper(Worker.java:111)
    at com.google.appengine.tools.mapreduce.v2.impl.handlers.Worker.handleMapperWorker(Worker.java:289)
    at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost(MapReduceServlet.java:190)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.walkaround.util.server.servlet.RequestStatsFilter.doFilter(RequestStatsFilter.java:95)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.walkaround.wave.server.servlet.ServerExceptionFilter.doFilter(ServerExceptionFilter.java:108)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.appengine.tools.appstats.AppstatsFilter.doFilter(AppstatsFilter.java:141)
    at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:102)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)
    at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:446)
    at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
    at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
    at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
    at java.lang.Thread.run(Thread.java:679)
Caused by: com.google.appengine.api.datastore.DatastoreTimeoutException: The 
datastore operation timed out, or the data was temporarily unavailable.
    at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:46)
    at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:76)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:106)
    at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72)
<continued in next message>
W 2012-04-02 00:51:53.459
<continued from previous message>
at 
com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
    at com.google.appengine.api.datastore.QueryResultsSourceImpl.peekQueryResultAndIfFirstRecordIndexList(QueryResultsSourceImpl.java:172)
    at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:108)
    at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:95)
    at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:161)
    at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:65)
    at com.google.walkaround.util.server.appengine.CheckedDatastore$CheckedIteratorImpl$1.run(CheckedDatastore.java:111)
    at com.google.walkaround.util.server.appengine.CheckedDatastore$CheckedIteratorImpl$1.run(CheckedDatastore.java:109)
    at com.google.walkaround.util.server.appengine.CheckedDatastore.safeRun(CheckedDatastore.java:456)
    ... 82 more
I 2012-04-02 00:51:53.460
com.google.walkaround.util.server.RetryHelper run: Sleeping for 3 millis

Original issue reported on code.google.com by vega113 on 2 Apr 2012 at 7:59

GoogleCodeExporter commented 9 years ago 
I hope 
http://code.google.com/p/walkaround/source/detail?r=ee3b32bed5ad9f58147036ea1ad6
099ebbbec33f fixes this, but if not, please re-open this bug.

Original comment by oh...@google.com on 3 Apr 2012 at 6:30