search

google / upvote_py2

A multi-platform binary whitelisting solution
Apache License 2.0
452 stars 35 forks source link

500 server error from /_ah/queue/deferred? #67

Closed ivwang closed 3 years ago

ivwang commented 3 years ago

Hi,

Just recently I deployed an upvote (built with bazel-0.25.0, since this is more in line with date of last commit to this repo). Overall the app functions ok, Santa clients are able to sync. GAE cron jobs also run successfully.

Though looking into logs, there are quite a few 500 errors from /_ah/queue/deferred handler with backtraces like below

Is this something expected with current repo?

Thanks a lot Ivan.

backtrace:

'tuple' object has no attribute 'get_async'
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 316, in post
    self.run_from_request()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 311, in run_from_request
    run(self.request.body)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 153, in run
    return func(*args, **kwds)
  File "/base/data/home/apps/u~proj-id/20210111t191206.432278677749216289/upvote/gae/datastore/utils.py", line 375, in _QueuedPaginatedBatchApply
    callback(last_page_results, *extra_args, **extra_kwargs)
  File "/base/data/home/apps/u~proj-id/20210111t191206.432278677749216289/upvote/gae/cron/role_syncing.py", line 269, in _SpiderBite
    hosts = ndb.get_multi(host_keys)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3933, in get_multi
    for future in get_multi_async(keys, **ctx_options)]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3918, in get_multi_async
    return [key.get_async(**ctx_options) for key in keys]
AttributeError: 'tuple' object has no attribute 'get_async'
ivwang commented 3 years ago

Ok.. after observing for weeks, it looks like caused by residual tasks queued by lockspider from previous unsuccessful (?) deployments. After purging query queue those 500s haven't come back. I'll close the issue.

Thanks Ivan.