search

lnklnklnk / ga-bq

Stream raw hit-level Google Analytics data into BigQuery
90 stars 38 forks source link

split pull-queue #25

Open antonzol opened 4 years ago

antonzol commented 4 years ago

If tasks in queue there too much cron cant process them all. Errol log there. I thing that better is to split queue in to packages.

<HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/kuluarpohod-147905/datasets/GAfirst/tables/GAhits/insertAll?alt=json returned "The row insert id length 1182 is too long."> (/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py:1552)
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7cb976f64e72c78c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~kuluarpohod-147905/ga-tracker:20200103t165006.423608064662716982/process_queue.py", line 29, in get
    bq_loader.insert_rows(rows)
  File "/base/data/home/apps/s~kuluarpohod-147905/ga-tracker:20200103t165006.423608064662716982/bqloader.py", line 202, in insert_rows
    body=body).execute()
  File "/base/data/home/apps/s~kuluarpohod-147905/ga-tracker:20200103t165006.423608064662716982/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/base/data/home/apps/s~kuluarpohod-147905/ga-tracker:20200103t165006.423608064662716982/apiclient/http.py", line 723, in execute
    raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/kuluarpohod-147905/datasets/GAfirst/tables/GAhits/insertAll?alt=json returned "The row insert id length 1182 is too long.">
Glomen commented 4 years ago

I have the same errors, how do I stop them ?