search

cloudflare / cloudflare-gcp

Google Cloud Function to push json files from GC Storage to Big Query
Apache License 2.0
84 stars 49 forks source link

Quota exceeded: Your table exceeded quota for imports or query appends per table. #72

Closed mattcobb closed 3 years ago

mattcobb commented 3 years ago

Setup a logpush to this gcp cloud function. Seemed to be working fine, but then we couldn't find some RayIDs we saw in our logs. On further investigation, we found the google function was logging the quota exceeded error below. The table insert quota is 1500/table/day. CF is creating a log every 30secs which is 2880/day so the quota gets exceeded. Is there a way to create a file no more than once per min (1440/day). Separately we are looking into doubling the Google quota.

Default 2021-05-06 15:07:52.007 PDTgcsbq2ddmbm47hmmcn Starting job for logs/20210506/20210506T220721Z_20210506T220751Z_477368e1.log.gz Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn { Error: Quota exceeded: Your table exceeded quota for imports or query appends per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn id: Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn at request (/workspace/node_modules/@google-cloud/bigquery/build/src/index.js:888:23) Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn 'pryon-demo-dev-215016:US.e6ef4b65-8472-4ab8-8142-6b5b4e568b1d', Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn at Util.handleResp (/workspace/node_modules/@google-cloud/common/build/src/util.js:141:9) Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn selfLink: Default 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn 'https://www.googleapis.com/bigquery/v2/projects/pryon-demo-dev-215016/jobs/e6ef4b65-8472-4ab8-8142-6b5b4e568b1d?location=US' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn at retryRequest (/workspace/node_modules/@google-cloud/common/build/src/util.js:422:22) 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn at onResponse (/workspace/node_modules/retry-request/index.js:200:7) 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn user_email: 'pryon-demo-dev-215016@appspot.gserviceaccount.com' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn configuration: { load: [Object], jobType: 'LOAD' } 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn at /workspace/node_modules/teeny-request/build/src/index.js:222:13 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn jobReference: 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn at process._tickCallback (internal/process/next_tick.js:68:7) 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn code: undefined, 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn { projectId: 'pryon-demo-dev-215016' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn jobId: 'e6ef4b65-8472-4ab8-8142-6b5b4e568b1d' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn errors: 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn [ { reason: 'quotaExceeded' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn location: 'US' } 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn location: 'load_job_per_table.long', 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn statistics: 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn message: 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn { creationTime: '1620338872353' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn startTime: '1620338872502' 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn 'Quota exceeded: Your table exceeded quota for imports or query appends per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors' } ], 2021-05-06 15:07:52.591 PDTgcsbq2ddmbm47hmmcn endTime: '1620338872502' },

shagamemnon commented 3 years ago

Hey @mattcobb - we have you covered with https://github.com/cloudflare/cloudflare-gcp/pull/70 :)

Should be released in the next few days.

mattcobb commented 3 years ago

Thank you!

mattcobb commented 3 years ago

@shagamemnon Any update on #70 release date?

openspacetrust commented 3 years ago

Hey there! I am having the same issue and I was wondering if there were any updates. Thanks!

mattcobb commented 3 years ago

It has been a month, any update on this? This feature is currently not viable as the quota kicks in around half way thru the day.