search

argasi / google-bigquery

Automatically exported from code.google.com/p/google-bigquery
0 stars 0 forks source link

Nearly all BQ loads seem to be failing with 502/503. #47

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Attempt to load a JSON file via the BQ tool.
2. For example: 
3. bq --headless --nosync load --source_format NEWLINE_DELIMITED_JSON 
hgs.hgs_20131030_00 monwork/monwork-worker01.tmp

What is the expected output? What do you see instead?

I expect to see a message that the load succeeded. Instead I recieve either a 
502 or 503. Examples of results:

[2013-10-29 18:57:44] <worker01> < Command result 1: 'BigQuery error in load 
operation: Could not connect with BigQuery server.\nHttp response status: 
502\nHttp response content:\n<!DOCTYPE html>\n<html lang=en>\n<meta 
charset=utf-8>\n<meta name=viewport content="initial-scale=1, 
minimum-scale=1,\nwidth=device-width">\n<title>Error 502 (Server 
Error)!!1</title>\n<style>\n*{margin:0;padding:0}html,code{font:15px/22px\narial
,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%\nauto 
0;max-width:390px;min-height:180px;padding:30px 0 15px}* 
>\nbody{background:url(//www.google.com/images/errors/robot.png) 100% 
5px\nno-repeat;padding-right:205px}p{margin:11px 
0\n22px;overflow:hidden}ins{color:#777;text-decoration:none}a 
img{border:0}@media\nscreen and 
(max-width:772px){body{background:none;margin-top:0;max-width:none;pa\ndding-rig
ht:0}}\n</style>\n<a href=//www.google.com/><img 
src=//www.google.com/images/errors/logo_sm.gif\nalt=Google></a>\n<p><b>502.</b> 
<ins>That\\ufffd\\ufffd\\ufffds an error.</ins>\n<p>The server encountered a 
temporary error and could not complete your\nrequest.<p>Please try again in 30 
seconds. <ins>That\\ufffd\\ufffd\\ufffds all we\nknow.</ins>'

[2013-10-29 19:00:44] <worker09> < Command result 1: 'BigQuery error in load 
operation: Could not connect with BigQuery server.\nHttp response status: 
503\nHttp response content:'

Additionally, the very few jobs that are successfully submitted generally fail. 
Examples:

bqjob_r3a054f09f7c6b47_0000014205e35217_1 - failed with "Connection error. 
Please try again."
bqjob_r105ecaaa4269450c_0000014205caa77a_1 - failed with "Unexpected. Please 
try again."

What version of the product are you using? On what operating system?

Tried the BigQuery CLI versions 2.0.15 and 2.0.17 with identical results on 
CentOS 5.

Please provide any additional information below.

Previously this seemed to be working fine. Then, over the past couple days I 
noted some 502/503 failures that would occur sporadically and some instances 
where it seemed to occur consistently for 30-90min. All load requests seem to 
have failed today.

According to the documentation any sort of quota issues should cause a 4xx 
error not 5xx, so I don't believe the problem could be that. Additionally, we 
should be submitting load requests considerably below the threshold for 
throttling. We use one BQ table per hour, and roughly 2 loads per minute (when 
BQ is actually handling requests). This is 120 loads per table/day (limit is 
1,000) and 2,880 loads per day (limit is 10,000).

I am attaching logs from the tool we use to submit queries. Hopefully the 
format will be self-explanatory. The log will show the exact command executed 
and the exact output from the BQ tool.

I will also attach a small excerpt of the files we are submitting.

We just deployed the project that depended on this data for production, so any 
help would be appreciated!

Original issue reported on code.google.com by vulp...@teliax.com on 30 Oct 2013 at 1:21

Attachments:

GoogleCodeExporter commented 9 years ago 
I also experienced the same issue. Every upload request returns 503 error 
status.

Original comment by ngu...@demandstreet.com on 30 Oct 2013 at 6:42

GoogleCodeExporter commented 9 years ago 
Sorry for the trouble--I'm investigating this issue now.

Are all of these requests direct uploads, rather than imports from Google Cloud 
Storage? If it's practical, one workaround to consider is to try loading via 
GCS instead.

I'll update this bug when I learn more.

Original comment by jcon...@google.com on 30 Oct 2013 at 7:01

GoogleCodeExporter commented 9 years ago 
I've confirmed that this issue affects only the direct upload path. Your best 
bet for working around the problem is to stage your data in Google Cloud 
Storage (via gsutil) and import from there.

You can also try splitting your data into smaller chunks, which will likely 
increase your chances of uploads succeeding, though it's no guarantee.

I have some ideas about the root problem and will work on getting a better fix 
in place tomorrow. Apologies again for the trouble!

Original comment by jcon...@google.com on 30 Oct 2013 at 8:19

GoogleCodeExporter commented 9 years ago 
This problem should now be addressed. Please let us know if you still see any 
problems!

Original comment by jcon...@google.com on 30 Oct 2013 at 2:31

GoogleCodeExporter commented 9 years ago 
This was a much faster response than I expected to get here. All our requests 
are direct uploads, and they seem to be humming along now. Thank you so much, 
semi-anonymous Google person!

Original comment by vulp...@teliax.com on 30 Oct 2013 at 3:07

GoogleCodeExporter commented 9 years ago 
No problemo! Thanks for the detailed bug report. :-)

Original comment by jcon...@google.com on 30 Oct 2013 at 5:53

GoogleCodeExporter commented 9 years ago 
Wow you guys are really fast in responding. I have tried the direct upload and 
it works. Thank you so much.

Original comment by ngu...@demandstreet.com on 30 Oct 2013 at 5:56