search

catchpoint / WebPageTest.bulk-tester

Google App Script for spreadsheet that uses the WPT API to bulk test URLs
MIT License
260 stars 22 forks source link

Getting SyntaxError: Unterminated string literal error #17

Open desmondtam opened 8 years ago

desmondtam commented 8 years ago

Hi Andy,

Because of the 200 test cap per API key, I sometimes provision a series of tests and use the WPT-Bulk-Tester sheet to fetch the results by pasting the WPT results URL to the WPT ID column and running Get Results function. Recently, the sheet seems to be failing with the following error:

SyntaxError: Unterminated string literal

It correctly parses and fetches the results of other (older) WPT results though.

Here is one example WPT results ID where it fails:

https://www.webpagetest.org/result/161003_10_5327eec3e3713d1a1a3d75f233666a36/

Any idea why?

andydavies commented 8 years ago

Thanks for the report I'll take a look on the weekend

andydavies commented 8 years ago

The appscript command that parses the JSON is failing, but the JSON coming back from WPT looks valid so I need to dig in more.

javiergarza commented 7 years ago

I am getting the same errors. I believe the problem is the JSON returned is returned in chunks and the UrlFetchApp.fetch function may just "think" its done after receiving the first chunk and return the incomplete JSON body causing that error.

If you check the execution transcript you will see something like:

[17-07-28 18:26:29:043 PDT] Sheet.getRange([M1]) [0.053 seconds] [17-07-28 18:26:29:044 PDT] Range.setValue([http://www.webpagetest.org/result/170728_S5_8ed3adcafa935c40f214e82b2bd6af0e/?f=json]) [0 seconds] [17-07-28 18:26:30:385 PDT] UrlFetchApp.fetch([http://www.webpagetest.org/result/170728_S5_8ed3adcafa935c40f214e82b2bd6af0e/?f=json, {muteHttpExceptions=true}]) [1.34 seconds] [17-07-28 18:26:30:398 PDT] HTTPResponse.getContentText() [0.012 seconds] [17-07-28 18:26:31:003 PDT] Execution failed: SyntaxError: Unterminated string literal (line 186, file "webpagetest") [2.063 seconds total runtime]

Here is the code (marking in bold the code that triggers that error): SpreadsheetApp.getActiveSheet().getRange('M1').setValue(wptAPI); var response = UrlFetchApp.fetch(wptAPI); var result = JSON.parse(response.getContentText());

I tried adding the optional parameter "muteHttpExceptions: true" to the UrlFetchApp.fetch call but didn't fix the issue var response = UrlFetchApp.fetch( wptAPI, { muteHttpExceptions: true, } );

Any ideas? (This error comes randomly on different rows, but once it triggers it does not go away)

javiergarza commented 7 years ago

I did some debugging (added the Logger line below) and I can confirm in the logs the JSON body is truncated at around 8KB (no idea why)

  var response = UrlFetchApp.fetch(wptAPI);
  Logger.log(response.getContentText());
  var result = JSON.parse(response.getContentText());

Logs: [17-07-28 18:47:30:973 PDT] {"data":{"id":"170728_S5_8ed3adcafa935c40f214e82b2bd6af0e","url":"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","summary":"http:\/\/www.webpagetest.org\/results.php?test=170728_S5_8ed3adcafa935c40f214e82b2bd6af0e","testUrl":"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","location":"ec2-us-west-1:Chrome","from":"California USA - EC2 - Chrome<\/b> - 3GFast<\/b>","connectivity":"3GFast","bwDown":1600,"bwUp":768,"latency":150,"plr":"0","mobile":0,"completed":1501283907,"tester":"i-72d75ac7","testerDNS":"172.31.0.2","runs":{"1":{"firstView":{"numSteps":1,"run":1,"tester":"i-72d75ac7","URL":"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","loadTime":39820,"TTFB":1035,"bytesOut":422499,"bytesOutDoc":352961,"bytesIn":6387013,"bytesInDoc":5950643,"connections":87,"requests":[{"ip_addr":"23.34.148.155","method":"GET","host":"www.playstation.com","url":"\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","responseCode":200,"load_ms":550,"ttfb_ms":360,"load_start":675,"bytesOut":347,"bytesIn":28640,"objectSize":28364,"cacheControl":"max-age=9526","contentType":"text\/html","contentEncoding":"gzip","type":3,"socket":10,"score_cache":-1,"score_cdn":-1,"score_gzip":100,"score_cookies":-1,"score_keep-alive":100,"score_minify":-1,"score_combine":-1,"score_compress":-1,"score_etags":-1,"is_secure":1,"dns_ms":-1,"connect_ms":153,"ssl_ms":198,"gzip_total":28364,"gzip_save":0,"minify_total":0,"minify_save":0,"image_total":0,"image_save":0,"cache_time":-1,"dns_start":0,"dns_end":263,"connect_start":263,"connect_end":416,"ssl_start":417,"ssl_end":615,"server_count":1,"server_rtt":153,"client_port":61483,"jpeg_scan_count":0,"priority":"VeryHigh","request_id":1,"was_pushed":0,"initiator_type":"other","initiator_detail":"{\"type\":\"other\"}","protocol":"HTTP\/2","http2_stream_id":1,"http2_stream_dependency":0,"http2_stream_weight":256,"http2_stream_exclusive":1,"certificate_bytes":4198,"objectSizeUncompressed":194759,"full_url":"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","score_progressive_jpeg":-1,"load_end":1225,"ttfb_start":675,"ttfb_end":1035,"download_start":1035,"download_end":1225,"download_ms":190,"all_start":263,"all_end":1225,"all_ms":901,"headers":{"request":[":method: GET",":authority: www.playstation.com",":scheme: https",":path: \/en-us\/games\/uncharted-the-lost-legacy-ps4\/","upgrade-insecure-requests: 1","user-agent: Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/60.0.3112.78 Safari\/537.36 PTST\/387","accept: text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/webp,image\/apng,\/;q=0.8","accept-encoding: gzip, deflate, br","accept-language: en-US,en;q=0.8","pragma: akamai-x-cache-on, akamai-x-check-cacheable, akamai-x-get-true-cache-key, akamai-x-cache-remote-on"],"response":[":status: 200","server: Apache","content-type: text\/html; charset=UTF-8","x-check-cacheable: YES","x-akamai-staging: ESSL","vary: Accept-Encoding","content-encoding: gzip","cache-control: max-age=9526","date: Fri, 28 Jul 2017 23:13:50 GMT","content-length: 28364","x-cache: TCP_HIT from a23-50-49-29.deploy.akamaitechnologies.com (AkamaiGHost\/9.0.2.0.1-20438731) (-)","x-true-cache-key: \/L\/origin-us.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/"]},"index":0,"number":1},{"ip_addr":"23.34.148.155","method":"GET","host":"www.playstation.com","url":"\/en-us\/etc\/designs\/pdc\/clientlibs_audiencemanager\/js\/sne_cloud_id.js","responseCode":200,"load_ms":246,"ttfb_ms":199,"load_start":1165,"bytesOut":124,"bytesIn":12138,"objectSize":11787,"expires":"Fri, 28 Jul 2017 23:43:50 GMT","cacheControl":"max-age=1800","contentType":"application\/javascript","contentEncoding":"gzip","type":3,"socket":10,"score_cache":0,"score_cdn":0,"score_gzip":100,"score_cookies":-1,"score_keep-alive":100,"score_minify":-1,"score_combine":-1,"score_compress":-1,"score_etags":-1,"is_secure":1,"dns_ms":-1,"connect_ms":-1,"ssl_ms":-1,"gzip_total":11787,"gzip_save":0,"minify_total":0,"minify_save":0,"image_total":0,"image_save":0,"cache_time":1800,"dns_start":0,"dns_end":0,"connect_start":0,"connect_end":0,"ssl_start":0,"ssl_end":0,"initiator":"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","initiator_line":7,"server_count":1,"server_rtt":153,"client_port":61483,"jpeg_scan_count":0,"priority":"High","request_id":2,"was_pushed":0,"initiator_type":"parser","initiator_detail":"{\"lineNumber\":7,\"type\":\"parser\",\"url\":\"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/\"}","protocol":"HTTP\/2","http2_stream_id":3,"http2_stream_dependency":1,"http2_stream_weight":220,"http2_stream_exclusive":1,"certificate_bytes":0,"objectSizeUncompressed":29970,"full_url":"https:\/\/www.playstation.com\/en-us\/etc\/designs\/pdc\/clientlibs_audiencemanager\/js\/sne_cloud_id.js","score_progressive_jpeg":-1,"load_end":1411,"ttfb_start":1165,"ttfb_end":1364,"download_start":1364,"download_end":1411,"download_ms":47,"all_start":1165,"all_end":1411,"all_ms":246,"headers":{"request":[":method: GET",":authority: www.playstation.com",":scheme: https",":path: \/en-us\/etc\/designs\/pdc\/clientlibs_audiencemanager\/js\/sne_cloud_id.js","user-agent: Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/60.0.3112.78 Safari\/537.36 PTST\/387","accept: \/","referer: https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","accept-encoding: gzip, deflate, br","accept-language: en-US,en;q=0.8","pragma: akamai-x-cache-on, akamai-x-check-cacheable, akamai-x-get-true-cache-key, akamai-x-cache-remote-on"],"response":[":status: 200","accept-ranges: bytes","content-type: application\/javascript","last-modified: Fri, 21 Jul 2017 19:30:42 GMT","server: Apache","vary: Accept-Encoding","content-encoding: gzip","cache-control: max-age=1800","expires: Fri, 28 Jul 2017 23:43:50 GMT","date: Fri, 28 Jul 2017 23:13:50 GMT","content-length: 11787","x-cache: TCP_MEM_HIT from a23-50-49-29.deploy.akamaitechnologies.com (AkamaiGHost\/9.0.2.0.1-20438731) (-)","x-true-cache-key: \/L\/origin-us.playstation.com\/en-us\/etc\/designs\/pdc\/clientlibs_audiencemanager\/js\/sne_cloud_id.js","x-akamai-staging: ESSL","x-check-cacheable: YES"]},"index":1,"number":2},{"ip_addr":"23.34.148.155","method":"GET","host":"www.playstation.com","url":"\/en-us\/etc\/clientlibs\/granite\/jquery.min.js","responseCode":200,"load_ms":1232,"ttfb_ms":309,"load_start":1168,"bytesOut":52,"bytesIn":38183,"objectSize":37851,"expires":"Fri, 28 Jul 2017 23:43:50 GMT","cacheControl":"max-age=1800","contentType":"application\/javascript","contentEncoding":"gzip","type":3,"socket":10,"score_cache":0,"score_cdn":0,"score_gzip":100,"score_cookies":-1,"score_keep-alive":100,"score_minify":-1,"score_combine":-1,"score_compress":-1,"score_etags":-1,"is_secure":1,"dns_ms":-1,"connect_ms":-1,"ssl_ms":-1,"gzip_total":37851,"gzip_save":0,"minify_total":0,"minify_save":0,"image_total":0,"image_save":0,"cache_time":1800,"dns_start":0,"dns_end":0,"connect_start":0,"connect_end":0,"ssl_start":0,"ssl_end":0,"initiator":"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/","initiator_line":165,"server_count":1,"server_rtt":153,"client_port":61483,"jpeg_scan_count":0,"priority":"High","request_id":3,"was_pushed":0,"initiator_type":"parser","initiator_detail":"{\"lineNumber\":165,\"type\":\"parser\",\"url\":\"https:\/\/www.playstation.com\/en-us\/games\/uncharted-the-lost-legacy-ps4\/\"}","protocol":"HTTP\/2","http2_stream_id":5,"http2_stream_dependency":3,"http2_stream_weight":220,"http2_stream_exclusive":1,"certificate_bytes":0,"objectSizeUncompressed":110890,"full_url":"https:\/\/www.playstation.com\/en-us\/etc\/clientlibs\/granite\/jquery.min.js","score_progressive_jpeg":-1,"load_end":2400,"ttfb_start":1168,"ttfb_end":1477,"download_start":1477,"download_end":2400,"download_ms":923,"all_start":1168,"all_end":2400,"all_ms":1232,"headers":{"request":[":method: GET",":authority: www.playstation.com",":scheme: https",":path: \/en-us\/etc\/clientlibs\/granite\/jquery.min.js","use

andydavies commented 7 years ago

@javiergarza Thanks for digging into this - I'll try to take a look at it over the next week or so

JasonCoe-FL commented 7 years ago

I'm curious if this issue has been looked into any further as I'm running into the same thing frequently but only on certain tests. As @javiergarza stated, the response is limited to 8kb which sticks out to me only because I know it's a common practice by CDN's, servers, and hosting providers to limit the response headers to a max of 8kb. Not sure if it's related but I'm curious if there's a correlation.

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.