Malformed queries towards google?

[rkgb]

If i try the call from the example: $gt->getAllOneKeyWord('Donald Trump')); I only get an empty error - and if i check the request directly - google states something like: "The server cannot process the request because it is malformed. It should not be retried."

{"widgets":[{"request":{"time":"2004-01-01 2024-08-06","resolution":"MONTH","locale":"en-US","comparisonItem":[{"geo":{"country":"US"},"complexKeywordsRestriction":{"keyword":[{"type":"BROAD","value":"Donald Trump"}]}}],"requestOptions":{"property":"","backend":"IZG","category":0},"userConfig":{"userType":"USER_TYPE_SCRAPER"}},"lineAnnotationText":"Search interest","bullets":[{"text":"Donald Trump"}],"showLegend":false,"showAverages":false,"helpDialog":{"title":"Interest over time","content":"Numbers represent search interest relative to the highest point on the chart for the given region and time. A value of 100 is the peak popularity for the term. A value of 50 means that the term is half as popular. A score of 0 means there was not enough data for this term."},"token":"APP6_UEAAAAAZrNJoFH3Ng3V6B0LUDp7x3eZxb45aKEx","id":"TIMESERIES","type":"fe_line_chart","title":"Interest over time","template":"fe","embedTemplate":"fe_embed","version":"1","isLong":true,"isCurated":false},{"request":{"geo":{"country":"US"},"comparisonItem":[{"time":"2004-01-01 2024-08-06","complexKeywordsRestriction":{"keyword":[{"type":"BROAD","value":"Donald Trump"}]}}],"resolution":"REGION","locale":"en-US","requestOptions":{"property":"","backend":"IZG","category":0},"userConfig":{"userType":"USER_TYPE_SCRAPER"}},"geo":"US","resolution":"provinces","searchInterestLabel":"Search interest","displayMode":"regions","helpDialog":{"title":"Interest by subregion","content":"See in which location your term was most popular during the specified time frame. Values are calculated on a scale from 0 to 100, where 100 is the location with the most popularity as a fraction of total searches in that location, a value of 50 indicates a location which is half as popular. A value of 0 indicates a location where there was not enough data for this term. \u003cp\u003e\u003cp\u003e \u003cb\u003eNote:\u003c/b\u003e A higher value means a higher proportion of all queries, not a higher absolute query count. So a tiny country where 80% of the queries are for \"bananas\" will get twice the score of a giant country where only 40% of the queries are for \"bananas\".","url":"https://support.google.com/trends/answer/4355212"},"color":"PALETTE_COLOR_1","index":0,"bullet":"Donald Trump","token":"APP6_UEAAAAAZrNJoAfemcWYMsbxiRTEK3S1l8pQYboy","id":"GEO_MAP","type":"fe_geo_chart_explore","title":"Interest by subregion","template":"fe","embedTemplate":"fe_embed","version":"1","isLong":true,"isCurated":false},{"request":{"restriction":{"geo":{"country":"US"},"time":"2004-01-01 2024-08-06","originalTimeRangeForExploreUrl":"all","complexKeywordsRestriction":{"keyword":[{"type":"BROAD","value":"Donald Trump"}]}},"keywordType":"ENTITY","metric":["TOP","RISING"],"trendinessSettings":{"compareTime":"2004-01-01 2005-01-01"},"requestOptions":{"property":"","backend":"IZG","category":0},"language":"en","userCountryCode":"DE","userConfig":{"userType":"USER_TYPE_SCRAPER"}},"helpDialog":{"title":"Related topics","content":"Users searching for your term also searched for these topics. You can view by the following metrics: \u003cp\u003e* \u003cb\u003eTop\u003c/b\u003e - The most popular topics. Scoring is on a relative scale where a value of 100 is the most commonly searched topic and a value of 50 is a topic searched half as often as the most popular term, and so on. \u003cp\u003e* \u003cb\u003eRising\u003c/b\u003e - Related topics with the biggest increase in search frequency since the last time period. Results marked \"Breakout\" had a tremendous increase, probably because these topics are new and had few (if any) prior searches."},"color":"PALETTE_COLOR_1","keywordName":"Donald Trump","token":"APP6_UEAAAAAZrNJoOKdzu3v8X5feIHhSmNlwjPAUTTY","id":"RELATED_TOPICS","type":"fe_related_searches","title":"Related topics","template":"fe","embedTemplate":"fe_embed","version":"1","isLong":false,"isCurated":false},{"request":{"restriction":{"geo":{"country":"US"},"time":"2004-01-01 2024-08-06","originalTimeRangeForExploreUrl":"all","complexKeywordsRestriction":{"keyword":[{"type":"BROAD","value":"Donald Trump"}]}},"keywordType":"QUERY","metric":["TOP","RISING"],"trendinessSettings":{"compareTime":"2004-01-01 2005-01-01"},"requestOptions":{"property":"","backend":"IZG","category":0},"language":"en","userCountryCode":"DE","userConfig":{"userType":"USER_TYPE_SCRAPER"}},"helpDialog":{"title":"Related queries","content":"Users searching for your term also searched for these queries. You can sort by the following metrics: \u003cp\u003e* \u003cb\u003eTop\u003c/b\u003e - The most popular search queries. Scoring is on a relative scale where a value of 100 is the most commonly searched query, 50 is a query searched half as often as the most popular query, and so on. \u003cp\u003e* \u003cb\u003eRising\u003c/b\u003e - Queries with the biggest increase in search frequency since the last time period. Results marked \"Breakout\" had a tremendous increase, probably because these queries are new and had few (if any) prior searches.","url":"https://support.google.com/trends/answer/4355000"},"color":"PALETTE_COLOR_1","keywordName":"Donald Trump","token":"APP6_UEAAAAAZrNJoKvGqzCU0FUAUaVANxfvbJ1lvrW4","id":"RELATED_QUERIES","type":"fe_related_searches","title":"Related queries","template":"fe","embedTemplate":"fe_embed","version":"1","isLong":false,"isCurated":false}],"keywords":[{"keyword":"Donald Trump","name":"Donald Trump","type":"Search term"}],"timeRanges":["2004 - present"],"examples":[],"shareText":"Explore search interest for Donald Trump by time, location and popularity on Google Trends","shouldShowMultiHeatMapMessage":false} 2024/08/06 12:08:04 [info] [application] [127.0.0.1] [] <html lang=en><meta charset=utf-8><meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"><title>Error 429 (Bad Request)!!1</title><style nonce="wFoswnVGCBt_WYLGujjrAg">*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{color:#222;text-align:unset;margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px;}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}pre{white-space:pre-wrap;}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}</style><main id="af-error-container" role="main"><a href=//www.google.com><span id=logo aria-label=Google role=img></span></a><p><b>429.</b> <ins>That’s an error.</ins><p>The server cannot process the request because it is malformed. It should not be retried. <ins>That’s all we know.</ins></main> 2024/08/06 12:08:05 [info] [application] [127.0.0.1] [] <html lang=en><meta charset=utf-8><meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"><title>Error 429 (Bad Request)!!1</title><style nonce="UdmFG3TCeT5HUNx5tqfRlg">*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{color:#222;text-align:unset;margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px;}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}pre{white-space:pre-wrap;}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}</style><main id="af-error-container" role="main"><a href=//www.google.com><span id=logo aria-label=Google role=img></span></a><p><b>429.</b> <ins>That’s an error.</ins><p>The server cannot process the request because it is malformed. It should not be retried. <ins>That’s all we know.</ins></main> 2024/08/06 12:08:05 [info] [application] [127.0.0.1] [] <html lang=en><meta charset=utf-8><meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"><title>Error 429 (Bad Request)!!1</title><style nonce="3kLVHDbgKAoNLNRz9R4KmA">*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{color:#222;text-align:unset;margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px;}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}pre{white-space:pre-wrap;}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}</style><main id="af-error-container" role="main"><a href=//www.google.com><span id=logo aria-label=Google role=img></span></a><p><b>429.</b> <ins>That’s an error.</ins><p>The server cannot process the request because it is malformed. It should not be retried. <ins>That’s all we know.</ins></main> 2024/08/06 12:08:05 [info] [application] [127.0.0.1] [] <html lang=en><meta charset=utf-8><meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"><title>Error 429 (Bad Request)!!1</title><style nonce="hEaBh-MeV1RrR1P8CZWodQ">*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{color:#222;text-align:unset;margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px;}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}pre{white-space:pre-wrap;}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}</style><main id="af-error-container" role="main"><a href=//www.google.com><span id=logo aria-label=Google role=img></span></a><p><b>429.</b> <ins>That’s an error.</ins><p>The server cannot process the request because it is malformed. It should not be retried. <ins>That’s all we know.</ins></main>

Other calls work quite fine like $gt->getDailySearchTrends();

Is this a general issue?

[jabeztadesse]

+1

[rahuljc]

If you look for the property 'userType' in your pasted JSON response, you can see that it has detected you as a scraper ('USER_TYPE_SCRAPER'). The ideal response should have userType as 'USER_TYPE_LEGIT_USER'

Maybe you tried too many requests that Google detected you as a scraper. You can give some time before making another request. Although, I am not sure what algorithm does Google use to determine a scraper.

[jabeztadesse]

If you look for the property 'userType' in your pasted JSON response, you can see that it has detected you as a scraper ('USER_TYPE_SCRAPER'). The ideal response should have userType as 'USER_TYPE_LEGIT_USER'

Maybe you tried too many requests that Google detected you as a scraper. You can give some time before making another request. Although, I am not sure what algorithm does Google use to determine a scraper.

I don't think it's a rate limit ...

[rahuljc]

@jabeztadesse , You are right. It's not related to the number of requests. I think Google has updated their algorithm to detect a scraper ? There are issues being raised in pytrends project as well