search

SSujitX / google-news-url-decoder

A Python script to decode Google News article URLs.
MIT License
56 stars 3 forks source link

HTTP 429 too many requests #3

Open Clement-Lelievre opened 4 weeks ago

Clement-Lelievre commented 4 weeks ago

Hi

thank you for your code.

This line : https://github.com/SSujitX/google-news-url-decoder/blob/main/decode_google_news_url_v2.py#L19 when used multiple times in a row eg in a loop, quickly gets a 429 error. Do you know of any workaround?

Thank you

SSujitX commented 4 weeks ago

@Clement-Lelievre Share your code, let me check.

Clement-Lelievre commented 4 weeks ago

@Clement-Lelievre Share your code, let me check.

Cannot atm, but just iterate over your code say in à for loop 50 times

Minus-Sinus commented 4 weeks ago

I have the same issue. It seems the google server is shutting down after a certain number of requests within a time span - so the code is not wrong, but google prevents scripted high-volume use. Work-around could be a wait period between requests:

from time import sleep 

# Your loop of requests:
for url in urls:
    # ...
    sleep(2) # Wait for 2 seconds.

in your iteration @Clement-Lelievre . Don't know if 2 seconds is long enough though, you need to try it out

SSujitX commented 4 weeks ago

I have the same issue. It seems the google server is shutting down after a certain number of requests within a time span - so the code is not wrong, but google prevents scripted high-volume use. Work-around could be a wait period between requests:

from time import sleep 

# Your loop of requests:
for url in urls:
    # ...
    sleep(2) # Wait for 2 seconds.

in your iteration @Clement-Lelievre . Don't know if 2 seconds is long enough though, you need to try it out

absolutely.

Clement-Lelievre commented 3 weeks ago

I have the same issue. It seems the google server is shutting down after a certain number of requests within a time span - so the code is not wrong, but google prevents scripted high-volume use. Work-around could be a wait period between requests:

from time import sleep 

# Your loop of requests:
for url in urls:
    # ...
    sleep(2) # Wait for 2 seconds.

in your iteration @Clement-Lelievre . Don't know if 2 seconds is long enough though, you need to try it out

absolutely.

In my use case, this part of my logic is the bottleneck and 2 seconds is not an option unfortunately

SSujitX commented 3 weeks ago

I have the same issue. It seems the google server is shutting down after a certain number of requests within a time span - so the code is not wrong, but google prevents scripted high-volume use. Work-around could be a wait period between requests:

from time import sleep 

# Your loop of requests:
for url in urls:
    # ...
    sleep(2) # Wait for 2 seconds.

in your iteration @Clement-Lelievre . Don't know if 2 seconds is long enough though, you need to try it out

absolutely.

In my use case, this part of my logic is the bottleneck and 2 seconds is not an option unfortunately

@Clement-Lelievre, can you share the list of news links?

SSujitX commented 3 weeks ago

Well, I gathered some. Let me try.

["https://news.google.com/read/CBMilgFBVV95cUxOM0JJaFRwV2dqRDk5dEFpWmF1cC1IVml5WmVtbHZBRXBjZHBfaUsyalRpa1I3a2lKM1ZnZUI4MHhPU2sydi1nX3JrYU0xWjhLaHNfU0N6cEhOYVE2TEptRnRoZGVTU3kzZGJNQzc2aDZqYjJOR0xleTdsemdRVnJGLTVYTEhzWGw4Z19lR3AwR0F1bXlyZ0HSAYwBQVVfeXFMTXlLRDRJUFN5WHg3ZTI0X1F4SjN6bmFIck1IaGxFVVZyOFQxdk1JT3JUbl91SEhsU0NpQzkzRFdHSEtjVGhJNzY4ZTl6eXhESUQ3XzdWVTBGOGgwSmlXaVRmU3BsQlhPVjV4VWxET3FQVzJNbm5CUDlUOHJUTExaME5YbjZCX1NqOU9Ta3U?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiiAFBVV95cUxQOXZLdC1hSzFqQVVLWGJVZzlPaDYyNjdWTURScV9BbVp0SWhFNzZpSWZxSzdhc0tKbVlHMU13NmZVOFdidFFkajZPTm9SRnlZMWFRZ01CVHh0dXU0TjNVMUxZNk9Ibk5DV3hrYlRiZ20zYkIzSFhMQVVpcTFPc00xQjhhcGV1aXM00gF_QVVfeXFMTmtFQXMwMlY1el9WY0VRWEh5YkxXbHF0SjFLQVByNk1xS3hpdnBuUDVxOGZCQXl1QVFXaUVpbk5lUGgwRVVVT25tZlVUVWZqQzc4cm5MSVlfYmVlclFTOUFmTHF4eTlfemhTa2JKeG14bmNabENkSmZaeHB4WnZ5dw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMimwFBVV95cUxQSEx1UHVQQ3ktb25xZjMzQ2NvY215NlBvemRWVG1VY2hIV2EyTWtwbE0zMTRYRm1rcWc3c0NLS1htRV85Ni1jM011a09EOUNRS3c2MU5UTXgwR1ozLVc5SGR4c2VISzNoSEpvdzB2SWF2WHdRY1lYckFreTB1QTA5MWJPVWtfWjRFVXV0cGRRLVBNQndGMWZaS1lQYw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMixgFBVV95cUxQNDE5VHlYeWl5QjhRM0RJZ0JoNUw5WXphdEhCLTZOSmtTWVlmQUViOTBEV3Z0R2R4TW1jRGc4QXFWd3gyeFBuT0p5aFREVTN0dUhnQndxLTRLOEJmQzN6MWFxdUFRdW16ZmthbTBRZTIzQnVQd2h2WnNrWnZmOGw3T2V2MzlMcXBWOVJvTE0zbjUxeGN6VzN1OVFPMHNKbjVBNEZyeEpsMXdyMjZySlNacWx2YWVNOTRIQ0MtTnJsb1BaN1NyUWfSAcsBQVVfeXFMUF9GTEstV2ROc21mSm9pX1VVNjhwdXhTQ0wzY3NLU0k0cVlMZFZDR19wZlE0andpcXNyZ2tidDFWSkowUW5rRTRXMlZIZHJoZlhWNHpfazQ3OF9QOG9YSy1SZ0dmMDlTMnlUSVRjVU1zS3FhMHczeEN3cFFJUlB2S2ktMEdheGUydm1saGROVDNLVzU5MHV1SmNOVEQyN3NGSXA0OEpNWUx5LTUwNzd4eUJRdjdPVUc0SnZrcy1VWUtLbzdpMUR4RWZuNE0?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMirgFBVV95cUxNWEZsLWJrbEZ0V08zR1E0RUxkWHdCMGo2ODFIYkpuWFFOMlVPQmNkeE1lVTV3eUdEWElGZFhhVjVEeDFUMEdhQ1IzZzVod2dFQkRsZnRrQ3pzNUxnT2tSMkpCRm9CLWZHaFljaElnMUwtZ05SVVpaYUZra3g5aFk4SDB3LVlXQnI1Tnp1dnZPLXl5azVadTV4aDJKeHVfMy1hTTI1UUl4T01GTU1uakE?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMid0FVX3lxTFA3MWFLVkszWEFnamw5UDZVTHVNMkNvM2FFaERmMjhKYS1OTGsyal9sbkhPdW9DbzgtOW9ncURUZ1FVZGE1dFNBdkxBNzhaUE85dEY3bDFYMXdveGNGdTJjeEZadGpUZ181UEZ2ZmNLcGVUR1pFZDNV?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMicEFVX3lxTE9RM3MzcXAxMmE5bHR0VGUtMjdCb0t0THpEZFNXbzl1OFFUMV9CdnMyUjU0T1NhMmR2UWtTZGViZFZiZi05akhrLWFFN05LeUVtdDFNcDVTTS16VTdob2FEeDdYZHJsaDdrYVNZeWxWZGs?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMi1gFBVV95cUxPR0tCV2Y5c3J0aDZNeUF1NVcyaDBWNVlsVDVXTnpUVUZvZURRYkVjb1BmVXBjR3VmRjEwTGJLVjZSRE9zV1NGU2ZJVENFOFN0VHlPVThxbzdIbE1lbUN1bWRDZGVmMzAybW5nMTYyMmxyb1JhWGRzTEJkSXc0SW1TRHUxbGdpU0V2NDQyQkgtSnQ0YXh0T29BMElJcndGQ0Vkankzb3lKQ0p3cm5DelBoQ01RU0YtQmRDVWFfdVlHa3NkVTBKMkdMdkM3RzF2XzlzMFdjQThn?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMihwFBVV95cUxPcVdNT25ua3ZHT045cjQ3QU5TQ29EaWdXRnBobTNaNERxYU12NlZKcXBpREtXYjhSZ0dWYkR5bGhwaWw1cGVCY0ZBNFhPY0c5MFIyZTJlelAwUndIc2lPUGRnX1ozUmRYSmdsRXZFcTlEZkxOVzVjeExYTWZlRkFXTFBiUHhPQms?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMisgFBVV95cUxPOGY5ZFNBWTk3cXVucDFaMHZUMGE5OXVFYnlLdnhoUXB6RWNneWJEaFlQXzN3QW5OaVFiUVZYcUt2YzZQQXNMcnZ0ZENDTExzZ25RR190ZlhjTFQ2MDVXMkllTDJ5UHFnMVdjVG5ZUnhMQ1lUZWtmeTZtYmJXTmY4dG9qQmVMclQxd3I5UFVsNV9jeTl3cDlQMTZ5VVAwWVhqR0NWb2RGTWJMQjhqVktRUTBR?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMigwFBVV95cUxNbVNkWU1CWnBUa0dxRlo2N19fVGdEVWNBMEd1QlVqOURST1ZnTjlSWTZxRHpvVllQZkliMkRLcnhySHVheDVPNFBrb0d4RGJHUUVYQ1ZCOU1fdnJrcXZWczh4SVlXZUc2cjB2YXBGLWdGYmY4V19sY0dYX3U2cW51bTZ5Zw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMipgFBVV95cUxPcXlYSE5HdVRhS09SY2NQZjY4RGtiRmRWdURWdjJOeDlFcmxVWFltd2hjalpHZmRIWnRVUnI1T3RTVHRnSlJkN3Zja0NybFMzZ2VhWWh2cXRKdHkzcTZmZUhVY3lpN1hGRFlJcURWbl9ESkdzMFRuYk5LMms3a19PXzJFU2UyOXhDRDNiTkVQUU5WVkRzR2pHaEh3THJoUmJ5VEdNQ09R?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMie0FVX3lxTFBtZTlGX2ZxdVVvNTBsWDBkZXhYN3d1dU9wVlhqOXREaEJOTFY5TUxnMnoxa3JnLTBnUWJQRXNmM1YyRUkzTTQ2aDdmLTNreDdueG9HUUFKNkJPM0w1TVp3UGpqNER2ckVQNHllZjJZMDFCVmc0SXFBaFd6NA?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiiwFBVV95cUxOM2ZvQmozcEdJUlRlc2ZZUEhTMzVBSThzVzZnVVByaTY5RlpxRFFkLXFiYjBtY09VT3RuYmhQUzNrUVJsRHRWUWRRb1RtcXV0b2RNakg1R2stamlfOGNkTXc0TmxiemFfZF93VjFrNmxXOXdWdmhfdHFjQ0ZqU19PUFFIUUw1MWFNeG1z?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiiwFBVV95cUxNUTRIam1EMzNhcVVmdVBFNC1HQ3M0eGNmMkN4WHdBaUxWdDFLdS12c1ZXTFdRZktHLTZ4WUhNOUFuUkdlemhrMHBManR2c2hfMnZMbFBhS0pxdHo4WEt2TEx2TjhNd1NyelRxaldkVVlyZ3kxdFVRNW1KQXJMWmhiYTZTaGFoUXNzVkw0?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiYEFVX3lxTE93NExPTnJZSU1BMTV3c0gtOFE3VU5pd2dCcjRBc3hIUjlGNWZ4b09XUnE3Q2pVODNjUkl2QWllb2x2VlBQNTFILVZaYVlGNXlwTGlkYzY3OHV1dG50R3hzMw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiZ0FVX3lxTFBzSGtFeThyY3RZOFBLMDU4NDQyMk5ySWtCTVB3NlI5WFBmNnpUUHoxYmZBMlFVbUQ4Z21RUTY4WnJKZnp5NlNmUExsU0tCWHF4cjMtdHlTeDJxY05ZSlBTNDZWaEdJaEE?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMisAFBVV95cUxOLVNKekdqYjY2RkhLNmxFM0pIZ3dkY0VWZE81MFNKdlVyRG1MUlBXQ2xCVUJJT3lxdTQ0Vi0xTnVVU2kzSjljUTRnSmNiWTdaMVRvWEpHd1RaNVV6dmFtVWVpc0ZYazNvWTF6ZklZMHhHN1pldjIyamE5LWdaNk42TEtDZTFMX25TYktuUHN2OHExSnFpQ0x2aE94RFZVdTBsdWtqOU9EazVid00yNFRhetIBtgFBVV95cUxOSEpfa2U2MFdfeGNWOHNDYnpKTE9zc3ZjWkR1MkItRkFlTjRxallLWnpPSzU0eWEwYTFhTl90STZBRTFDbldnTHV4VXVxSXRHMUJqdXFCREFIU0FBR3V2TmJoRzRGd1VnWTgwOGZRMWVfYUtYbGhINnpJeTNGOHM2M29JRzlfUHBQdmswVmZ4aVZDOVR5NGVub21NUjRHaDNsLV9JcEg4ZTFEemdqenhVLTBGRFFZQQ?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiqwFBVV95cUxNRFA4WUMxckI3NlZBUmhZaFBHV2o5RGowZnUzdTdOTVV5OTZYLXhnbXZkYWlxbWpJTzIxUzBZbVpoZVA2QVBaUHJSUzVfSXNLM2RNU2hiX1hGQjBSbW0tZktPejBLd3NMNklVcC11Mk5wemthUGZqbGYyaUNOSFpmRWkyVzQtZlBNSzRXLVlJd19TemlJTTBUc3Z0eHNWRGFMNVR6WU9zR2xURzA?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiiAJBVV95cUxQeFhvN1Y3eXNOZC0zS3JYM1BIQVVTQWdMQ3QwamZqUHNkNDF3a0xsU2VmOHZMM3pKODBoaG5oaGRCUkpTdTVVcmxNWmRkNFRSUmxqd1hfSElyREdOWE94cWxETGppZkhuNmhvZl9OSHVwSmU2b0FFbEtYYVpjQ0hPT0hVblZRakR1a19GelVzb3kzYTNKN3p6RUtHQzc0UmpINnhrZExDcmVfZkI4OW1OSFNHTTBjR1ZjY05RSWVuUnlEOFhuVkt5bTJxaTY1NXBPWDE4eDZhY0tMYnZZTXdkTlRvaHAxVTU3Z29yTEhndXZYRUdjd0ZQV0FWSjlZTmFBY3Z0N3hIWEk?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiugFBVV95cUxOeFRIWFFSOUFQeTRmMXdzSnBxNHZoZ3Z1SUpDNGRVWjFzeU4zalRjSG1mMWtjT3Njc1FhQk1xeTFwQ3VrdVZnTUd2NlllN3dBOXdtT1ZkVEtOUWJYOHJMV1RrVzFlb016czh3YkltSGxzMWVHck9ES0VLOVJYU3E4LU12M3JtOTZPMGNSNnRrSXh0Ni1PUERoSWlNR2tCNzFIZm1OOWRqTXJRQWdCZzRvcVRmTFAyMDJTVHc?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiWkFVX3lxTE5zdFUzTVNuWHh2MGpGaGRxSlVzcnhSM0FOTjRFd1hDa3Y2RWVwSzF0TnpQWFdoeXRhenZIeV9WSTVnZGgyLVhTMUtKN1duUmNReU5DMDc0NTJ3UdIBX0FVX3lxTFBQMjAybWpDNU5MN1BpRHEzcFFqeTVac3Zva1FlOUZ1RXVoUDU1S0R5b1NCVXJqQjZlQ1pIWFBSYWpUbEtuYktNdmFHTE1PejhMbFJWOExocTdkZDRrRnZN?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMihwFBVV95cUxPSUxubG95UWxXcjZvTUFVZG1iNjFib1Vvcm1ZdGE0ekFvVHVKZmRGaFphTjlHNGdHMXFfbUdHRTNIYW1YdjI4TExwNlc2M2ZybUVQSXRmeUYwdlpBTkFMcjM1OUlSZ1NiWVBHQWU1cFZuWmZrX05JbGIwY3phLTJaS2hPV0pqUDA?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMirwFBVV95cUxOS0hObjB2QWVodnRRQk12MWlUWENIcURSUUcyYzk5eS10dTlSTHkzc08ycHVxOHJDcmJBbWI5azdkVkFmLUJsMVJtOFJuNmxiZWtXQ0VXNExvUFE4N2JlQ0dpb0Z1dlRkTGtIeEJxMzd3Tjl3WS05aUlKWUE0YTNwVEJIZnZ4ME9NLUFMSl9hd0djd29LdGJkUVRrLUVtMTROUmNwVllrdWI3Nm8zeVh3?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiyAFBVV95cUxQa2pPR2k5VzdkWTRjTzEwaElDT3NkMk40OWpNM1NzNzVhYmJQWnJoTkVIV2dOM2laYmxqQ2p1bTZHb2NVdi1IcGVfekhWdXl0Y3B3UkQzRGh6VVV5RlFFSGFqNzJqQ2xQLUs2RUltWHVPOFY1UkZwZXpHUWppRllwVk9UUHBZNzhTNERObk50YXI4cUl1TkJpbjdtRVY1emNFRzBIY243YWNDWC0yZExfTlZ5NkVjc3QyLUNMZ2NxOEphNXFJejhfOQ?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMi1AFBVV95cUxQZXNhdnh3Zl9sZk00Q3RfbFB1bWdRVk5DMXBjc0NUMVh0U3JOeG5ZWmZNbHlTUFZRWDZCYjZPNTVQQzhJYXJqQS1FcDg3cUNqdHVOWmxXNkZDWjlwaGhId1pRV2tCSzJkSEVfd0FYSG5kVDEwbUNDR2ppQllZdGU5bk0yblJ6UE43UkxjR1RRbDM1MHJqek9kSEJhRm5rYUNMd0xaQTVXOVhlTl9JNHl6ZE1IQldBbTlRT0ZTeTZfa25UYmpoRzNLSE5mNjFGQzNtWTBCdg?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiY0FVX3lxTE5ZQTg0bnVuZmZUbDY5eTg0V3g3UmxOTEhZcm5TaDh2aGk2bzZIdVN3TV9ITmt2eHAxZk9JWkxnYlptQTFHUktJQlo5MC1VenhCYXA3UFRZQkhnUUdqbjlZalRBRQ?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMib0FVX3lxTFAwMVBtQVZvNm5oU09QOWx4ekhVR3VlQnYwV3duVnl6MXpUYUY3bjE2ZC1EUEpCVDBzTFJqYWtYZTlyemI3WVU4alBKdm9mc1MtdExweFpXSzZvaE1Ub1JuaE9DXzg0dTk5RVhNS3Bqcw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMifkFVX3lxTE5kWVNVRVFNR052dl9SQjR6WG1ZZElxT2RXUGVHNGgwVzhPT1dQd3ZScXotR0RmT3RpT3o2T21aZV96bjdaS3d5Ujd5YlJxZWVLNEo1aGU2cFQydU5QbDVHTnFMeHdzY2NBTnN3Ull1NW1EYnM0SnNTUlRaMTdGZw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMinAFBVV95cUxQYUp4RVhjbzBENTdxejNhbmV4enIxNmhYUHRtV3VFUlpxMmJWV1RwdkI1VGxyVjFVT0Q0LW5oNnZMVk50bHcxVWFGUm5NZGMybDJLXy1tQ3EzaURXdm5yTm84eW5SRUFaOHItbmppVmxqVHRIR0FLSFJhWEU5aUlmTEpoX3lzQzFwS1VCeGhqM0VpQkY0RkVOUUtsajE?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMigwFBVV95cUxNWDVrMlNHanZkSThJdDRQTFpHUURtb05DT3k4a0V6NVdETDlTZlE1dTFreVBldWE0ZTcwSmVabTJnRFQ2YmUtOE00Y1EzSDJwaGlkWVZSRkRpbk8tbjJhRWFOYlhCZ1hfaDVPdm1YMzlGTlZlZHVIVzlrUlYwenl2MGJQcw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMigwFBVV95cUxPY1c5LTZkeGJGb0JxdlZHRmZEbGNoQTJFdTJIV3ByQm9qUHZKT0FYVXN0VzJTdU54OGo3QWlvbmFoeXBKV2tyWV9PTXlINl8xRVFlTzZWWmM0RTZXci1qMDh1UkF3ZVE1SnhKdm5XSkpOME03RkNXdkNrSV9DOTVmU3JnMA?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMitAFBVV95cUxNZE5kbndnOWtwSGxLZjB1X3pWRHR4LUpXdV9KLVZPNEVLYmtybERQbDVQdGNrd3FtNVg2YVBGaHFOZXgyU0xIY0JaSUZZVjl4OW1yWmZfRHM1SkhSM1hmTTduNU1WYS1neWZNV2VFaGFTaWx1Qk45Zl81c1FaRDd1Q0xTUVc2ZlNOaW92b2RmRXUzNXlCUmpKZ2FOcERCdUZILTE3aEQxTGpuWV95clhZUHBhTDXSAboBQVVfeXFMTzRrMGFNYXdpVjBqZWhRQWFRZnZuYWFwdndDdzV6bklkNGpoTTlBb0E0SmZ0bGZZUUVWWUtpay1FMG5RM2JPZmhfMi1mOGtJUHRSbkVoZkE0WGxjVUdleGVmYVlDNGJmSTA2RmY4MExkdUlqNXlGRjQyX1Z5YzBRaXBmSFRMNVNES1d1d2g2OC12NlZ2NF8zQThtamVDeUlBVEtucTBac2FYTW1WSl8yT05mck9uZkhVUjdn?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiYEFVX3lxTFBjWlVEWGIxZjZhelY3Zl9lQVBsdTVOdFl4MXo2MTBzVjNTaDdqVmh5MGMzVEF6WnR1UDREZzlKM2JjUFBoR1dhZDd4TTNiZUpvYUdkLWdCdUUyWHBtRTRZdNIBaEFVX3lxTE5aSGNSOURIZlB3V3htUXQxQUxQZGVsRk90Qnh0SXBwb0lHMDRwVUFzZXI5VURGODlQUG9lc1NHLUlISmRVX0E4SUFtQzRGWVM3QmYzQ0RWYTlyNEJDcWlZYVZhUnFjVkdl?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMi7AFBVV95cUxNVFFIcHZfV1FQdDQtQ1Jibmhkd0J3cEk1NzFMQWVuTVNSSVhmV1BBZ1dLMWFVdHJqUm1lc3pXMS1ib1lLLTFudlkwcUZGX3JBNFBCVFZ3Z2Nhb2J1MEpuTjNKWjE2V0VLVW9LVTdZM0VIaEN1dTloVm91b1dfWG9udUhkb2p2Y2tJeVdUNlZGR3JjS1pyRjkwUlBCM3FnMWY0eVl1NDJZdnFnZVdTN1oyS01fTENDLVdhWHlYTjY4b0tHVTFTdk10Z1BiZW8wMXlrX2lxRUREN3pzM1JJUUdndVBrbDI4eURWWGVRZQ?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMikwFBVV95cUxQemRSNjJoeXYxTTM1MWc0S3NjczVYNkY3MzFvTVloTmd5dFRkR2hSNnhnMFlzNEtEem9uV1QzS1F5elJqUmdDRUlZYml2LUJHM2duNDhuUVNVWUg0UldtVkxSaXZHbVJDeG82VXFpdzBFNVV5T280ZEsydFBUbnFSTmJsMjJUdWNzYmpwa2NGZ00waTDSAZgBQVVfeXFMTlo3SVlfQ0RIb1A2YWhXOS0tT1JRbGRfdXZZWHNuWDVqQ0dQZm04ZnVSa2NBSTdtdEk5cWNSNHQyYnBHYjJfMHVfM2JCQTQyRzRpYVNhSFFuT1JwOGZTWDVTOEJDSXpaY0ZjWGFETVM3VV8zaGsyOXVqVmEyUnd1dWRVYTlHQ3pFY3ZTZmJLY05jN29feFBGZDY?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMixAFBVV95cUxNYjZLV0lSRXJrRlJtYXp3R0pldXoxR0RiZmo5a0UtRkRYdVZyWUxfX3pQMzZmbFJZcWQ0TGhsWnJrR2plek4wMktzVVBUQ19qb0RhNmFEaUNmMzhienBrOTJCY2Frc045SGhmSlEzMWZ3d0x6VVVWMW1KUHJHT3pVc0ZaNnZHV2lyV1ZYQjFmVTdlOElFaFBwYUsyZlN1WEYyd2Q0OTNROTlOeXROOGZmdVp5a3NnOGFjRzU3Z0UzeW1UR1Vv?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMijwFBVV95cUxOSXg5MUdEbFR3ODhienl3ck93WEEwc25iWXdQS2hHM2dZd2tLZ0M0Y1BhbjdDTUNGVEtTXzRxaEcwYXpYUE1UMXVvMVlSdEo3NHZBNHVpYXVnMlN6VGFPUlNtOUhTLWdhU2l4LVFfV3BlWGRJSXVpbGkwNV9RNHN5SUtLTU5FMl9keFNLRThOcw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMimwFBVV95cUxQandXbEVwWldRSy1ZZmRjbFl6QVplU05CRFlna3MyX3RHTnQ2VjhXUkZyM3M1Z2ZOQTVIVkQzLXVTS2NGUGt3bmozczBKbjFibHB0Z2ZSUlloa3VTNTNBUVM1d0g3ODFaVlV2dldVbTdwc2c0aWFXMm9JUFVBVXVLQ21ScmlhTlozNU1DV2lPNWxCRUxCVThDSHlDbw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMimwFBVV95cUxOc0Q4LWdtNXdHZXEtOTdheDBaZ2FxNzRpdHNwQldfOVIwcGFlREtFWjRBSmZVZEJNeG5VSU5LRTZKeWRQQmU3MnVHc0FWWDNVZWE2YnBLc2ZfNzZTMHhjZ2l4QXFIWjllTVBjUmhrSmpCX2dZekZnYlpVdkFYVUpBc3NHM0dDelVhVlRyeEVRaUxzaWNNWVRZaDVDYw?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMic0FVX3lxTFBQWmEzSkxTcHpLUkJvNEJWY3BuLXFmeDE3b1RiVk9yRmQyRXIySzNrZzFQNGVFR01taUlraVBiUlU0by1rVGZqNWgtMFBMWFd2OWw1NGdjelJfeEhvYVo3a1h2dGNVMFY0b1ZKTHdQbXZwa1E?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMivwFBVV95cUxQbTdRMjFBSXdJazNRRzZwZ1dUTl9LNlkxVVVZWmtHNTNYbjFLcjZpd2FOU1F2ZTd4WDVYdjdhOFhiallLQmJ5dlpHYW5rMlBLdXlCdTV4UXlITnlUazVGWElrOFNxYzRoU2pPWUlaVVF2QlplVWZFS2RiM3RGZUVPYXdFZFFXZkl1VTMyOTJRTmNOcEZucDZNemFmU0xHZWpzQnJtdlQySHhtNjlBMUFLdVpSb3ZnbldtOVl0WmM4OA?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMilwFBVV95cUxNMVEyY0dvYXViaHVib3gxRUZURXZENWFpcTBpUnZhMTNmUmwtVzRKOElvQnE3RDVROEpFYnZ5V2NFUmhvdDVmalJsRGp3Z1A5TGhlaGVBaWtBcFI4VzhTMHk2QkppOEpaazhTSVZyaFdsYUtBU1FKYk45UVBsQmtsTHJ1NElYRzdLYXNUNGtGbkt1TzhkWnRz?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMiogFBVV95cUxPOG13dGlNYk9rbjJGZUcySEVtSU9wZUx4R2E2ODRYTHhxTGJ1SDJYcE9yTl9fVHhMTEUyN25uTWpiRUJTOEotTl9oXzFxUlBEX0NROVJDRDBCUTN2OXBmeGJlVVVFWE80WjEyMU1lUDU1di1yN29VZ1hNNS1pOXpkVTlYck1SQnlQclFaYlp1VmpDNjR6VmtBaU8yS2xtdDRWSkE?hl=en-US&gl=US&ceid=US%3Aen","https://news.google.com/read/CBMimwFBVV95cUxQaC1jVzQ1TXhyS2F4bTV5NUNCSUFFaW5TMTdzS0wxQnI4cFlhUTRISkNXYUpMWW4yWVc2OG1XTHBucnpGWmpiOTg3eEU1X2ttdlo4MmtYWEpPTVpQZDlrbmFHb2FmSXNrU2t4MWUzUllyQ2FZWG1MUjNUUHJxZkxiSV9KbFF1alBRQmRGcy1pdXNwNWw4a1VoWmtWRQ?hl=en-US&gl=US&ceid=US%3Aen"]

SSujitX commented 3 weeks ago

newss

felipehertzer commented 3 weeks ago

It is a batch execution process, and I believe it currently accepts up to 100 URLs per request. There may be potential for an increase in the number of URLs processed simultaneously with that change.

Info: https://kovatch.medium.com/deciphering-google-batchexecute-74991e4e446c

jlchen14 commented 1 week ago

I had an application which involves the use of GoogleNews, 5 months before, I used the base64 method to decode the url, and that method worked no more after Google changed the structure and the rules behind its urls. I am very glad to find this decoder, and it worked! I met the same problem after a certain number of requests, I got the "HTTP 429 too many requests " error, I planed on trying to set the interval time to 5s, because I run the application during the night, I hope this will work.

bertelschmitt commented 1 week ago

The new version of Google News Decoder works, but Goggle doesn’t seem to like us using it.

After a few calls, Google will come back with a return code 429 Client Error: Too Many Requests. Google News Decoder tries to circumvent this with a delay between calls, but Google seems to have wisened up to that trick: Even when using the delay, the 429 Client error will come back after 15 iterations or so. The 15 iterations are not cast in stone, could be more, could be less.

I started a testbed where Google News Decoder is called with https://news.google.com/rss/articles/xxx urls, pulled at random from a list of > 200 different urls. Between calls, a delay was introduced. I tried various delays, up to one minute. Each time, Google complained after around 15 calls. When I increased the delay to 70 seconds, Google became a bit more cooperative. So far, the test has run more than 1,000 iterations with the 70 second delay in-between, and no 429 error.

Careful: Once the 429 error is triggered, it won’t be seconds, but many minutes until Google grants new access. During my testing, I occasionally had to wait up to one hour until Google stopped giving the 429 error. Re-running the decoder within the no access period seems to restart the 429 time-out.

Bottom line: Unless you plan on using Google News Decoder for only a few accesses, or if you can’t wait more than a minute until the next access, Google News Decoder will not work for you. Also, Google can tighten the parameters at any time, or come up with yet another algorithm.

If you process thousands of daily headlines, you will have to consider looking for another way to decode the elusive Google url.

SSujitX commented 1 week ago

Yes, Google News is being strict now. I will implement proxy.

piyushgarg commented 6 days ago

How will a proxy solve this problem?

jlchen14 commented 6 days ago

actually using a proxy did solve the problem, i bought a tunnel and change the ip proxy every time i try decode the url by accessing the google re-directing link. only problem is that the cost of my project is 3 times higher than before

bertelschmitt commented 6 days ago

"actually using a proxy did solve the problem, i bought a tunnel and change the ip proxy every time i try decode the url by accessing the google re-directing"

Are you sure? I've run the code from multiple sites and multiple IPs, and each time Google crapped out after 15 or so accesses. Going that route, one would think that the originating IP would have to be changed after 13 or so iterations, and that the IP could not be used until after a to-be-determined waiting period. That would be lots of IPs. Not very practical.

piyushgarg commented 6 days ago

Yes, this doesn't seem to be feasible. My only purpose to decode is to open the external link directly. There is a lag when the Google news link has been opened which gets redirected to original website.

jlchen14 commented 6 days ago

"actually using a proxy did solve the problem, i bought a tunnel and change the ip proxy every time i try decode the url by accessing the google re-directing"

Are you sure? I've run the code from multiple sites and multiple IPs, and each time Google crapped out after 15 or so accesses. Going that route, one would think that the originating IP would have to be changed after 13 or so iterations, and that the IP could not be used until after a to-be-determined waiting period. That would be lots of IPs. Not very practical.

you are right, i change the ip everytime and it worked, and it is expensive

hotszhin223 commented 15 hours ago

So the only way to solve the 429 client error is to using proxy?