search

ddsky / world-news-api-clients

MIT License
7 stars 2 forks source link

Extracting news fails if there's a redirect involved #8

Closed Boscop closed 9 months ago

Boscop commented 10 months ago

Extracting news fails if there's a redirect involved. E.g. https://news.google.com/rss/articles/CBMiZGh0dHBzOi8vYmxpdHouYmcvc3Z5YXQvb3JiYW4tc2tsb25pLXZkaWdhLXZldG90by16YS1maW5hbnNvdmF0YS1wb21vc2hjaC16YS1raWV2LWFrb19uZXdzOTk1OTc2Lmh0bWzSAQA?oc=5 redirects to: https://blitz.bg/svyat/orban-skloni-vdiga-vetoto-za-finansovata-pomoshch-za-kiev-ako_news995976.html If I pass the original google news URL to the extract-news endpoint: https://api.worldnewsapi.com/extract-news?api-key=<api-key>&url=https%3A%2F%2Fnews.google.com%2Frss%2Farticles%2FCBMiZGh0dHBzOi8vYmxpdHouYmcvc3Z5YXQvb3JiYW4tc2tsb25pLXZkaWdhLXZldG90by16YS1maW5hbnNvdmF0YS1wb21vc2hjaC16YS1raWV2LWFrb19uZXdzOTk1OTc2Lmh0bWzSAQA%3Foc%3D5 it fails to extract the article:

{
  "title": "Google News",
  "text": "Opening https://blitz.bg/svyat/orban-skloni-vdiga-vetoto-za-finansovata-pomoshch-za-kiev-ako_news995976.html",
  "url": "https://news.google.com/rss/articles/CBMiZGh0dHBzOi8vYmxpdHouYmcvc3Z5YXQvb3JiYW4tc2tsb25pLXZkaWdhLXZldG90by16YS1maW5hbnNvdmF0YS1wb21vc2hjaC16YS1raWV2LWFrb19uZXdzOTk1OTc2Lmh0bWzSAQA?oc=5",
  "image": "https://lh3.googleusercontent.com/J6_coFbogxhRI9iM864NL_liGXvsQp2AupsKei7z0cNNfDvGUmWUy20nuUhkREQyrpY4bEeIBuc=s0-w300-rw",
  "authors": [],
  "language": "en"
}

If I try a different news extractor API, that one doesn't have a problem following the redirect: https://api.articlextractor.com/v1/extract?language=en&api_token=<api-key>&url=https%3A%2F%2Fnews.google.com%2Frss%2Farticles%2FCBMiZGh0dHBzOi8vYmxpdHouYmcvc3Z5YXQvb3JiYW4tc2tsb25pLXZkaWdhLXZldG90by16YS1maW5hbnNvdmF0YS1wb21vc2hjaC16YS1raWV2LWFrb19uZXdzOTk1OTc2Lmh0bWzSAQA%3Foc%3D5 and extracts it successfully:

{
  "data": {
    "publish_date": "2024-01-09 17:09:18+02:00",
    "source_url": "https://news.google.com",
    "url": "https://news.google.com/rss/articles/CBMiZGh0dHBzOi8vYmxpdHouYmcvc3Z5YXQvb3JiYW4tc2tsb25pLXZkaWdhLXZldG90by16YS1maW5hbnNvdmF0YS1wb21vc2hjaC16YS1raWV2LWFrb19uZXdzOTk1OTc2Lmh0bWzSAQA?oc=5",
    "canonical_link": "https://blitz.bg/svyat/orban-skloni-vdiga-vetoto-za-finansovata-pomoshch-za-kiev-ako_news995976.html",
    "title": "Орбан склони! Вдига ветото за финансовата помощ за Киев, ако...",
    "top_image": "https://static.blitz.bg/documents/thumbnails/730/202311/1700668940-shutterstock-1046465572.jpg",
    "images": [
      "https://static.blitz.bg/documents/thumbnails/255/201707/1499086981-pretty-woman-1509956-960-720.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704451925-sistemen-bufer-16.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202301/1672903705-1578376561-66176-1.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202312/1703949621-byashchla-roza.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704345983-1feb0b41-b251-453e-be00-6a361eb35254.jpg",
      "https://static.blitz.bg/documents/thumbnails/sport/95/202401/1704997211417728727-677595191208821-8340412429521846224-n-717x543.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704479722-verrr.jpg",
      "https://static.blitz.bg/documents/thumbnails/sport/95/202401/1704971860t.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704317296-an1.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/201908/1566212710-06d75f6c6270b52a9e7e27ce06168cd6.jpg",
      "https://news.google.com/assets/img/google-news.png",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704310796-pl6.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202312/1703913351-22.jpg",
      "https://static.blitz.bg/documents/thumbnails/730/202311/1700668940-shutterstock-1046465572.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704338397-35.jpg",
      "https://static.blitz.bg/documents/thumbnails/95/202401/1704984734-screenshot-2024-01-11-at-16-51-59-zbite-na-drakona-sa-se-poyavili-blizo-do-ukrainskata-granitsa-v-belarus.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202312/1702223177-sistemen-bufer-8.jpg",
      "https://news.google.com/images/fire.png",
      "https://static.blitz.bg/documents/thumbnails/255/202306/1686117817-shutterstock-1935376036.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202303/1678640820-sa1.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704399567-745-425-profesiia.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202301/1672913135-ant.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704180207-1-show04-krasi-2.jpg",
      "https://static.blitz.bg/documents/thumbnails/800/202311/1700668940-shutterstock-1046465572.jpg",
      "https://blitz.bg/assets/img/logo-xmas.png",
      "https://static.blitz.bg/documents/thumbnails/95/202105/1620976966-clipboard13.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202401/1704479522-frtt.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/202208/1660320212-mon2.jpg",
      "https://static.blitz.bg/documents/thumbnails/255/201911/1574603612-bank1.jpg"
    ],
    "videos": [],
    "text": "https://blitz.bg/svyat/orban-skloni-vdiga-vetoto-za-finansovata-pomoshch-za-kiev-ako_news995976.html Blitz.bg\n\nУнгария даде знак, че може да вдигне ветото си върху пакета от финансова помощ на Европейския съюз за Украйна на стойност 50 милиарда евро, ако финансирането се преценява и гласува на годишна база, съобщи изданието \"Политико\".\n\nБудапеща даде да се разбере, че може да вдигне ветото, ако Европейския съвет единодушно одобрява финансирането на годишна база, съобщиха трима европейски дипломати.\n\nТова би дало на унгарския премиер Виктор Орбан възможността да блокира европейското финансиране за Украйна всяка конкретна година, както и да иска отстъпки от Брюксел в замяна на вдигане на ветото, се посочва в публикацията на \"Политико\".\n\nВ нея се отбелязва, че няколко европейски дипломати остават скептични, като казват, че решението за предоставяне на помощ на годишна база няма да осигури предсказуемост на Украйна.\n\n\"Седемгодишната бюджетна рамка на ЕС е многогодишна рамка и не може да бъде променяна всяка година\", каза един от европейските дипломати.\n\nФинансовата помощ на ЕС за Украйна ще бъде обсъдена на насрочена за утре среща на постоянните представители в Съвета на ЕС.\n\nОЧАКВАЙТЕ ПОДРОБНОСТИ В БЛИЦ",
    "tags": [
      "украйна",
      "помощ",
      "виктор орбан",
      "европейски съюз",
      "унгария"
    ],
    "authors": [
      "Име"
    ],
    "meta_image": "https://static.blitz.bg/documents/thumbnails/730/202311/1700668940-shutterstock-1046465572.jpg",
    "meta_description": "Финансовата помощ на ЕС за Украйна ще бъде обсъдена на насрочена за утре среща на постоянните представители в Съвета на ЕС",
    "meta_keywords": [
      "унгария",
      "европейски съюз",
      "помощ",
      "украйна",
      "виктор орбан"
    ],
    "meta_lang": "bg",
    "meta_favicon": "/icons/apple-icon-57x57.png",
    "meta_site_name": "Blitz.bg",
    "meta_data": [
      "language",
      "viewport",
      "robots",
      "revisit-after",
      "title",
      "description",
      "keywords",
      "news_keywords",
      "author",
      "rating",
      "distribution",
      "og",
      "fb",
      "article",
      "thumbnail",
      "google-site-verification",
      "msvalidate.01",
      "copyright",
      "twitter",
      "contact",
      "msapplication-TileColor",
      "msapplication-TileImage",
      "theme-color"
    ],
    "html": null
  }
}

It would be great if your API would also be able to deal with such links, by following redirects :)

ddsky commented 9 months ago

@Boscop thank you for noticing, this is now resolved along with the author extraction