A very simple news crawler in Python. Developed at Humboldt University of Berlin.
[Quick Start](#quick-start) | [Tutorials](#tutorials) | [News Sources](/docs/supported_publishers.md) | [Paper](https://arxiv.org/abs/2403.15279)
## Quick Start To install from pip, simply do: ``` pip install fundus ``` Fundus requires Python 3.8+. ## Example 1: Crawl a bunch of English-language news articles Let's use Fundus to crawl 2 articles from publishers based in the US. ```python from fundus import PublisherCollection, Crawler # initialize the crawler for news publishers based in the US crawler = Crawler(PublisherCollection.us) # crawl 2 articles and print for article in crawler.crawl(max_articles=2): print(article) ``` That's already it! If you run this code, it should print out something like this: ```console Fundus-Article: - Title: "Feinstein's Return Not Enough for Confirmation of Controversial New [...]" - Text: "Democrats jammed three of President Joe Biden's controversial court nominees through committee votes on Thursday thanks to a last-minute [...]" - URL: https://freebeacon.com/politics/feinsteins-return-not-enough-for-confirmation-of-controversial-new-hampshire-judicial-nominee/ - From: FreeBeacon (2023-05-11 18:41) Fundus-Article: - Title: "Northwestern student government freezes College Republicans funding over [...]" - Text: "Student government at Northwestern University in Illinois "indefinitely" froze the funds of the university's chapter of College Republicans [...]" - URL: https://www.foxnews.com/us/northwestern-student-government-freezes-college-republicans-funding-poster-critical-lgbtq-community - From: FoxNews (2023-05-09 14:37) ``` This printout tells you that you successfully crawled two articles! For each article, the printout details: - the "Title" of the article, i.e. its headline - the "Text", i.e. the main article body text - the "URL" from which it was crawled - the news source it is "From" ## Example 2: Crawl a specific news source Maybe you want to crawl a specific news source instead. Let's crawl news articles from Washington Times only: ```python from fundus import PublisherCollection, Crawler # initialize the crawler for The New Yorker crawler = Crawler(PublisherCollection.us.TheNewYorker) # crawl 2 articles and print for article in crawler.crawl(max_articles=2): print(article) ``` ## Example 3: Crawl articles from CC-NEWS If you're not familiar with CC-NEWS, check out their [paper](https://paperswithcode.com/dataset/cc-news). ````python from fundus import PublisherCollection, CCNewsCrawler # initialize the crawler for news publishers based in the US crawler = CCNewsCrawler(*PublisherCollection.us) # crawl 2 articles and print for article in crawler.crawl(max_articles=2): print(article) ```` ## Tutorials We provide **quick tutorials** to get you started with the library: 1. [**Tutorial 1: How to crawl news with Fundus**](docs/1_getting_started.md) 2. [**Tutorial 2: How to crawl articles from CC-NEWS**](docs/2_crawl_from_cc_news.md) 3. [**Tutorial 3: The Article Class**](docs/3_the_article_class.md) 4. [**Tutorial 4: How to filter articles**](docs/4_how_to_filter_articles.md) 5. [**Tutorial 5: How to search for publishers**](docs/5_how_to_search_for_publishers.md) If you wish to contribute check out these tutorials: 1. [**How to contribute**](docs/how_to_contribute.md) 2. [**How to add a publisher**](docs/how_to_add_a_publisher.md) ## Currently Supported News Sources You can find the publishers currently supported [**here**](/docs/supported_publishers.md). Also: **Adding a new publisher is easy - consider contributing to the project!** ## Evaluation benchmark Check out our evaluation [benchmark](https://github.com/dobbersc/fundus-evaluation). | **Scraper** | **Precision** | **Recall** | **F1-Score** | |-------------|---------------------------|---------------------------|---------------------------| | [Fundus](https://github.com/flairNLP/fundus) | **99.89**±0.57 | 96.75±12.75 | **97.69**±9.75 | | [Trafilatura](https://github.com/adbar/trafilatura) | 90.54±18.86 | 93.23±23.81 | 89.81±23.69 | | [BTE](https://github.com/dobbersc/fundus-evaluation/blob/master/src/fundus_evaluation/scrapers/bte.py) | 81.09±19.41 | **98.23**±8.61 | 87.14±15.48 | | [jusText](https://github.com/miso-belica/jusText) | 86.51±18.92 | 90.23±20.61 | 86.96±19.76 | | [news-please](https://github.com/fhamborg/news-please) | 92.26±12.40 | 86.38±27.59 | 85.81±23.29 | | [BoilerNet](https://github.com/dobbersc/fundus-evaluation/tree/master/src/fundus_evaluation/scrapers/boilernet) | 84.73±20.82 | 90.66±21.05 | 85.77±20.28 | | [Boilerpipe](https://github.com/kohlschutter/boilerpipe) | 82.89±20.65 | 82.11±29.99 | 79.90±25.86 | ## Cite Please cite the following [paper](https://arxiv.org/abs/2403.15279) when using Fundus or building upon our work: ```bibtex @misc{dallabetta2024fundus, title={Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions}, author={Max Dallabetta and Conrad Dobberstein and Adrian Breiding and Alan Akbik}, year={2024}, eprint={2403.15279}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` ## Contact Please email your questions or comments to [**Max Dallabetta**](mailto:max.dallabetta@googlemail.com?subject=[GitHub]%20Fundus) ## Contributing Thanks for your interest in contributing! There are many ways to get involved; start with our [contributor guidelines](docs/how_to_contribute.md) and then check these [open issues](https://github.com/flairNLP/fundus/issues) for specific tasks. ## License [MIT](LICENSE)