Closed fhamborg closed 7 years ago
Hi Felix,
du musst nur den Download-Crawler verwenden und die Heuristiken ausschalten. Im Anhang findest du ein entsprechendes JSON.
Beste Grüße Sören
# Furthermore this is first of all the actual config file, but as default just filled with examples. { # Every URL has to be in an array-object in "base_urls". # The same URL in combination with the same crawler may only appear once in this array. "base_urls" : [ { "crawler": "Download", "url": [ # Cubs win Championship ~03.11.2016 "http://www.dailymail.co.uk/news/article-3899956/Chicago-Cubs-win-World-Series-epic-Game-7-showdown-Cleveland.html", "http://www.mirror.co.uk/sport/other-sports/american-sports/chicago-cubs-win-world-series-9185077", "https://www.theguardian.com/sport/2016/nov/03/world-series-game-7-chicago-cubs-cleveland-indians-mlb", "http://www.telegraph.co.uk/baseball/2016/11/03/chicago-cubs-break-108-year-curse-of-the-billy-goat-winning-worl/", "https://www.thesun.co.uk/sport/othersports/2106710/chicago-cubs-win-world-series-hillary-clinton-bill-murray-and-barack-obama-lead-celebrations-as-cubs-end-108-year-curse/", "http://www.bbc.com/sport/baseball/37857919", "http://www.thetimes.co.uk/article/chicago-cubs-end-108-year-wait-for-world-series-win-g09t0kgfm", "http://www.independent.co.uk/sport/us-sport/major-league-baseball/world-series-chicago-cubs-cleveland-indians-108-year-title-drought-a7394706.html", "http://www.independent.co.uk/sport/us-sport/major-league-baseball/chicago-cubs-fans-celebrate-world-series-title-a7394736.html", "http://www.standard.co.uk/sport/other-sports/chicago-cubs-win-world-series-to-end-108year-curse-and-earn-invite-from-barack-obama-a3386411.html", "http://www.nytimes.com/2016/11/03/sports/baseball/chicago-cubs-beat-cleveland-indians-world-series-game-7.html?_r=0", "https://www.washingtonpost.com/sports/believe-it-chicago-cubs-win-classic-game-7-to-win-first-world-series-since-1908/2016/11/03/99cfc9c2-a0b3-11e6-a44d-cc2898cfab06_story.html", "https://www.washingtonpost.com/sports/nationals/you-knew-it-couldnt-come-easy-but-the-cubs-are-world-series-champions/2016/11/03/a4487ade-a0b3-11e6-a44d-cc2898cfab06_story.html", "http://www.usatoday.com/story/sports/ftw/2016/11/03/sports-world-reacts-to-the-chicago-cubs-winning-their-first-world-series-since-1908/93225730/", "http://www.wsj.com/articles/chicago-cubs-win-the-world-series-ending-108-year-drought-1478149589", "http://nypost.com/2016/11/03/cubs-end-drought-in-chaotic-epic-world-series-finale/", # FBI clears Clinton ~06.11-2016 "http://www.dailymail.co.uk/wires/reuters/article-3910804/Trump-Clinton-focus-crucial-states-campaigns-final-days.html", "http://www.mirror.co.uk/news/world-news/hillary-clinton-cleared-fbi-over-9210739", "https://www.theguardian.com/us-news/2016/nov/06/fbi-director-hillary-clinton-email-investigation-criminal-james-comey", "http://www.bbc.com/news/election-us-2016-37892348", "https://www.thesun.co.uk/news/2130219/trumps-fury-after-fbi-says-hillary-clinton-has-committed-no-crime-in-email-scandal/", "http://www.thetimes.co.uk/article/clinton-off-the-hook-as-fbi-drops-investigation-into-emails-m9t6pnr0s", "http://www.express.co.uk/news/world/729545/hillary-clinton-chelsea-clinton-wedding-funds-wikileaks-rudy-giuliani", "http://www.telegraph.co.uk/news/2016/11/06/us-election-hillary-clinton-up-in-polls-as-hispanic-surge-threat/", "http://nypost.com/2016/11/06/fbi-stands-by-decision-not-to-charge-clinton-after-review-of-additional-emails/", "http://www.wsj.com/articles/fbis-comey-says-new-emails-dont-change-conclusions-about-hillary-clinton-1478464650", "http://www.usatoday.com/story/news/politics/elections/2016/2016/11/06/fbi-not-recommending-charges-over-new-clinton-emails/93395808/", "https://www.washingtonpost.com/blogs/post-partisan/wp/2016/11/06/comey-to-country-the-jury-will-disregard/?utm_term=.914ce12b2617", "http://www.nytimes.com/2016/11/06/us/politics/presidential-election.html", # China: rescue boy from well ~08.11.2016 "http://www.bbc.com/news/world-asia-china-37906226", "http://www.bbc.com/news/world-asia-china-37946716", "http://www.dailymail.co.uk/news/article-3916560/Dramatic-footage-shows-rescuers-using-eighty-diggers-save-boy-fell-130ft-deep-picking-cabbages-Chinese-farm.html", "http://www.dailymail.co.uk/news/article-3923808/Mystery-Chinese-boy-fell-deep-massive-rescue-operation-involving-80-diggers-chute-empty.html", "http://www.thetimes.co.uk/article/millions-watch-as-rescue-effort-fails-to-save-boy-52c6dlpml", # Toblerone redesign outrage ~08.11.2016 "http://www.standard.co.uk/news/uk/toblerone-bar-shape-change-sparks-anger-among-fans-a3389711.html", "http://www.independent.co.uk/news/uk/home-news/toblerone-new-shape-outrage-chocolate-scandal-a7404011.html", "https://www.thesun.co.uk/news/2138318/toblerone-fans-outraged-after-gap-between-triangles-is-increased-to-reduce-the-amount-of-chocolate-in-bars/", "http://www.thetimes.co.uk/article/toblerone-redesign-runs-into-a-mountain-of-trouble-82h785rfr", "http://www.telegraph.co.uk/news/2016/11/08/toblerone-faces-mountain-of-criticism-over-stupid-change-to-its/", "https://www.theguardian.com/business/2016/nov/08/toblerone-gets-more-gappy-but-its-fans-are-not-happy", "http://www.mirror.co.uk/news/uk-news/is-diet-version-outrage-toblerone-9217538", "http://www.dailymail.co.uk/news/article-3915960/Toblerone-increase-gaps-bar-s-iconic-peaks-make-lighter.html", "http://www.bbc.com/news/uk-37904703", "http://www.mirror.co.uk/news/weird-news/best-way-use-controversial-new-9233101", "http://www.express.co.uk/life-style/food/730108/Toblerone-shrinks-chocolate-bar-denies-Brexit-link", "http://nypost.com/2016/11/08/people-are-pissed-over-toblerones-new-candy-size/", "http://blogs.wsj.com/moneybeat/2016/11/08/mondelezs-toblerone-moves-mountain-to-hide-price-increase/", "http://www.usatoday.com/story/money/nation-now/2016/11/08/while-us-talks-election-uk-outraged-over-toblerone-chocolate/93465240/", "https://www.washingtonpost.com/news/worldviews/wp/2016/11/08/brits-blame-strange-new-toblerone-shape-on-brexit/", "http://www.nytimes.com/2016/11/09/world/europe/toblerone-triangle-change-uk.html", # Trump wins election # Anti-Trump protests ~09.11.2016 "http://www.independent.co.uk/news/world/americas/us-elections/us-election-donald-trump-wins-protests-los-angeles-california-oregon-a7407521.html", "http://www.thetimes.co.uk/article/thousands-protest-against-election-result-in-us-cities-5v6ncl6pg", "http://www.telegraph.co.uk/news/2016/11/10/demonstrations-erupt-across-the-us-as-country-begins-to-imagine/", "http://www.express.co.uk/news/world/730363/protests-Donald-Trump-violence-US-election-Hillary-Clinton", "http://www.mirror.co.uk/news/world-news/trump-win-sparks-riots-across-9225317", "http://www.bbc.com/news/election-us-2016-37946231", "http://www.dailymail.co.uk/wires/ap/article-3920168/Trump-victory-sets-protests-California-Oregon.html", "https://www.theguardian.com/us-news/2016/nov/09/anti-donald-trump-protests-new-york-chicago-san-francisco", "http://www.nytimes.com/2016/11/10/us/trump-election-protest-berkeley-oakland.html", "https://www.washingtonpost.com/politics/trumps-white-house-win-promises-to-reshape-us-political-landscape/2016/11/09/62baa5e4-a66a-11e6-ba59-a7d93165c6d4_story.html" "http://www.usatoday.com/story/news/politics/2016/11/10/hundreds-protest-trump-downtown-milwaukee/93617960/", "http://www.wsj.com/articles/thousands-protest-outside-trump-tower-1478742884", "http://nypost.com/2016/11/09/protests-erupt-in-california-after-trump-victory/", "http://nypost.com/2016/11/11/anti-trump-protests-continue-in-wake-of-election/", # Lady Gaga protests ~09.11.2016 "http://www.standard.co.uk/showbiz/celebrity-news/lady-gaga-stages-protest-outside-trump-towers-after-donald-trump-beats-hillary-clinton-a3391526.html", "http://www.mirror.co.uk/3am/celebrity-news/lady-gaga-protests-outside-trump-9228523", "http://www.independent.co.uk/news/people/president-donald-trump-lady-gaga-protest-tower-new-york-a7407081.html", "http://www.telegraph.co.uk/news/2016/11/09/lady-gaga-protests-outside-trump-tower/", "http://www.dailymail.co.uk/news/article-3918926/Hollywood-starts-panic-results-aren-t-going-Clinton-s-way.html", # Croydon tram accident "https://www.theguardian.com/uk-news/2016/nov/09/croydon-tram-crash-kills-at-least-seven-and-injures-more-than-50", "http://www.standard.co.uk/news/transport/croydon-tram-derailment-people-trapped-after-tram-overturns-in-at-sandilands-a3390796.html", "http://www.bbc.com/news/uk-england-london-37919658", "http://www.express.co.uk/news/uk/730639/Croydon-tram-crash-carnage-survivor-derailment-seven-dead", "http://www.mirror.co.uk/news/uk-news/huge-rescue-operation-sandilands-station-9226276", "http://www.dailymail.co.uk/wires/pa/article-3919284/Five-trapped-40-injured-tram-overturns-tunnel.html", "http://www.telegraph.co.uk/news/2016/11/10/croydon-tram-crash-police-check-drivers-mobile-phone-records/", "https://www.thesun.co.uk/news/2150294/croydon-tram-crash-derailment-cause/", "http://www.thetimes.co.uk/article/at-least-four-dead-and-dozens-injured-as-tram-derails-vqpsbrjb3", "http://www.independent.co.uk/news/uk/home-news/five-trapped-40-injured-after-tram-overturns-south-london-croydon-a7406496.html", "http://nypost.com/2016/11/09/several-dead-and-dozens-injured-after-tram-overturns-in-london/", "http://www.usatoday.com/story/news/2016/11/09/least-7-killed-tram-accident-south-london/93549248/", "http://www.nytimes.com/2016/11/10/world/europe/tram-derails-croydon-london.html", # Croydon tram accident follow-up "http://www.standard.co.uk/news/london/croydon-tram-crash-police-identify-all-seven-victims-killed-in-derailment-tragedy-a3394126.html", "http://www.independent.co.uk/news/uk/home-news/croydon-tram-crash-victims-named-last-london-derail-tributes-a7414006.html", "https://www.thesun.co.uk/news/2172708/croydon-tram-crash-victims-named/", "http://www.dailymail.co.uk/news/article-3929748/Croydon-tram-crash-carriages-carried-away-lorry-police-probe-claims-derailed-just-days-seven-people-died-tragedy.html", "http://www.thetimes.co.uk/article/young-father-killed-in-tram-crash-x3zh5dp0v", #shooting near protests ~10.11.2016 "https://www.thesun.co.uk/news/2154878/shot-seattle-protests-donald-trump-election/", "http://www.express.co.uk/news/world/730721/Five-gunned-Seattle-shot-anti-Donald-Trump-US-President-Washington-victory-Republican", "http://www.dailymail.co.uk/news/article-3922446/Report-shooting-multiple-victims-near-Trump-protest-Seattle-PD.html", "http://www.wsj.com/articles/five-people-shot-in-seattle-unclear-if-connected-to-trump-protest-1478749638", # Trump meets Obama in the white house "http://www.bbc.com/news/election-us-2016-37932231", "http://www.dailymail.co.uk/news/article-3922932/Transition-Obama-Trump-meet-White-House.html", "http://www.standard.co.uk/news/world/barack-obama-describes-first-meeting-with-donald-trump-at-white-house-as-excellent-a3392866.html", "https://www.theguardian.com/us-news/live/2016/nov/10/donald-trump-barack-obama-white-house-us-election-live-updates", "http://www.independent.co.uk/news/world/americas/donald-trump-meets-barack-obama-body-language-president-president-elect-a7412186.html", "http://www.mirror.co.uk/news/world-news/donald-trump-barack-obama-hold-9234917", "http://www.thetimes.co.uk/article/rbtrump-3dc5hngts", "http://www.thetimes.co.uk/article/two-bitter-rivals-meet-at-the-white-house-br30bmkhn", "http://www.express.co.uk/news/world/730940/Donald-Trump-Barack-Obama-White-House-Washington-US-election-2016", "http://www.telegraph.co.uk/news/2016/11/10/donald-trump-and-barack-obamas-meeting-was-awkward-they-looked-l/", "https://www.thesun.co.uk/news/2159514/donald-trump-arrives-in-washington-ahead-of-power-transition-talks-with-president-barack-obama/", "https://www.washingtonpost.com/news/post-politics/wp/2016/11/10/obama-to-welcome-trump-to-white-house-for-first-meeting-since-election/", "http://www.usatoday.com/story/news/politics/elections/2016/11/10/obama-trump-white-house-transition/93581810/", "http://www.wsj.com/articles/trump-obama-set-to-begin-transition-1478787730", # german consulate attack ~10.11.2016 "https://www.theguardian.com/world/2016/nov/10/taliban-attack-german-consulate-mazar-i-sharif-afghanistan-nato-airstrikes-kunduz", "http://www.bbc.com/news/world-asia-37944115", "http://www.independent.co.uk/news/world/middle-east/german-consulate-afghanistan-attacked-bomb-suicide-taliban-revenge-mazar-i-sharif-kunduz-attack-two-a7410746.html", "http://www.thetimes.co.uk/article/two-killed-in-bomb-attack-on-consulate-mttnh9pt9", "https://www.thesun.co.uk/news/2162467/taliban-suicide-bomber-truck-german-consulate-afghanistan-killing-two/amp/", "http://www.telegraph.co.uk/news/2016/11/10/taliban-attack-german-consulate-in-northern-afghan-city-of-mazar/", "http://www.express.co.uk/news/world/731052/German-consulate-explosion-gunfire-Afghanistan", "http://www.nytimes.com/2016/11/11/world/asia/taliban-strike-german-consulate-in-afghan-city-of-mazar-i-sharif.html?mtrref=query.nytimes.com&gwh=792F9F9ECEB17B00C71C4F8444293AD8&gwt=pay", "http://www.wsj.com/articles/german-consulate-in-afghanistan-attacked-1478817411" # Clinton blames FBI director Comey ~12.11.2016 "http://www.independent.co.uk/news/people/hillary-clinton-blames-fbi-director-james-comeys-decision-to-to-reopen-email-probe-for-defeat-to-a7414021.html", "http://www.thetimes.co.uk/article/clinton-accuses-fbi-chief-of-costing-her-the-election-7czltxsxm", "http://www.bbc.com/news/election-us-2016-37963965", "https://www.thesun.co.uk/news/2168742/hillary-clinton-aide-blames-us-presidential-loss-to-donald-trump-on-fbi-chief-because-he-cleared-her-of-wrongdoing-over-weiner-emails/", "http://www.telegraph.co.uk/news/2016/11/12/hillary-clinton-blames-election-loss-on-fbis-james-comey-in-call/", "http://www.express.co.uk/news/world/731721/I-BLAME-COMEY-Bitter-Clinton-blames-FBI-chief-James-Comey-election-defeat-Donald-Trump", "http://www.mirror.co.uk/news/world-news/hillary-clinton-blames-fbi-director-9250867", "http://www.dailymail.co.uk/news/article-3930928/Hillary-Clinton-blames-FBI-Director-James-Comey-election-defeat.html", "https://www.theguardian.com/us-news/2016/nov/12/hillary-clinton-james-comey-letters-emails-election-defeat", "http://nypost.com/2016/11/12/clinton-blames-comeys-email-probe-for-her-defeat/", "http://www.wsj.com/articles/hillary-clinton-attributes-fbi-letters-as-factor-in-election-loss-1478994890", "https://www.washingtonpost.com/news/post-politics/wp/2016/11/12/hillary-clinton-blames-one-comey-letter-for-stopping-momentum-and-the-other-for-turning-out-trump-voters/", "http://www.nytimes.com/2016/11/13/us/politics/hillary-clinton-james-comey.html", # F1 Grand Prix Brazil ~13.11.2016 # Hamilton wins "http://www.bbc.com/sport/formula1/37953887", "http://www.telegraph.co.uk/formula-1/2016/11/13/brazilian-grand-prix-live/", "http://www.thetimes.co.uk/article/champion-reigns-in-drenched-brazil-to-keep-title-hopes-alive-t922kw5hj", "https://www.thesun.co.uk/sport/2177105/lewis-hamilton-wins-the-brazilian-grand-prix-after-two-red-flags/", "https://www.theguardian.com/sport/live/2016/nov/13/f1-brazilian-grand-prix-live", # Verstappen avoids crash "https://www.theguardian.com/sport/blog/2016/nov/14/max-verstappen-brazilian-grand-prix-felipe-massa", "http://www.telegraph.co.uk/formula-1/2016/11/13/max-verstappen-even-stuns-his-dad-by-storming-home-into-third-pl/", "http://www.express.co.uk/sport/f1-autosport/731858/Max-Verstappen-avoids-crash-Kimi-Raikkonen-Brazilian-Grand-Prix-wet", "http://www.mirror.co.uk/sport/formula-1/red-bull-boss-christian-horner-9254708", "http://www.dailymail.co.uk/sport/formulaone/article-3932890/Max-Verstappen-amazes-Red-Bull-principal-Christian-Horner-performance-Brazil-witnessed-special.html", "http://www.dailymail.co.uk/sport/sportsnews/article-3934424/Formula-One-star-Max-Verstappen-shows-nerves-steel-avoid-accident.html", # Reikonnen / Massa crash "http://www.mirror.co.uk/sport/formula-1/brazilian-f1-grand-prix-riddled-9253267", "http://www.dailymail.co.uk/sport/formulaone/article-3932386/F1-legend-Felipe-Massa-makes-emotional-farewell-crashing-Brazil-Grand-Prix-Interlagos.html", "https://www.thesun.co.uk/sport/2177804/felipe-massa-retires-f1-legend-makes-a-very-emotional-farewell-after-crashing-in-his-last-home-race-in-brazil/", "http://www.dailymail.co.uk/sport/formulaone/article-3932252/Brazilian-Grand-Prix-thrown-chaos-Kimi-Raikkonen-accident-brings-red-flag-Sebastian-Vettel-fumes-stupid-conditions-mad.html", "http://www.standard.co.uk/sport/brazilian-grand-prix-redflagged-after-dramatic-kimi-raikkonen-crash-a3394411.html", "http://www.usatoday.com/story/sports/motor/formula1/2016/11/13/brazils-massa-crashes-but-gets-warm-farewell-at-home-gp/93771246/", "https://www.washingtonpost.com/sports/auto-racing/brazils-massa-crashes-but-gets-warm-farewell-at-home-gp/2016/11/13/007a15d4-a9e6-11e6-8f19-21a1c65d2043_story.html" ], "overwrite_heuristics": { "meta_contains_article_keyword": true, "og_type": false, "linked_headlines": false, "self_linked_headlines": false }, } ] }
added in https://github.com/fhamborg/news-please/wiki/user-guide
fhamborg
commented
7 years ago fhamborg
commented
7 years ago
Hi Felix,
du musst nur den Download-Crawler verwenden und die Heuristiken ausschalten. Im Anhang findest du ein entsprechendes JSON.
Beste Grüße Sören