Anonyfox / elixir-scrape

Scrape any website, article or RSS/Atom Feed with ease!
https://github.com/Anonyfox/elixir-scrape
GNU Lesser General Public License v3.0
327 stars 43 forks source link

technewsworld.com/perl/syndication/rssfull.pl scrap feed failed #5

Closed usmanasif closed 8 years ago

usmanasif commented 8 years ago

use this url and scrap feed will fail

markman123 commented 8 years ago

The full link is http://www.technewsworld.com/perl/syndication/rssfull.pl

Anonyfox commented 8 years ago

Works for me now, the updates seem to have fixed underlying issues. I added a test to ensure it will work from now on: https://github.com/Anonyfox/elixir-scrape/blob/master/test/feed_test.exs#L26

iex(1)> Scrape.feed "http://www.technewsworld.com/perl/syndication/rssfull.pl"
[%{description: "Every once in a while I run into a little company that comes at an existing market as if the folks already in it are idiots -- and sometimes they are right. Here's the thing: What often happens is a company breaks out in a segment, and everyone groups around that company's ideas and emulates them. Few initially stand up and say, \"Wait a minute -- what if they're wrong?\"",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "What If We've Got Big Data and Analytics All Wrong?",
   url: "http://www.technewsworld.com/story/83656.html?rss=1"},
 %{description: "I don't think any product range beats speakers for running the design gamut from gorgeous to ghastly. There are so many to choose from, in all shapes, sizes and colors, that there's something for everyone -- plus a few things no one should buy. But what if your sound system were a piece of art, literally? Case of Bass has created a boombox print that houses a speaker system in a shadow box.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Gadget Ogling: Artsy Audio, Stress Detectives, and Wee Violins",
   url: "http://www.technewsworld.com/story/83660.html?rss=1"},
 %{description: "Google on Wednesday unveiled its latest investment in long-haul undersea fiber optic cabling. Dubbed the \"Faster Cable System,\" it is the highest-capacity undersea cable built to date, providing 60-Tbps bandwidth between the United States and Japan. It is about 10 million times faster than an average cable modem, Google noted. Google is one of six members of the Faster Consortium.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Google Opens Faster Big Data Pipe Between US and Japan",
   url: "http://www.technewsworld.com/story/83659.html?rss=1"},
 %{description: "Devialet, the French company that has built a reputation in the audio market for high-end wireless speakers, on Tuesday announced its newest addition: the $3,000 Phantom Gold. The Phantom Gold looks something like a futuristic Faberge egg, and it's priced like one, too. That said, it has some ear-pleasing and ear-splitting specs. Devialet has given it a power boost to 4,500 watts.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Nothing Ghostly About Devialet's Phantom Gold Sound",
   url: "http://www.technewsworld.com/story/83657.html?rss=1"},
 %{description: "Manufacturers are eager to lay the bricks and help pave the road to better, more personalized healthcare through integration of connected devices in the new Internet of Medical Things. Compared to other industries, healthcare has been \"inherently conservative and slow\" to embrace innovations such as the cloud and the Internet of Things -- but that's changing.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "The Internet of Medical Things, Part 1: A New Concept in Healthcare",
   url: "http://www.technewsworld.com/story/83654.html?rss=1"},
 %{description: "Hackers late Sunday broke into CEO Sundar Pichai's Quora account and through it accessed his Twitter followers, according to reports. The group taking credit for the breach, OurMine Security, previously hit other prominent high-tech figures, including Facebook CEO Mark Zuckerberg, Spotify CEO Daniel Elk, Amazon CTO Werner Vogels and former Twitter CEO Dick Costolo.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Pichai Account Trespassers Claim Their Hacking Heart's in the Right Place",
   url: "http://www.technewsworld.com/story/83651.html?rss=1"},
 %{description: "If you want a classy distro with the look and feel of the Microsoft platform, the MakuluLinux LinDoz edition will feed your nostalgia. MakuluLinux developer Jacque Montague Raymer released MakuluLinux 11 LinDoz edition earlier this month after a year and a half in the making. Forked from the Original Aero Edition, it first appeared with the MakuluLinux 9 series.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Makulu's LinDoz Is a Smooth Windows-Cinnamon Blend",
   url: "http://www.technewsworld.com/story/83647.html?rss=1"},
 %{description: "Twitter fans who want to dress up pix they post to the service can do it with #Stickers, a new feature announced Monday. \"Soon, you can browse our rotating sets of stickers to join in on real-time conversations, and select from the library of hundreds of accessories, emoji, and props to make your photos more fun,\" said Sasank Reddy, a senior product manager at Twitter.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Twitter Tries to Get Stickier",
   url: "http://www.technewsworld.com/story/83648.html?rss=1"},
 %{description: "Anki on Monday introduced Cozmo, a little robot powered by an advanced artificial intelligence. Cozmo promises to be far more interactive than any toy robot that has been released to date. Although he's designed with play in mind, the combination of technology from toys, videogames and robotics represents a major leap forward, according to Cozmo's creators.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "AI Playmate Cozmo Wants to Get to Know You",
   url: "http://www.technewsworld.com/story/83644.html?rss=1"},
 %{description: "We live in an unprecedented age of information and computing power -- yet whether publicly or privately, we still make decisions largely on how we feel rather than based on hard information. With this massive influx of data and computing power, when it comes to issues like global warming or gun control, we are more likely to decide based on how we feel rather than figuring out the facts.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Gun Control and Learning to Make Data-Driven Decisions",
   url: "http://www.technewsworld.com/story/83641.html?rss=1"},
 %{description: "Red Hat this week announced the release of Fedora 24, an open source Linux operating system maintained by the Fedora Project community. Fedora Linux is the community version of Red Hat Enterprise Linux, or RHEL. Fedora 24 is comprised of a set of base packages that form the foundation of three distinct editions: Fedora 24 Cloud, Fedora 24 Server and Fedora 24 Workstation.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Fedora 24 Pushes Linux Boundaries",
   url: "http://www.technewsworld.com/story/83642.html?rss=1"},
 %{description: "BitTorrent on Thursday announced a revamp of its Bundle offering as a pilot project, giving it a new name, \"BitTorrent Now.\" Heavily used by independent musicians and filmmakers, as well as the BBC and other organizations, the service now accepts ads. BitTorrent Now's first streaming app is available for Android. Apps for iOS and Apple TV are in the works.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "BitTorrent Dips Toes Into Ad Stream",
   url: "http://www.technewsworld.com/story/83638.html?rss=1"},
 %{description: "Since mobile computing put an end to the good old days when IT departments had absolute control over software deployed in the enterprise, there's been a rise in employees' use of third-party applications -- a rise that poses security risks to corporate environments. That is one of the findings in a report CloudLock released last week.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Study: Third-Party Apps Pose Risks for Enterprises",
   url: "http://www.technewsworld.com/story/83635.html?rss=1"},
 %{description: "The rumor that Apple will get rid of the headphone jack in the iPhone 7 resurfaced Tuesday. Apple is expected to launch the new iPhone later this year. The iPhone's Lightning connector reportedly will do double duty as its charging port and headphone connector. The result will be a thinner device and better water resistance. If true, the move \"is right in terms of timing,\" said analyst Jeff Orr.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Headphone Jack-less iPhone 7 Rumor Gains Ground",
   url: "http://www.technewsworld.com/story/83634.html?rss=1"},
 %{description: "Twitter on Tuesday launched enhanced video services that will allow users to post longer videos -- up to 140 seconds. It also announced a new platform for celebrities and public figures to share their stories. Because sports, news, music and other videos increasingly are the type of content that goes viral on Twitter, the company decided to expand content parameters.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Twitter Applies '140' Rule to Video Seconds",
   url: "http://www.technewsworld.com/story/83633.html?rss=1"},
 %{description: "Docker on Monday announced Docker Engine 1.12 with built-in orchestration, which allows automated deployment and management of Dockerized distributed applications and microservices at scale in production. Users can select Docker Swarm mode to turn on built-in orchestration, or they can use their own custom tooling or third-party orchestrators that run on Docker Engine.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Docker Tunes Up Engine Orchestration",
   url: "http://www.technewsworld.com/story/83632.html?rss=1"},
 %{description: "Google on Monday began rolling out a new two-step authentication feature, Google Prompt, targeting enterprise employees. The new option consists of a pop-up that displays a mobile user's name and profile image, and that specifies the location and device involved in the attempted sign-in. The device owner is asked whether to allow or deny the sign-in.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Google Makes It Easier to Do the 2-Step",
   url: "http://www.technewsworld.com/story/83629.html?rss=1"},
 %{description: "Lenovo has peeled back the curtain on a smartphone that you can wrap around your wrist, sporting a full-color screen. The \"CPlus,\" as Lenovo calls the prototype, runs Android and has a 4.26-inch display. It will be available in 12 colors if and when it goes on sale. It's clear bendable smartphones will be a viable consumer product in the near future.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Gadget Ogling: Pliable Phones, Clever Kicks, and Sumptuous Speakers",
   url: "http://www.technewsworld.com/story/83628.html?rss=1"},
 %{description: "Plume last week introduced its eponymous self-optimizing WiFi system for the home. Plume adapts in real life to the different network demands made by various devices and ensures that each gets the fastest speeds possible, according to the company. The recommended setup is one pod per room and connecting space, such as a hallway, plus any cable or DSL modem or any modem/router combination.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Plume Podifies Home WiFi",
   url: "http://www.technewsworld.com/story/83626.html?rss=1"},
 %{description: "Andy Rubin, Android's daddy, last week made some interesting comments about quantum computing and artificial intelligence. The part I agree with is it won't be long until most things we have are connected to an intelligent machine. However, when referring to something that will be far smarter than we are, the use of the term \"artificial\" would not just be inaccurate -- it would be rude.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "AI for President",
   url: "http://www.technewsworld.com/story/83616.html?rss=1"},
 %{description: "Snapcraft -- the Linux package format Canonical developed for Ubuntu -- now works on multiple Linux distros, including Arch, Debian, Fedora and various flavors of Ubuntu. They're being validated on CentOS, Elementary, Gentoo, Mint, OpenSUSE, OpenWrt and RHEL. \"Distributing applications on Linux is not always easy,\" said Canonical's Manik Taneja, product manager for Snappy Ubuntu Core.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Linux Snap Package Format Goes Multi-Distro",
   url: "http://www.technewsworld.com/story/83623.html?rss=1"},
 %{description: "Google's Cardboard has proved to the masses that virtual reality is more than a pipe dream. Inexpensive cardboard headsets leverage smartphones to create makeshift head-mounted displays for low-level VR experiences. For those whose fancies of owning a US$600 Rift or a $900 Vive were out of reach, Cardboard was a way to keep their imaginations captive while Google was dreaming of Daydream.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Google: Dare to Daydream",
   url: "http://www.technewsworld.com/story/83620.html?rss=1"},
 %{description: "Facebook on Thursday unveiled its redesigned Messenger app, which replaces the list of conversations it previously displayed with a home screen that will let users perform more actions within a chat. The Home tab now organizes Messenger into recent conversations, favorites, active users, and messages awaiting a response. It has a shortcut that reminds users about Facebook friends' birthdays.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Facebook Gives Messenger More Jobs to Do",
   url: "http://www.technewsworld.com/story/83619.html?rss=1"},
 %{description: "Nintendo on Tuesday offered E3 attendees a preview of The Legend of Zelda: Breath of the Wild, the latest title in the action-adventure franchise. This time the story involves much more than rescuing a princess. The game actually has been years in the making, and at E3 Nintendo pulled out all stops, devoting much of its booth to the game. Nintendo presented two different demos of Breath of Wild.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Nintendo's New Zelda Adventure Draws E3 Cheers",
   url: "http://www.technewsworld.com/story/83613.html?rss=1"},
 %{description: "Apple finally responded to concerns that were threatening to scuttle its once-promising wearables category. Its introduction of watchOS 3 at this week's WWDC has drawn high praise. The Apple Watch represented the company's attempt to marry the ultimate Dick Tracy smart gadget with the genius of the iPhone, but it fell short. Instead of a convenience, it was far too slow and complicated.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "WatchOS 3 Offers New Hope for Smartwatch Category",
   url: "http://www.technewsworld.com/story/83614.html?rss=1"},
 %{description: "Facebook plans to roll out its suicide prevention tools and resources around the world. Already available in the U.S., the resources are designed to help members who may be thinking about committing suicide or otherwise hurting themselves, as well as family and friends who are concerned about their loved ones' well being. The tools are available in the local languages where members are based.",
   image: nil, pubdate: #<Date(2016-07-04)>, tags: [],
   title: "Facebook Tackles Suicide Prevention Worldwide",
   url: "http://www.technewsworld.com/story/83612.html?rss=1"}]
iex(2)>