search

sergiotapia / magnetissimo

Web application that indexes all popular torrent sites, and saves it to the local database.
MIT License
2.99k stars 187 forks source link

Error: (FunctionClauseError) no function clause matching in Floki.Finder.traverse/4 #34

Closed terwanerik closed 6 years ago

terwanerik commented 7 years ago

I have to say, great project, but after crawling for about 5 minutes i'm getting this error:

Error: https://www.limetorrents.cc/browse-torrents/Anime/date/1/ just ain't workin.
[error] GenServer #PID<0.455.0> terminating
** (FunctionClauseError) no function clause matching in Floki.Finder.traverse/4
    lib/floki/finder.ex:61: Floki.Finder.traverse(nil, [], %Floki.Selector{attributes: [], classes: ["tt-name"], combinator: %Floki.Combinator{match_type: :descendant, selector: %Floki.Selector{attributes: [], classes: [], combinator: nil, id: nil, namespace: nil, type: "a"}}, id: nil, namespace: nil, type: nil}, [])
    lib/floki/finder.ex:73: Floki.Finder.traverse/4
    lib/floki/finder.ex:47: Floki.Finder.find_selectors/2
    (magnetissimo) lib/crawler/limetorrents.ex:74: Magnetissimo.Crawler.LimeTorrents.torrent_links/1
    (magnetissimo) lib/crawler/limetorrents.ex:37: Magnetissimo.Crawler.LimeTorrents.process/2
    (magnetissimo) lib/crawler/limetorrents.ex:26: Magnetissimo.Crawler.LimeTorrents.handle_info/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :work

Looks like it fails when it cannot reach a certain site.

nvanmeurs commented 7 years ago

I got the exact same issue:

Error: https://www.limetorrents.cc/browse-torrents/Anime/date/1/ just ain't workin.
[error] GenServer #PID<0.435.0> terminating
** (FunctionClauseError) no function clause matching in Floki.Finder.traverse/4
    lib/floki/finder.ex:61: Floki.Finder.traverse(nil, [], %Floki.Selector{attributes: [], classes: ["tt-name"], combinator: %Floki.Combinator{match_type: :descendant, selector: %Floki.Selector{attributes: [], classes: [], combinator: nil, id: nil, namespace: nil, type: "a"}}, id: nil, namespace: nil, type: nil}, [])
    lib/floki/finder.ex:73: Floki.Finder.traverse/4
    lib/floki/finder.ex:47: Floki.Finder.find_selectors/2
    (magnetissimo) lib/crawler/limetorrents.ex:74: Magnetissimo.Crawler.LimeTorrents.torrent_links/1
    (magnetissimo) lib/crawler/limetorrents.ex:37: Magnetissimo.Crawler.LimeTorrents.process/2
    (magnetissimo) lib/crawler/limetorrents.ex:26: Magnetissimo.Crawler.LimeTorrents.handle_info/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :work

OS: Win 10 X64 DB: Postgres 9.6 Elixir: 1.4.1

It runs for a couple of minutes, then fails and crashes.

sergiotapia commented 7 years ago

Thanks, I'll take a look at the parsing error and correct the crawler.

nvanmeurs commented 7 years ago

It seems to be stable now, it might just be that anime page.

yaroto98 commented 7 years ago

I'm currently hitting this error too. As said above, runs for a few minutes, fails, and crashes.

sergiotapia commented 7 years ago

image

Yup, sometimes Limetorrents just chokes out. I have a fix for this in mind.

jasonm23 commented 7 years ago

Also getting same error, while it's crawling limetorrents

yaroto98 commented 7 years ago

Having hit this a few more times today, the biggest problem isn't the error itself. It's that the app crashes.

The app should toss the error into a log and keep on processing.

comotion commented 7 years ago

yup, this crashes the server here as well. I pretty much restart magnetissimo the 5 minutes I do use it every day as it's just guranteed to crash again with this bug.

sergiotapia commented 7 years ago

I've removed the LimeTorrents parser in #72 because they're riddled with ads now. I added in other parsers though. Once the PR lands Magnetissimo will be much more stable and recover gracefully.

tchoutri commented 6 years ago

We'll consider LimeTorrent in the future. For now I'm closing this issue, please reopen one if the new crawler doesn't work.