search

sergiotapia / magnetissimo

Web application that indexes all popular torrent sites, and saves it to the local database.
MIT License
2.99k stars 187 forks source link

GenServer terminates on duplicate torrents #95

Closed skwerlman closed 6 years ago

skwerlman commented 6 years ago

Using f0e20b639773e7919d25eeaf5989015f72119cca, GenServer terminates on the first duplicate torrent, before handling any other torrents in the queue. The scrapers are then restarted after only a few seconds, ignoring the intended 30min wait time.

┌─[arch@home] - [/mnt/code/magnetissimo] - [Wed Jan 03, 01:04]
└─[$]> mix phx.server
[info] Running MagnetissimoWeb.Endpoint with Cowboy using http://0.0.0.0:4000
[info] Starting Crawler Supervisor
[info] Starting EZTV crawler
[info] Starting NyaaPantsu crawler
[info] Starting NyaaSi crawler
[info] [NyaaSi] Downloading torrents from page: https://nyaa.si/rss?c=1_0&m
[debug] QUERY ERROR db=4.1ms
INSERT INTO "torrents" ("leechers","magnet","name","outbound_url","seeders","size","website_source","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) RETURNING "id" [0, "magnet:?xt=urn:btih:YRSHWD254WXMCPSZ2UNHEMGLBRN36PBA&dn=%5BAVT%5D+Inferno+Cop+%5B720p%5D+%5B8+Bits%5D+%5BPT-BR%5D&tr=http%3A%2F%2Fnyaa.tracker.wf%3A7777%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce", "[AVT] Inferno Cop [720p] [8 Bits] [PT-BR]", "https://nyaa.si/view/992965", 1, "626943590", "nyaasi", {{2018, 1, 3}, {6, 4, 50, 718434}}, {{2018, 1, 3}, {6, 4, 50, 720028}}]
[info] Starting NyaaSi crawler
[error] GenServer #PID<0.465.0> terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * unique: torrents_magnet_index

If you would like to convert this constraint into an error, please
call unique_constraint/3 in your changeset and define the proper
constraint name. The changeset defined the following constraints:

    * unique: unique_magnet_name

    (ecto) lib/ecto/repo/schema.ex:574: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
    (elixir) lib/enum.ex:1273: Enum."-map/2-lists^map/1-0-"/2
    (ecto) lib/ecto/repo/schema.ex:559: Ecto.Repo.Schema.constraints_to_errors/3
    (ecto) lib/ecto/repo/schema.ex:222: anonymous fn/14 in Ecto.Repo.Schema.do_insert/4
    (magnetissimo) lib/magnetissimo/contents.ex:9: Magnetissimo.Contents.save_torrent/1
    (magnetissimo) lib/magnetissimo/crawler/nyaasi.ex:77: anonymous fn/2 in Magnetissimo.Crawler.NyaaSi.process/2
    (elixir) lib/enum.ex:1829: Enum."-reduce/3-lists^foldl/2-0-"/3
    (magnetissimo) lib/magnetissimo/crawler/nyaasi.ex:76: Magnetissimo.Crawler.NyaaSi.process/2
    (magnetissimo) lib/magnetissimo/crawler/nyaasi.ex:61: Magnetissimo.Crawler.NyaaSi.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :work
State: {[page_link: "https://nyaa.si/rss?c=6_0&m", page_link: "https://nyaa.si/rss?c=5_0&m"], [page_link: "https://nyaa.si/rss?c=1_0&m", page_link: "https://nyaa.si/rss?c=2_0&m", page_link: "https://nyaa.si/rss?c=3_0&m", page_link: "https://nyaa.si/rss?c=4_0&m"]}
[debug] QUERY ERROR db=0.9ms
INSERT INTO "torrents" ("leechers","magnet","name","outbound_url","seeders","size","website_source","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) RETURNING "id" [0, "magnet:?xt=urn:btih:7345C64CB7B87189180453716FC2CB3F8DF638E5&dn=NCIS.S15E11.iNTERNAL.720p.WEB.x264-BAMBOOZLE%5Beztv%5D.mkv&tr=udp%3A%2F%2Ftracker.publicbt.com%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Ftracker.trackerfix.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Fexodus.desync.com%3A6969", "NCIS S15E11 iNTERNAL 720p WEB x264-BAMBOOZLE", "https://eztv.ag/ep/509527/ncis-s15e11-internal-720p-web-x264-bamboozle/", 0, "934.63 MB", "eztv", {{2018, 1, 3}, {6, 4, 53, 611617}}, {{2018, 1, 3}, {6, 4, 53, 611625}}]
[error] GenServer Magnetissimo.Crawler.EZTV terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * unique: torrents_magnet_index

If you would like to convert this constraint into an error, please
call unique_constraint/3 in your changeset and define the proper
constraint name. The changeset defined the following constraints:

    * unique: unique_magnet_name

    (ecto) lib/ecto/repo/schema.ex:574: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
    (elixir) lib/enum.ex:1273: Enum."-map/2-lists^map/1-0-"/2
    (ecto) lib/ecto/repo/schema.ex:559: Ecto.Repo.Schema.constraints_to_errors/3
    (ecto) lib/ecto/repo/schema.ex:222: anonymous fn/14 in Ecto.Repo.Schema.do_insert/4
    (magnetissimo) lib/magnetissimo/contents.ex:9: Magnetissimo.Contents.save_torrent/1
    (elixir) lib/enum.ex:678: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:678: Enum.each/2
    (magnetissimo) lib/magnetissimo/crawler/eztv.ex:49: Magnetissimo.Crawler.EZTV.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :work
State: [[{"title", [], ["NCIS S15E11 iNTERNAL 720p WEB x264-BAMBOOZLE"]}, {"category", [], ["TV"]}, {"link", [], ["https://eztv.ag/ep/509527/ncis-s15e11-internal-720p-web-x264-bamboozle/"]}, {"guid", [], ["https://eztv.ag/ep/509527/ncis-s15e11-internal-720p-web-x264-bamboozle/"]}, {"pubdate", [], ["Wed, 03 Jan 2018 00:12:01 -0500"]}, {"torrent:contentlength", [], ["980026574"]}, {"torrent:infohash", [], ["7345C64CB7B87189180453716FC2CB3F8DF638E5"]}, {"torrent:magneturi", [], ["magnet:?xt=urn:btih:7345C64CB7B87189180453716FC2CB3F8DF638E5&dn=NCIS.S15E11.iNTERNAL.720p.WEB.x264-BAMBOOZLE%5Beztv%5D.mkv&tr=udp%3A%2F%2Ftracker.publicbt.com%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Ftracker.trackerfix.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Fexodus.desync.com%3A6969"]}, {"torrent:seeds", [], ["0"]}, {"torrent:peers", [], ["0"]}, {"torrent:verified", [], ["0"]}, {"torrent:filename", [], ["NCIS.S15E11.iNTERNAL.720p.WEB.x264-BAMBOOZLE[eztv].mkv"]}, {"enclosure", [{"url", "https://zoink.ch/torrent/NCIS.S15E11.iNTERNAL.720p.WEB.x264-BAMBOOZLE[eztv].mkv.torrent"}, {"length", "980026574"}, {"type", "application/x-bittorrent"}], []}], [{"title", [], ["The Untitled Action Bronson Show 2018 01 02 720p WEB x264-TBS"]}, {"category", [], ["TV"]}, {"link", [], ["https://eztv.ag/ep/509526/the-untitled-action-bronson-show-2018-01-02-720p-web-x264-tbs/"]}, {"guid", [], ["https://eztv.ag/ep/509526/the-untitled-action-bronson-show-2018-01-02-720p-web-x264-tbs/"]}, {"pubdate", [], ["Wed, 03 Jan 2018 00:06:01 -0500"]}, {"torrent:contentlength", [], ["422424650"]}, {"torrent:infohash", [], ["9D5CAF52DA0CBC29CFF066BCFD3693D548A0D255"]}, {"torrent:magneturi", [], ["magnet:?xt=urn:btih:9D5CAF52DA0CBC29CFF066BCFD3693D548A0D255&dn=The.Untitled.Action.Bronson.Show.2018.01.02.720p.WEB.x264-TBS%5Beztv%5D.mkv&tr=udp%3A%2F%2Ftracker.publicbt.com%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Ftracker.trackerfix.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Fexodus.desync.com%3A6969"]}, {"torrent:seeds", [], ["0"]}, {"torrent:peers", [], ["0"]}, {"torrent:verified", [], ["0"]}, {"torrent:filename", [], ["The.Untitled.Action.Bronson.Show.2018.01.02.720p.WEB.x264-TBS[eztv].mkv"]}, {"enclosure", [{"url", "https://zoink.ch/torrent/The.Untitled.Action.Bronson.Show.2018.01.02.720p.WEB.x264-TBS[eztv].mkv.torrent"}, {"length", "422424650"}, {"type", "application/x-bittorrent"}], []}], [{"title", [], ["The Opposition with Jordan Klepper 2018 01 02 Natasha Bertrand WEB x264-TBS"]}, {"category", [], ["TV"]}, {"link", [], ["https://eztv.ag/ep/509242/the-opposition-with-jordan-klepper-2018-01-02-natasha-bertrand-web-x264-tbs/"]}, {"guid", [], ["https://eztv.ag/ep/509242/the-opposition-with-jordan-klepper-2018-01-02-natasha-bertrand-web-x264-tbs/"]}, {"pubdate", [], ["Tue, 02 Jan 2018 23:51:02 -0500"]}, {"torrent:contentlength", [], ["296572333"]}, {"torrent:infohash", [], ["8919EB9FBD59064290C99912FB00C5AAFB99C428"]}, {"torrent:magneturi", [], ["magnet:?xt=urn:btih:8919EB9FBD59064290C99912FB00C5AAFB99C428&dn=The.Opposition.with.Jordan.Klepper.2018.01.02.Natasha.Bertrand.WEB.x264-TBS%5Beztv%5D.mkv&tr=udp%3A%2F%2Ftracker.publicbt.com%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Ftracker.trackerfix.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Fexodus.desync.com%3A6969"]}, {"torrent:seeds", [], ["0"]}, {"torrent:peers", [], ["0"]}, {"torrent:verified", [], ["0"]}, {"torrent:filename", [], ["The.Opposition.with.Jordan.Klepper.2018.01.02.Natasha.Bertrand.WEB.x264-TBS[eztv].mkv"]}, {"enclosure", [{"url", "https://zoink.ch/torrent/The.Opposition.with.Jordan.Klepper.2018.01.02.Natasha.Bertrand.WEB.x264-TBS[eztv].mkv.torrent"}, {"length", "296572333"}, {"type", "application/x-bittorrent"}], []}], [{"title", [], ["The Opposition with Jordan Klepper 2018 01 02 Natasha Bertrand 720p WEB x264-TBS"]}, {"category", [], ["TV"]}, {"link", [], ["https://eztv.ag/ep/509241/the-opposition-with-jordan-klepper-2018-01-02-natasha-bertrand-720p-web-x264-tbs/"]}, {"guid", [], ["https://eztv.ag/ep/509241/the-opposition-with-jordan-klepper-2018-01-02-natasha-bertrand-720p-web-x264-tbs/"]}, {"pubdate", [], ["Tue, 02 Jan 2018 23:51:02 -0500"]}, {"torrent:contentlength", [], ["422167003"]}, {"torrent:infohash", [], ["A269E6CDA5FF204550F5264720A329E0088BE97A"]}, {"torrent:magneturi", [], ["magnet:?xt=urn:btih:A269E6CDA5FF204550F5264720A329E0088BE97A&dn=The.Opposition.with.Jordan.Klepper.2018.01.02.Natasha.Bertrand.720p.WEB.x264-TBS%5Beztv%5D.mkv&tr=udp%3A%2F%2Ftracker.publicbt.com%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Ftracker.trackerfix.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Fexodus.desync.com%3A6969"]}, {"torrent:seeds", [], ["0"]}, {"torrent:peers", [], ["0"]}, {"torrent:verified", [], ["0"]}, {"torrent:filename", [], ["The.Opposition.with.Jordan.Klepper.2018.01.02.Natasha.Bertrand.720p.WEB.x264-TBS[eztv].mkv"]}, {"enclosure", [{"url", "https://zoink.ch/torrent/The.Opposition.with.Jordan.Klepper.2018.01.02.Natasha.Bertrand.720p.WEB.x264-TBS[eztv].mkv.torrent"}, {"length", "422167003"}, {"type", "application/x-bittorrent"}], []}], [{"title", [], ["The Middle S09E11 HDTV x264-KILLERS"]}, {"category", [], ["TV"]}, {"link", [], ["https://eztv.ag/ep/509240/the-middle-s09e11-hdtv-x264-killers/"]}, {"guid", [], ["https://eztv.ag/ep/509240/the-middle-s09e11-hdtv-x264-killers/"]}, {"pubdate", [], ["Tue, 02 Jan 2018 23:48:01 -0500"]}, {"torrent:contentlength", [], ["174482558"]}, {"torrent:infohash", [], ["841AADE3B1922568FFAF84483338EDE2D169EA81"]}, {"torrent:magneturi", [], ["magnet:?xt=urn:btih:841AADE3B1922568FFAF84483338EDE2D169EA81&dn=The.Middle.S09E11.HDTV.x264-KILLERS%5Beztv%5D.mkv&tr=udp%3A%2F%2Ftracker.publicbt.com%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Ftracker.trackerfix.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Fexodus.desync.com%3A6969"]}, {"torrent:seeds", [], ["0"]}, {"torrent:peers", [], ["0"]}, {"torrent:verified", [], ["0"]}, {"torrent:filename", [], ["The.Middle.S09E11.HDTV.x264-KILLERS[eztv].mkv"]}, {"enclosure", [{"url", "https://zoink.ch/torrent/The.Middle.S09E11.HDTV.x264-KILLERS[eztv].mkv.torrent"}, {"length", "174482558"}, {"type", "application/x-bittorrent"}], []}], [{"title", [], ["The Middle S09E11 720p HDTV x264-KILLERS"]}, {"category", [], ["TV"]}, {"link", [], ["https://eztv.ag/ep/509239/the-middle-s09e11-720p-hdtv-x264-killers/"]}, {"guid", [], ["https://eztv. (truncated)
[info] Starting EZTV crawler
[info] [NyaaSi] Downloading torrents from page: https://nyaa.si/rss?c=1_0&m
[debug] QUERY ERROR db=0.8ms
INSERT INTO "torrents" ("leechers","magnet","name","outbound_url","seeders","size","website_source","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) RETURNING "id" [0, "magnet:?xt=urn:btih:YRSHWD254WXMCPSZ2UNHEMGLBRN36PBA&dn=%5BAVT%5D+Inferno+Cop+%5B720p%5D+%5B8+Bits%5D+%5BPT-BR%5D&tr=http%3A%2F%2Fnyaa.tracker.wf%3A7777%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce", "[AVT] Inferno Cop [720p] [8 Bits] [PT-BR]", "https://nyaa.si/view/992965", 1, "626943590", "nyaasi", {{2018, 1, 3}, {6, 4, 58, 53377}}, {{2018, 1, 3}, {6, 4, 58, 53386}}]
[info] Starting NyaaSi crawler
[error] GenServer #PID<0.467.0> terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * unique: torrents_magnet_index

If you would like to convert this constraint into an error, please
call unique_constraint/3 in your changeset and define the proper
constraint name. The changeset defined the following constraints:

    * unique: unique_magnet_name

    (ecto) lib/ecto/repo/schema.ex:574: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
    (elixir) lib/enum.ex:1273: Enum."-map/2-lists^map/1-0-"/2
    (ecto) lib/ecto/repo/schema.ex:559: Ecto.Repo.Schema.constraints_to_errors/3
    (ecto) lib/ecto/repo/schema.ex:222: anonymous fn/14 in Ecto.Repo.Schema.do_insert/4
    (magnetissimo) lib/magnetissimo/contents.ex:9: Magnetissimo.Contents.save_torrent/1
    (magnetissimo) lib/magnetissimo/crawler/nyaasi.ex:77: anonymous fn/2 in Magnetissimo.Crawler.NyaaSi.process/2
    (elixir) lib/enum.ex:1829: Enum."-reduce/3-lists^foldl/2-0-"/3
    (magnetissimo) lib/magnetissimo/crawler/nyaasi.ex:76: Magnetissimo.Crawler.NyaaSi.process/2
    (magnetissimo) lib/magnetissimo/crawler/nyaasi.ex:61: Magnetissimo.Crawler.NyaaSi.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :work
State: {[page_link: "https://nyaa.si/rss?c=6_0&m", page_link: "https://nyaa.si/rss?c=5_0&m"], [page_link: "https://nyaa.si/rss?c=1_0&m", page_link: "https://nyaa.si/rss?c=2_0&m", page_link: "https://nyaa.si/rss?c=3_0&m", page_link: "https://nyaa.si/rss?c=4_0&m"]}
tchoutri commented 6 years ago

Thanks :) I can't reproduce this at home, do you feel like to could indeed turn the constraint into an error by tweaking the changeset function? Also, can you try doing it again, starting from 07396c2 this time?

skwerlman commented 6 years ago

I haven't touched ecto before, but i'll give it a go.

If you can't repro it, it might be OS-related, since Arch linux has a some other unique problems with erlang (and, by extension, elixir)

tchoutri commented 6 years ago

@skwerlman I doubt it, it's too high-level. But I'm using OTP 20 and Elixir 1.5, with kerl and kiex, maybe try with those?

skwerlman commented 6 years ago

i'm using OTP 20.2 with elixir 1.5.3, direct from the arch repos.

Either way, i was able to fix it by adding this line at the end of Magnetissimo.Torrent.changeset:

|> unique_constraint(:magnet, name: :torrents_magnet_index)

Again, i'm not super familiar with ecto, so this might not be the right way to solve it.

tchoutri commented 6 years ago

We'll see :) I'll merge that once I'm home, thank you again!