Dedupe snatched releases

[jkaberg]

I think the image says everything (yes all 6 got downloaded).

As you're already recording every release that get snatched, perhaps an "dedupe" feature which matches release against already snatched torrents and disregard if exists?

I've raised this issue with the Sonarr team, however they've disregarded it.

[weannounce]

Arrnounced keeps track of snatched releases only for the user to get an overview. It has no knowledge of what in fact is in your media library and can therefore not determine if something should be filtered out. I can imagine a scenario, albeit a bit far-fetched, where something was snatched, then removed from the library but then released again. In such a scenario it should not be filtered out. Repacks is another scenario which Sonarr already handles. Wouldn't make sense for Arrnounced to also implement all that.

I find it weird that Sonarr downloaded all these releases. Sonarr is the authority when it comes to what to download and what is already available/being downloaded. What was the reason they disregarded it? Do you have a link to the issue?

[jkaberg]

@weannounce I think you missunderstand, I meant this to be an isolated arrnounced feature. arrnounced shouldn't know whats in our librarys, it simply and only knows what its already snatched (by looking in the database).

It should only discard/dedup an exact match of the release title, and allow all other releases. An example would be The nevers S01E04 720p WEB H264-GGWP is pushed several times and get deduped after the 1 snatch, however The nevers S01E04 720p WEB H264-GGWP-PROPER would still be allowed (1 snatch) and deduped/discarded the following pushes.

So the code bellow would happend BEFORE any push to Sonarr and doesnt need to talk to *arr to evaluate any release.

Psuedo code;

// snatched is an table in the database, titles is an column with rows containg the title for previously snatched releases (eg The nevers S01E04 720p WEB H264-GGWP)
// release is an object with attributes, where title is one (for example title could be The nevers S01E04 720p WEB H264-GGWP)
// config.dedup_snatched is an boolean
if release.title in snatched.titles and config.dedup_snatched:
    continue
    log.debug('Dropping {0} as its already been snatched'.format(release.title))
else:
    send_to_sonarr(release)
    commit_db(table=snatched, release.title)

My wording is wrong, they didnt discard it. They simply didn't implement a fix; https://github.com/Sonarr/Sonarr/issues/3282 and https://github.com/Sonarr/Sonarr/issues/2975

[weannounce]

If I interpret the Sonarr issues correct, the same release will be grabbed multiple times if they are handled concurrently. Since they are processed at the same time Sonarr has not yet taken a decision to grab any of the pushes i.e. no reply has been sent to Arrnounced which would not know that it should suppress some of the releases. This scenario should be visible in the log. Three instances of "notifying" followed by three instances of approved.

Notifying Sonarr of release from ...
Notifying Sonarr of release from ...
Notifying Sonarr of release from ...
...
Sonarr approved release: ...
Sonarr approved release: ...
Sonarr approved release: ...

There is another option where Arrnounced could match earlier announcements (which has yet to be grabbed). But it would still not prevent different releases of the same title to be grabbed.

Creating a queue, as suggested in Sonarr/Sonarr#2975, for each backend could be an option but would, as the issue states, increase the delay. I will have to think about that.

A, not very nice, workaround would be to configure a delay for the trackers. E.g. no delay for one tracker, 1 second for the next, 2 seconds for the last one. Still not foolproof though.

[jkaberg]

@weannounce yes its a short comeing on Sonarrs end. So you think my suggestion for solution above will not mitigate this? From my POV I can't see why not

[weannounce]

Due to the reason I described I do not believe your suggestion would mitigate it. At the time of pushing the release to Sonarr, Arrnounced do not yet know if it will be grabbed.

[weannounce]

Are you able to provide the log from when this happened?

[weannounce]

Sonarr does currently not handle concurrent releases without the risk of grabbing the same release more than once. If the releases are pushed in a sequence that risk should be mitigated. Handling them in a sequence is the proposed solution in the Sonarr issue. This is my current understanding of this.

Here is what I don't understand. Arrnounced does not perform concurrent release pushes. It was using requests which has no asyncio capabilities. As of release 0.7.2 Arrnounced moved to aiohttp which allows asyncio HTTP requests. To avoid making concurrent pushes a lock per backend was added.

So... why was the release in this reported issues grabbed three times? If someone has a verbose log to share, where this happens, I am curious to see it. Until then I'm closing this issue.

[jkaberg]

@weannounce it happends because when the same release is pushed while still not imported in Sonarr (from a previous push of the same release from a different tracker)

Example:

Tracker1 -> release X -> Arrnounced -> Sonarr/Radarr Tracker2 -> release X -> Arrnounced -> Sonarr/Radarr (notice release from tracker1 hasn't finished downloading/imported so Sonarr still accepts new pushes because we don't have an imported release) Sonarr -> import file from disk/torrent which happends to be release X from tracker1 -> Release OK -> Sonarr doesn't accept any more pushes for release X

My proposal is simple; If arrnounced already has pushed an release with title my.release.hdrip.1080p (we lookup this in arrnounced database) then don't push anymore of my.release.hdrip.1080p.

This issue has nothing to do with the requests or aiohttp libs.

Alas this is an "race condition" in Sonarr, and not in arrnounced. It just that Sonarr team haven't implemented a good fix for this (for several years.)