Bazaar keeping HDDs spun up

[JHarding86]

Describe the bug When Bazaar is running, the hard drives that are associated with my media never spin down.

To Reproduce Steps to reproduce the behavior:

1. Connect Bazaar to Sonarr and Radarr
2. Watch your drives never spin down

Expected behavior If no new subtitles are found and written to a drive, after your set spin down time, hard drives should spin down.

Software (please complete the following information):

• Bazarr: 1.1.4
• Radarr version 4.3.2.6857
• Sonarr version 3.0.9.1549
• OS: Unraid 6.11.5

Additional context When I disable the Bazaar docker in Unraid, my drives spin down at the appropriate intervals

[morpheus65535]

Bazarr need to index your existing subtitles(embedded and external). If you want, you can tune some settings to reduce the impact on your system: https://wiki.bazarr.media/Additional-Configuration/Performance-Tuning/

[k29t59dh]

Same, also on unRAID. morpheus655535 said it's probably related to an un-cached hash calculation: https://github.com/morpheus65535/bazarr/issues/1900#issuecomment-1200446251

The best track for a solution may be waiting for (or contributing) a "give up after X amount of time" setting so it doesn't reexamine the same media for which subtitles are unavailable indefinitely.

[JHarding86]

Bazaar should not need to reindex the subtitles every time it goes to search for subtitles. Upon initial installation, it should search my media for what exists, cache that information, and then at predefined intervals go and seek out subtitles that have not been obtained. I don't use any other subtitle searching tool so it would be ok for Bazaar to assume that nothing has changed since the last time it made a search.

I know I am greatly simplifying this, but it seems that the simple explanation I have above should be the way the system works...

[morpheus65535]

@JHarding86 Bazarr is already caching the ffprobe results and, by default, won't access the file to re-index the embedded subtitles. When Bazarr download a subtitles, once it's been saved on disk, it re-index external subtitles. That's it, nothing more, nothing less.

[JHarding86]

It appears as though even when no new subtitles were downloaded, the disks are kept alive. Why would that be happening?

[k29t59dh]

ffprobe may be cached but it's accessing the files for some reason.

My setup (and probably JHardings) uses an SSD cache, so anything more recent than a couple of weeks is likely there where spinup's not an issue.

inotify shows older media (on the mechanical drives) randomly accessed, waking the drives. If I stop bazarr no random access, disks stay spun down.

[k29t59dh]

I should have been clearer in my initial post and my reply: I think the issue only affects media where bazaar can't find a requested subtitle.

Where it can it doesn't reexamine the media. Where it can't it re-scans periodically (waking the drives) forever and ever.

For libraries with 10s or 100s of items where no subtitle exists the effect is constant spin-ups.

As far as I can tell there's no way to disable search per item but even if there were, the initial fix would be time-consuming and it'd require constant monitoring as new media (for which subtitles are likely unavailable) is added.

[JHarding86]

@k29t59dh That would make sense, I have thousands of subtitles that Bazaar cannot find spanning most of the hard drives in my array.

In my mind, Bazaar just shouldn't even be looking at those disks once it has indexed all of the shows and determined if there are or are not subtitles. After that initial scan, it should rely on its index to determine if it should search for subtitles, and only if it finds one should it write the file to the designated location.

It sounds like Bazaar doesn't trust its own index...

[morpheus65535]

@k29t59dh I'll see if I can reproduce. Could be some obscure call to file system that I'm not aware of (we use dependencies and I haven't examine every line of code).

@JHarding86 I've read your point. It's not that Bazarr doesn't trust it own index, it just doesn't trust that nothing can happens on disk that it didn't request. Lot of user run in many different use case. Yours isn't the only one.

[JHarding86]

@morpheus65535 I'm interested in the other use cases, I know mine isn't the only one. Maybe they could help solve some of my problems and help alleviate this problem that I am having?

[morpheus65535]

I'm interested in the other use cases, I know mine isn't the only one. Maybe they could help solve some of my problems and help alleviate this problem that I am having?

External script, migration to cloud, embedding subtitles in media container, etc.

[k29t59dh]

It's not that Bazarr doesn't trust it own index, it just doesn't trust that nothing can happens on disk that it didn't request.

Although you could make the simplifying assumption that nothing happens on disk without affecting modification times of files or directories.

Could be some obscure call to file system that I'm not aware of (we use dependencies and I haven't examine every line of code).

Maybe I misunderstood but I thought you'd tracked it to the hash calculation:

Probably related to the hash calculation that we don't cache: https://github.com/morpheus65535/bazarr/blob/development/libs/subliminal/utils.py

[vitiko98]

We need to know about your setup (providers, general settings, etc). If possible, send your config.ini (obfuscating sensitive data, of course).

[morpheus65535]

Maybe I misunderstood but I thought you'd tracked it to the hash calculation:

That's one possibility. I'll let @vitiko98 investigate. I'm busy ATM with some other issues but your in good hands!

[k29t59dh]

We need to know about your setup (providers, general settings, etc). If possible, send your config.ini (obfuscating sensitive data, of course).

Sure, thanks! Here's my config.ini. And I'm happy to provide anything else that might be relevant, I appreciate your help. config.ini.zip

[vitiko98]

Are you still having this problem with Bazarr newer versions? Please confirm.

[k29t59dh]

Yes, unfortunately.

I just ran a test:

1. Stopped all containers that'd access the disks except bazarr
2. Spun down disks
3. Started inotify

In System -> Tasks I ran: Search for wanted Movies Subtitles, which woke the disks, and inotify reported access to the corresponding video files. Search for wanted Series Subtitles, which woke the disks, and inotify reported access to the corresponding video files.

I wanted to test Upgrade previously downloaded Subtitles as well but couldn't – I don't have any subs needing upgrades.

[morpheus65535]

@k29t59dh my guess is that's only the video file hashing that wake the drive (we don't cache it).

If you use only a provider that doesn't use video file hashing (like podnapisi), can you confirm if your drive stay asleep?

[k29t59dh]

If you use only a provider that doesn't use video file hashing (like podnapisi), can you confirm if your drive stay asleep?

Yes, I just confirmed this. No disk access with podnapisi as the only provider.

[morpheus65535]

@k29t59dh I've just added a new settings to prevent file hashing from happening so it won't wake-up your drive anymore for any providers. You'll find it in upcoming beta under Settings-->Subtitles-->Skip video file hash calculation.

[morpheus65535]

@JHarding86 I would appreciate to get your feedback once you enable the skip hashing settings. Can you confirm your drive stay asleep? Can I close this issue?

[JHarding86]

I will give this a try in the coming days.

[Mushin]

Would it be possible to cache the file hashes also?

[morpheus65535]

Would it be possible to cache the file hashes also?

Possible, maybe but it would require much more work than that. I see that as a feature request, not an issue. Feel free to create one and we'll see over time how it get upvoted.

[k29t59dh]

I'll also test it out, thanks for your work.

Re: caching hashes, Is the issue that it doesn't fit well with the current design or is it just a matter of replicating existing cache logic for hashes?

If it's the latter, I'll take a stab at it in the next few weeks.

[morpheus65535]

@k29t59dh it would require to cache and invalidate properly (new column in database tables) but also find a way to propagate this to this function in order to reuse the cached value. Nothing in libs should access database directly. Values must be passed as arguments.

[Mushin]

Would it be possible to cache the file hashes also?

Possible, maybe but it would require much more work than that. I see that as a feature request, not an issue. Feel free to create one and we'll see over time how it get upvoted.

Thanks - feature request submitted.

[JHarding86]

My unraid array is going through some tough spots. This is still on my list of things to test.

[JHarding86]

What version of Bazaar does this fix exist in? and in which settings section does the option reside in?

[morpheus65535]

What version of Bazaar does this fix exist in? and in which settings section does the option reside in?

1.2.1-beta

Settings-->Subtitles-->Skip video file hash calculation

[JHarding86]

It appears as though this is fixed. My disks do not spin up when I perform a scan. Thank you for spending the time to add an option for my use case. Feel free to close this issue.

[JHarding86]

This started happening again :(

[JHarding86]

@morpheus65535 Here is the version information that I am running that is keeping my array spun up again:

[morpheus65535]

@JHarding86 the only thing that changed is that Bazarr does check if a subtitles files still exists on disk before trying to upgrade it. I can't remove easily.

[JHarding86]

@morpheus65535 That's unfortunate. My arr stack (radarr, sonarr, overseerr, unpackerr, readarr, prowlarr) will just be short one. It's curious that Radarr and Sonarr software work with the same number of files and upgrading those files without keeping my disks spun up continually. Maybe some exploration into how those other pieces of software work could help in solving this issue.

[morpheus65535]

@JHarding86 we come to a point where you can't make an omelette without breaking eggs. If Bazarr doesn't make it for you, I'll be sorry to see you go but I can't make it so it never wake a disk.