morpheus65535 / bazarr

Bazarr is a companion application to Sonarr and Radarr. It manages and downloads subtitles based on your requirements. You define your preferences by TV show or movie and Bazarr takes care of everything for you.
https://www.bazarr.media
GNU General Public License v3.0
2.88k stars 224 forks source link

WebUI seems to randomly crash? #2470

Closed PassiveLemon closed 6 months ago

PassiveLemon commented 6 months ago

Describe the bug After running for a while, the WebUI will become inaccessible, however, the container will still be running. If I restart the container, the UI will become accessible again. Nothing particularly of interest shows in the logs when the issue is present so it's hard to track.

To Reproduce Not sure how.

Software (please complete the following information):

PassiveLemon commented 6 months ago

Side note: Adjust the position of the activity popup because it covers up page navigation image

PassiveLemon commented 6 months ago

bazarrui.log Just caught it doing it again so here's the last 150 lines from Docker logs. It appears that nothing functions anymore once it happens. If there isn't enough evidence, I can send a longer log.

morpheus65535 commented 6 months ago

This logs only tell me that a subtitles downloaded from animetosho encoding wasn't guessed properly. I think we'll need a longer log...

PassiveLemon commented 6 months ago

_bazarr_logs.txt Might as well send the whole thing

anderson-oki commented 6 months ago

@PassiveLemon What do you mean the web ui becomes inaccessible? Do you get like refused to connect error from browser? Also do you still get logs while the WebUI is inaccessible?

I suspect that the node process died. But the backend still running?

PassiveLemon commented 6 months ago

I get Firefoxes "Hmm. We're having trouble finding that site." error, not sure exactly type of error it is lol. And no, I do not seem to get any more logs once it happens.

anderson-oki commented 6 months ago

Yeah i guess python3 process died for some reason 🤔

flensburg2 commented 6 months ago

Doing some maintenance today on all my bazarr setup, disk mappings etc. Third time now in an hour where UI crashes and cant reach backend. So there seem to be an issue for sure.

morpheus65535 commented 6 months ago

Just for testing purpose, what happen if you disable animetosho provider for a while?

anderson-oki commented 6 months ago

Perhaps @flensburg2 can also provide some logs?

morpheus65535 commented 6 months ago

Perhaps @flensburg2 can also provide some logs?

https://github.com/morpheus65535/bazarr/issues/2470#issuecomment-2076137237

flensburg2 commented 6 months ago

Just for testing purpose, what happen if you disable animetosho provider for a while?

Never had animetosho enabled.

anderson-oki commented 6 months ago

I saw PassiveLemon logs i was wondering if they have different behaviors and if they are both using animetosho?

morpheus65535 commented 6 months ago

I saw PassiveLemon logs i was wondering if they have different behaviors and if they are both using animetosho?

My bad, sorry. It's still early here ;)

morpheus65535 commented 6 months ago

Never had animetosho enabled.

This message was for @PassiveLemon .

@flensburg2 can you provide debug logs?

PassiveLemon commented 6 months ago

I removed the animetosho provider and AniDB integration and I have yet to see it happen. I'll give it overnight before I say with confidence though.

PassiveLemon commented 6 months ago

It's been way more than one night and no problems so far. I guess it can be reasonably assumed that the animetosho provider or AniDB integration is a culprit

morpheus65535 commented 6 months ago

@flensburg2 still waiting for your logs.

@PassiveLemon thanks for checking that.

@anderson-oki are you able to look into this?

anderson-oki commented 6 months ago

The animetosho encoding issue is a separated issue and i can for sure reproduce but doesnt break anything else. I will be looking to fix that but it seems unrelated?

I have been running the docker version with an uptime of 32 hours fully using anidb and animetosho and i cant find anything that can be causing it. I will keep looking into it but im not sure if i can spot the root cause soon without more details.

anderson-oki commented 6 months ago

@morpheus65535 can we ask to try the beta 26 since it fixes some exceptions? My guess in the worst case the exceptions are accumulating some garbage that kills it fixed on the beta 26. Or maybe Docker disk space for the cache?

morpheus65535 commented 6 months ago

@morpheus65535 can we ask to try the beta 26 since it fixes some exceptions? My guess in the worst case the exceptions are accumulating some garbage that kills it fixed on the beta 26. Or maybe Docker disk space for the cache?

Of course we can ask @PassiveLemon to test latest beta. Can you also provide docker-compose just in case we see something?

PassiveLemon commented 6 months ago

Heres my docker-compose:

bazarr:
  image: lscr.io/linuxserver/bazarr:development-version-v1.4.3-beta.24
  container_name: bazarr
  restart: always
  volumes:
    - /home/docker/Containers/Media/Bazarr/config/:/config/:rw
    - /home/HDD2TBEXT4/Media/:/data/:rw
  environment:
    TZ: "${TZ}"
    PUID: "${PUID}"
    PGID: "${PGID}"
  ports:
    - 55367:6767/tcp
  networks:
    torrent:
  deploy:
    resources:
      limits:
        cpus: "1"
        memory: 300M

I'm going to bump it to beta 26, renable animetosho/anidb and let it run over night and see if anything happens

anderson-oki commented 6 months ago

Thanks @PassiveLemon, please re-send the logs of the version 26 if the breaks again 🙇🏻 And tell us around what is the time that you noticed it stopped working.

morpheus65535 commented 6 months ago

Limiting Bazarr to 300M of memory is probably the root cause. Depending of what feature you use, it may require more memory than that. Subtitles synchronization could use over a gigabyte of memory depending of audio track format.

anderson-oki commented 6 months ago

I can reproduce the issue lowering the memory. The /lsiopy/bin/python3 -u /app/bazarr/bin/bazarr/main.py --no-update --config /config gets killed on kernel level by out of memory.

PassiveLemon commented 6 months ago

Hm i didn't even think of that. Usually if i have memory problems, in other containers, they just become much slower but I've never had crashes before. I'll increase it.

Edit: Just checked my instance and it's still running.

morpheus65535 commented 6 months ago

So everything seems to be good now. I'll close this issue but let me know if you experience it again.

PassiveLemon commented 6 months ago

Hmm well it's happening again. Memory limit is set to 1 gigabyte

anderson-oki commented 6 months ago

@PassiveLemon Let me ask a couple more questions and ask you to try something:

1 - Does it happen on a specific time every time? Like more specifically on the time it is set on the Settings > Scheduler?

2 - Can you try to open System > Tasks and run some tasks like Index All Movie , or maybe the other tasks as well.

Can you see if while doing the 2 it crashes? Or maybe monitor the memory usage while running the tasks?

PassiveLemon commented 6 months ago

Looking at where the logs end (4:02 AM), it seems like either the search for missing movie or series subtitles is at least partially responsible. I also never saw it go above 300 MB, although I can't confirm that for the entire night as I wasn't watching it then. The issue is that I ran it last night and it didn't crash. I'm running it again to confirm though.

Edit: Ran movie and series scan and did not have a problem.

anderson-oki commented 6 months ago

So it seems to die at 4AM, what schedulers do you have for 4AM? Can you reproduce manually running the Task?

PassiveLemon commented 6 months ago

The only thing that runs specifically at 4 AM is indexing but other stuff could have happened to run at 4 AM at that time too just from default intervals. I've tried running all of the tasks multiple times and have not been able to reproduce it. In fact, it hasn't crashed since I sent the last comment. This is so finicky

anderson-oki commented 6 months ago

Some people reported that the Automatic Subtitles Synchronization is causing some out of memory on their environment, perhaps because it consumes a lot of memory for bigger movies. I'm not sure if it is anyhow related tho.