search

linuxserver / docker-jackett

GNU General Public License v3.0
395 stars 95 forks source link

Jackett starts, but after a day or two crashes [Restarting fixes it for sometime] #62

Closed samip5 closed 5 years ago

samip5 commented 5 years ago

Host OS is: Ubuntu Server 18.04 x64 My docker-compose file snippet:

jackett:
    image: linuxserver/jackett
    container_name: jackett
    volumes:
     - /etc/localtime:/etc/localtime:ro
     - /opt/docker/appdata/jackett:/config
    ports:
     - 27027:9117
    expose:
     - "9117"
    environment:
     - PUID=111
     - GUID=113
     - TZ=Europe/Helsinki

Docker log can be found at: https://p.kapsi.fi/?2945f2222bde8140#fNVjksAEliw9hxaV3+7bGq77XIF0wLuDft1OUyTUTno=

This is fixed by just restarting the container usually, but I have no idea why it's happening to begin with.

Output of docker inspect -f '{{ index .Config.Labels "build_version" }}' jackett:

Linuxserver.io version:- 343 Build-date:- December-01-2018-19:36:28-UTC

Output of docker inspect -f '{{ index .Config.Labels "build_version" }}' linuxserver/jackett:

Linuxserver.io version:- 343 Build-date:- December-01-2018-19:36:28-UTC

Related issue: Jackett/Jackett#4283

j0nnymoe commented 5 years ago

password for privatebin?

j0nnymoe commented 5 years ago

Please try using a PUID/PGID that isn't a system user. Needs to be a normal user you've created.

samip5 commented 5 years ago

Please try using a PUID/PGID that isn't a system user. Needs to be a normal user you've created.

I'm using a system user which I have myself created, this is also a related problem to a issue in Jackett repo. (Jackett/Jackett#4279)

hadim commented 5 years ago

I am having the same issue. @samip5 have you find a fix? Do you think it's related to Docker or to Jackett itself?

samip5 commented 5 years ago

@hadim Unfortunately, I have yet to find a solution. I think it's partly relates to docker AND Jackett itself. I have been unable to debug it further due to lack of know how in regards to getting trace outputs from a Docker container.

ghost commented 5 years ago

@samip5 @hadim Are either of you using a proxy in Jackett e.g http or SOCKS Are either of you using the MejorTorrent indexer?

hadim commented 5 years ago

No.

samip5 commented 5 years ago

@samip5 @hadim Are either of you using a proxy in Jackett e.g http or SOCKS Are either of you using the MejorTorrent indexer?

No, I don't have any proxies in Jackett, but Yes, I am using that indexer.

ghost commented 5 years ago

@samip5 Can you try removing it to see if that is the cause

hadim commented 5 years ago

I monitored the RAM usage on my NAS during 24h and found that Jackett RAM usage increases more than my other containers:

plot

I am not sure it's related but it could help to debug this.

Note that Jackett is still working after 24h. I'll keep you posted here.

hadim commented 5 years ago

Also when I first load the web application after sometime (24h here) it takes a very long time before Jackett load the page. aybe it's related to the RAM usage increase. Here are the log:

2019-01-16 21:28:32.8035 Debug Connection id "0HLJRB1P4LDIE" disconnecting. 
2019-01-16 21:28:32.8035 Debug Connection id "0HLJRB1P4LDIB" disconnecting. 
2019-01-16 21:28:32.8039 Debug Connection id "0HLJRB1P4LDIE" sending FIN because: "The Socket transport's send loop completed gracefully." 
2019-01-16 21:28:32.8039 Debug Connection id "0HLJRB1P4LDIB" sending FIN because: "The Socket transport's send loop completed gracefully." 
2019-01-16 21:28:32.8048 Debug Connection id "0HLJRB1P4LDIE" stopped. 
2019-01-16 21:28:32.8048 Debug Connection id "0HLJRB1P4LDIB" stopped. 
2019-01-16 21:28:33.8036 Debug Connection id "0HLJRB1P4LDID" disconnecting. 
2019-01-16 21:28:33.8036 Debug Connection id "0HLJRB1P4LDIF" disconnecting. 
2019-01-16 21:28:33.8036 Debug Connection id "0HLJRB1P4LDID" sending FIN because: "The Socket transport's send loop completed gracefully." 
2019-01-16 21:28:33.8048 Debug Connection id "0HLJRB1P4LDIF" sending FIN because: "The Socket transport's send loop completed gracefully." 
2019-01-16 21:28:33.8048 Debug Connection id "0HLJRB1P4LDID" stopped. 
2019-01-16 21:28:33.8056 Debug Connection id "0HLJRB1P4LDIF" stopped. 
2019-01-16 21:28:39.8039 Debug Connection id "0HLJRB1P4LDIG" disconnecting. 
2019-01-16 21:28:39.8039 Debug Connection id "0HLJRB1P4LDIC" disconnecting. 
2019-01-16 21:28:39.8039 Debug Connection id "0HLJRB1P4LDIG" sending FIN because: "The Socket transport's send loop completed gracefully." 
2019-01-16 21:28:39.8050 Debug Connection id "0HLJRB1P4LDIC" sending FIN because: "The Socket transport's send loop completed gracefully." 
2019-01-16 21:28:39.8050 Debug Connection id "0HLJRB1P4LDIG" stopped. 
2019-01-16 21:28:39.8059 Debug Connection id "0HLJRB1P4LDIC" stopped. 
2019-01-16 21:37:23.7174 Debug Connection id "0HLJRB1P4LDIH" started. 
2019-01-16 21:37:23.7185 Info Request starting HTTP/1.1 GET http://nas.hadim.fr:9117/UI/Dashboard   
2019-01-16 21:37:23.7189 Debug Request is continuing in applying rules. Current url is http://nas.hadim.fr:9117/UI/Dashboard 
2019-01-16 21:37:23.7189 Debug Request is continuing in applying rules. Current url is http://nas.hadim.fr:9117/UI/Dashboard 
2019-01-16 21:37:23.7199 Debug Request is continuing in applying rules. Current url is http://nas.hadim.fr:9117/UI/Dashboard 
2019-01-16 21:37:23.7199 Debug The request path /UI/Dashboard does not match a supported file type 
2019-01-16 21:37:23.7213 Debug AuthenticationScheme: Cookies was successfully authenticated. 
2019-01-16 21:37:23.7218 Debug Request successfully matched the route with name '(null)' and template 'UI/Dashboard' 
2019-01-16 21:37:23.7218 Debug Action 'Jackett.Server.Controllers.WebUIController.Dashboard (JackettConsole)' with id 'b81cfb4d-6365-4f23-9727-4eeb11694c33' did not match the constraint 'Microsoft.AspNetCore.Mvc.Internal.HttpMethodActionConstraint' 
2019-01-16 21:37:23.7227 Info Route matched with {action = "Dashboard", controller = "WebUI"}. Executing action Jackett.Server.Controllers.WebUIController.Dashboard (JackettConsole)
hadim commented 5 years ago

Crossposting in case it's useful: https://github.com/Jackett/Jackett/issues/2405

hadim commented 5 years ago

If someone wants to reproduce it here are the two scripts I used:

#!/usr/bin/env bash

# Monitor memory usage of Docker containers
# Execute at 21h49.
# Log every 10 minutes

LOG_FILE="docker-logs.txt"

while true;
do
    docker stats --format "{{.Name}}, {{.MemUsage}}" --no-stream >> $LOG_FILE
    echo "-" >> $LOG_FILE
    sleep 10m
done

And the python script to generate the plot:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

log_path = "docker-logs.txt"

with open(log_path) as f:
    raw_data = f.readlines()

nrows = 5
n = len(raw_data) // nrows

data = []
for i in range(n):
    start = i * nrows
    end = start + nrows - 1
    d = raw_data[start:end]

    datum = {}
    datum['i'] = i
    for line in d:
        name, stats = line.strip().split(',')
        stats = float(stats.split('/')[0].strip()[:-3])
        datum[name] = stats
    data.append(datum)

data = pd.DataFrame(data)
data['time (hour)'] = data['i'] * 10 / 60

ax = data.drop(columns='i').set_index('time (hour)').plot()
ax.set_ylabel('RAM Usage (MiB)')
ax.figure.savefig('plot.png')
hadim commented 5 years ago

Today I noticed Jackett stopped working so I looked at the memory usage:

plot

After increasing to a certain limit (1GiB here), Jackett memory usage drops suddenly, I guess this is where Jackett stops working.

Do you know how I could try another Mono version within this Docker image?

samip5 commented 5 years ago

@hadim To my understanding, you would need to build the docker image yourself, by downloading it's source from the repo, and modifying the DockerFile.

hadim commented 5 years ago

I don't see how I can specify the Mono version within this image since it depends to lsiobase/mono:xenial which I think define the Mono version. It would be nice to be able to test the Mono version here.

ghost commented 5 years ago

@hadim I wouldn't waste your time trying different Mono versions. The issue has been going on for years and never happens on Windows, only Mono. Google 'Radarr Memory' or 'Sonarr memory', there is hundreds of posts just on one issue e.g https://github.com/Radarr/Radarr/issues/1580

In the next couple of months, Jackett will use .NET Core and hence not require Mono, for now I'd just restart it automatically before it becomes a problem

bruvv commented 5 years ago

@hadim could you help with that script? Getting the following error: 

File "/stats/genstat.py", line 24, in <module>
 name, stats = line.strip().split(',')
ValueError: not enough values to unpack (expected 2, got 1)

This is the stats file

-
hassio, 150.9MiB / 3.855GiB
Jacket, 229.9MiB / 3.855GiB
Radarr-nodebug, 113.8MiB / 3.855GiB
Sonarr-nodebug, 134.3MiB / 3.855GiB
Ombii, 189.6MiB / 3.855GiB
watchtower, 11.5MiB / 50MiB
-
hadim commented 5 years ago

I don't know what's going on. I use Python 3.

bruvv commented 5 years ago

I don't know what's going on. I use Python 3.

I use 3.7.2 ;) so that isnt the issue

hadim commented 5 years ago

@flightlevel is there is an alpha version of Jackett using NET Core?

ghost commented 5 years ago

@hadim There is, everything should work except the automatic updater start with ./jackett .NET Core builds start with Experimental in the file name, available here https://ci.appveyor.com/project/Jackett/jackett/build/job/kqy30te6o5vvos1i/artifacts

sparklyballs commented 5 years ago

@flightlevel , you should absolutely NOT use automatic updating anyways.

causes the app to get stuck in a respawn loop, the init system we use is not designed for apps updating themselves. and if it were run without the init using a CMD or ENTRYPOINT, it would stop the docker dead if the app were to update itself.

basically, don't do it.

hadim commented 5 years ago

I just started your NET Core version. Let's see how the memory usage is in 1 or 2 days:

jackett         | 01-26 14:10:16 Info Starting Jackett v0.10.672.0 
jackett         | 01-26 14:10:21 Info Environment version: 4.0.30319.42000 (/app/Jackett/) 
jackett         | 01-26 14:10:21 Info OS version: Unix 4.2.8.0 (64bit OS) (64bit process) 
jackett         | 01-26 14:10:21 Info Jackett variant: CoreLinuxAmd64 
jackett         | 01-26 14:10:21 Info ThreadPool MaxThreads: 32767 workerThreads, 1000 completionPortThreads 
jackett         | 01-26 14:10:21 Info App config/log directory: /config/Jackett 
jackett         | 01-26 14:10:21 Info issue: Ubuntu 16.04.5 LTS \n \l 
jackett         | 01-26 14:10:21 Info Using HTTP Client: HttpWebClientNetCore 
jackett         | 01-26 14:10:24 Info Loading Cardigann definitions from: /config/cardigann/definitions/, /etc/xdg/cardigan/definitions/, /app/Jackett/Definitions

For those who would like to test the new NET Core version using this docker image of Jackette, here is the diff to apply to this repo: 

diff --git a/Dockerfile b/Dockerfile
index 663f6f0..9d6f205 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -17,6 +17,8 @@ RUN \
  apt-get update && \
  apt-get install -y \
        wget \
+       libunwind8 \
+       icu-devtools \
        jq && \
  echo "**** install jackett ****" && \
  mkdir -p \
@@ -27,6 +29,7 @@ RUN \
  fi && \
  jackett_url=$(curl -s https://api.github.com/repos/Jackett/Jackett/releases/tags/"${JACKETT_RELEASE}" \
        |jq -r '.assets[].browser_download_url' |grep Mono) && \
+ jackett_url="https://ci.appveyor.com/api/buildjobs/dabl7modl8nh74n9/artifacts/Experimental.Jackett.Binaries.LinuxAMD64
  curl -o \
  /tmp/jacket.tar.gz -L \
        ${jackett_url} && \
diff --git a/root/etc/services.d/jackett/run b/root/etc/services.d/jackett/run
index c60ce2a..a4a01aa 100644
--- a/root/etc/services.d/jackett/run
+++ b/root/etc/services.d/jackett/run
@@ -1,4 +1,4 @@
 #!/usr/bin/with-contenv bash

 exec \
-       s6-setuidgid abc mono /app/Jackett/JackettConsole.exe --NoUpdates "${RUN_OPTS}"
+       s6-setuidgid abc /app/Jackett/jackett --NoUpdates "${RUN_OPTS}"

And then you start it with docker-compose for example:

version: '3'
services:
  jackett:
    container_name: jackett
    #image: linuxserver/jackett
    build: docker-jackett
    volumes:
      - /share/Multimedia/Configurations/Jackett:/config
    environment:
      - PGID=100
      - PUID=1001
      - TZ=EST
      - RUN_OPTS="-t -l"
    ports:
      - "9117:9117"
    restart: always
hadim commented 5 years ago

So far so good.

plot

binhex commented 5 years ago

sharing the love here, LSIO dev's - i have stabilised the jackett docker image i have created by hard setting to a particular version of mono, this has (for me at least) stabilised ram usage, allowing jackett to run over a week with no significant movement in ram usage over this period, the version of mono ive used is '5.14.0.177' (shamelessly stolen from one of the issues on jackett repo).

any questions just ping me guys.

bruvv commented 5 years ago

how did you do that @binhex ;) ?

binhex commented 5 years ago

how did you do that @binhex ;) ?

i dont mind detailing it, but as im using a completely different base OS its not really relevant to the LSIO jackett image, you can see my code here if you want to see how to achieve it for Arch Linux:-

https://github.com/binhex/arch-jackett/blob/master/build/root/install.sh

sparklyballs commented 5 years ago

@d1slact0r there is a branch here that uses a snapshot of mono 5.14

https://github.com/linuxserver/docker-jackett/tree/fix_mono_5.14-snapshot

it's a pr , but i'm not sure how to pull the testing version from our ci build, but you can build it locally from that branch.....

sparklyballs commented 5 years ago

thanks @binhex for the headsup

aptalca commented 5 years ago

The PR'ed build is right here: lspipepr/jackett

Please test and let us know

evoL commented 5 years ago

As a further heads-up: other Mono-based software have also similarly-looking problems with HTTP requests timeouting. There's an issue for Sonarr and I've observed it also with Radarr. A downgrade of those images to be based on Mono 5.12 (lsiobase/mono:75) seemed to help so far.

Thanks for the Mono 5.14 image! I've just switched my Jackett container to use lspipepr/jackett and will report if something breaks.

evoL commented 5 years ago

@aptalca Reporting that after 7 days on lspipepr/jackett, everything works like a charm! No restarts needed.

aptalca commented 5 years ago

@evoL excellent, thanks!! We'll go ahead and rebase the image then

Can you comment on sonarr/radarr? Do they work fine with 5.14 as well? If so, we'll rebase all of those mono apps to use 5.14

I don't use jackett, but use sonarr and radarr. Unfortunately (fortunately for me I guess) I never had such issues so I can't test.

evoL commented 5 years ago

I'm running Sonarr and Radarr on Mono 5.12 (since that was the version somebody previously reported to be working). I can confirm that they work properly for 7 days as well.

I believe it's worth rebasing them on 5.14 as well.

aptalca commented 5 years ago

yeah, but we're not going to rebase them until folks who were previously having issues confirm that they don't have issues on 5.14

thelamer commented 5 years ago

@evoL We have images for radarr and sonarr also, if you want to be involved in testing: (both using mono 5.14 stable)

lspipepr/sonarr:2.0.0.5301-pkg-f35751c9-pr-99 lspipepr/radarr:v0.2.0.1293-pkg-b6ec9679-pr-36

evoL commented 5 years ago

@thelamer Thanks, will test and report after a week or so :+1:

kiwijunglist commented 5 years ago

I have an issue of jackett crashing periodically and comes back to life when i restart the container.

To test the fix do i just comment out image and use build, then add run options?

eg.

    #image: linuxserver/jackett
    build: docker-jackett
    environment:
      - RUN_OPTS="-t -l"

Edit: I see we are supposed to try this one here - lspipepr/jackett

evoL commented 5 years ago

@aptalca @thelamer I'm happy to report that after two weeks on Mono 5.14 using the images mentioned in here and here, Jackett, Radarr and Sonarr work flawlessly. Thanks!

aptalca commented 5 years ago

Great, thanks for the report

IronTooch commented 5 years ago

For anyone getting here from the Google for a problem with 

 Fatal Unable to start Kestrel. System.BadImageFormatException: Could not load file or assembly 'System.Security.Cryptography.X509Certific
ates, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The module was expected to contain an assembly manifest.
May  2 07:20:19 casino jackett[1063]: File name: 'System.Security.Cryptography.X509Certificates, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
May  2 07:20:19 casino jackett[1063]:    at Microsoft.AspNetCore.Server.Kestrel.KestrelConfigurationLoader.Load()
May  2 07:20:19 casino jackett[1063]:    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.ValidateOptions()
May  2 07:20:19 casino jackett[1063]:    at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
May  2 07:20:19 casino jackett[1063]: 05-02 07:20:19 Fatal Unable to start Kestrel. System.BadImageFormatException: Could not load file or assembly 'System.Security.Cryptography.X509Certific
ates, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The module was expected to contain an assembly manifest.

The answer is to make the .config and Jackett path in your user's home directory.