search

ipsingh06 / seedsync

Sync your seedbox. Fast. And more.
https://ipsingh06.github.io/seedsync/
Apache License 2.0
312 stars 43 forks source link

[Solved] Error parsing lftp job status [v0.8.6] #106

Open casperse opened 3 years ago

casperse commented 3 years ago

FYI: NOT SOLVED FOR THE DOCKER - MOVED SEEDSYNC TO VM!

Running the latest docker on unraid Extra parameters: --cpus=2 --memory=3G --restart unless-stopped --user 99:100

The docker stopped before - so I added the : --restart unless-stopped I can see it connects and start downloading (Great speed by the way!) But then it stops.

I am using the SSH user and password. And I have settings:

[General] debug = True verbose = False

[Lftp] remote_address = domain.seedhost remote_username = username remote_password = password remote_port = 22 remote_path = /home/username/downloads local_path = /downloads remote_path_to_scan_script = /tmp use_ssh_key = False num_max_parallel_downloads = 2 num_max_parallel_files_per_download = 4 num_max_connections_per_root_file = 4 num_max_connections_per_dir_file = 4 num_max_total_connections = 16 use_temp_file = True

[Controller] interval_ms_remote_scan = 30000 interval_ms_local_scan = 10000 interval_ms_downloading_scan = 1000 extract_path = /tmp use_local_path_as_extract_path = True

[Web] port = 8800

[AutoQueue] enabled = True patterns_only = True auto_extract = False

`lftp_statuses = self.lftp.status() File "/app/python/lftp/lftp.py", line 322, in status statuses = self.job_status_parser.parse(out) File "/app/python/lftp/job_status_parser.py", line 94, in parse raise LftpJobStatusParserError("Error parsing lftp job status")

lftp.job_status_parser.LftpJobStatusParserError: Error parsing lftp job status

2021-09-30 11:46:11,459 - INFO - seedsync.Controller (MainProcess/ControllerJob) - Exited controller

2021-09-30 11:46:34,495 - INFO - web_access (MainProcess/worker 1) - 192.168.0.14 - - [30/Sep/2021:11:46:34 +0200] "GET /server/config/get HTTP/1.1" 200 801 "http://192.168.0.6:8800/autoqueue" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:46:34,496 - INFO - web_access (MainProcess/worker 2) - 192.168.0.14 - - [30/Sep/2021:11:46:34 +0200] "GET /server/autoqueue/get HTTP/1.1" 200 185 "http://192.168.0.6:8800/autoqueue" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:46:53,422 - INFO - web_access (MainProcess/worker 3) - 192.168.0.14 - - [30/Sep/2021:11:46:53 +0200] "GET /settings HTTP/1.1" 304 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:47:46,404 - INFO - web_access (MainProcess/worker 4) - 192.168.0.14 - - [30/Sep/2021:11:47:41 +0200] "GET /server/stream HTTP/1.1" 200 - "http://192.168.0.6:8800/settings" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:47:46,441 - INFO - web_access (MainProcess/worker 5) - 192.168.0.14 - - [30/Sep/2021:11:47:46 +0200] "GET /server/config/get HTTP/1.1" 200 801 "http://192.168.0.6:8800/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:54:39,724 - INFO - seedsync (MainProcess/MainThread) - Caught signal SIGTERM 2021-09-30 11:54:39,724 - INFO - seedsync (MainProcess/MainThread) - Exiting Seedsync 2021-09-30 11:54:50,887 - INFO - seedsync (MainProcess/MainThread) - Debug mode is disabled. 2021-09-30 11:54:50,887 - INFO - seedsync (MainProcess/MainThread) - Starting SeedSync 2021-09-30 11:54:50,890 - INFO - seedsync (MainProcess/MainThread) - Platform: x86_64 2021-09-30 11:54:51,398 - INFO - seedsync.WebApp (MainProcess/MainThread) - Html path set to: /app/html 2021-09-30 11:54:51,835 - INFO - RemoteScanner.RemoteScanner (RemoteScanner/RemoteScanner) - Skipping remote scanfs installation: already installed 2021-09-30 11:54:54,531 - INFO - web_access (MainProcess/worker 0) - 192.168.0.14 - - [30/Sep/2021:11:54:54 +0200] "GET /logs HTTP/1.1" 200 764 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:54:54,893 - INFO - web_access (MainProcess/worker 1) - 192.168.0.14 - - [30/Sep/2021:11:54:54 +0200] "GET /server/stream HTTP/1.1" 200 - "http://192.168.0.6:8800/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:54:55,190 - INFO - web_access (MainProcess/worker 2) - 192.168.0.14 - - [30/Sep/2021:11:54:55 +0200] "GET /assets/favicon.png HTTP/1.1" 200 1000 "http://192.168.0.6:8800/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:55:01,943 - INFO - seedsync.AutoQueue (MainProcess/ControllerJob) - Auto queueing '#TVSHOWS-C' for pattern '#TVSHOWS-C' 2021-09-30 11:55:01,944 - INFO - seedsync.AutoQueue (MainProcess/ControllerJob) - Auto queueing '#MUSIC-SABnzbd' for pattern '#MUSIC-SABnzbd' 2021-09-30 11:55:02,444 - INFO - seedsync.Controller (MainProcess/ControllerJob) - Received command Action.QUEUE for file #TVSHOWS-C

2021-09-30 11:55:02,496 - INFO - seedsync.Controller (MainProcess/ControllerJob) - Received command Action.QUEUE for file #MUSIC-SABnzbd

2021-09-30 11:55:05,533 - INFO - web_access (MainProcess/worker 4) - 192.168.0.14 - - [30/Sep/2021:11:55:05 +0200] "GET /server/autoqueue/get HTTP/1.1" 200 185 "http://192.168.0.6:8800/autoqueue" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:55:05,565 - INFO - web_access (MainProcess/worker 5) - 192.168.0.14 - - [30/Sep/2021:11:55:05 +0200] "GET /server/config/get HTTP/1.1" 200 801 "http://192.168.0.6:8800/autoqueue" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" 2021-09-30 11:55:12,318 - ERROR - seedsync.Controller.Lftp.LftpJobStatusParser (MainProcess/ControllerJob) - LftpJobStateParser error: Unrecognized line 'pget: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Loose.(Expanded.Edition).(2021).Mp3.320kbps.[PMEDIA].⭐️-xpost/10..Say.It.Right.mp3: Permission denied'

2021-09-30 11:55:12,323 - ERROR - seedsync.Controller.Lftp.LftpJobStatusParser (MainProcess/ControllerJob) - Status:

pget: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Loose.(Expanded.Edition).(2021).Mp3.320kbps.[PMEDIA].⭐️-xpost/10..Say.It.Right.mp3: Permission denied 2021-09-30 11:55:12,328 - WARNING - seedsync.Controller.Lftp (MainProcess/ControllerJob) - Ignoring status error (count=1)

2021-09-30 11:55:19,193 - ERROR - seedsync.Controller.Lftp.LftpJobStatusParser (MainProcess/ControllerJob) - LftpJobStateParser error: Unrecognized line 'pget: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Loose.(Expanded.Edition).(2021).Mp3.320kbps.[PMEDIA].⭐️-xpost/16..Te.Busque.(Spanish.Version).mp3: Permission denied'

2021-09-30 11:55:19,193 - ERROR - seedsync.Controller.Lftp.LftpJobStatusParser (MainProcess/ControllerJob) - Status:

pget: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Loose.(Expanded.Edition).(2021).Mp3.320kbps.[PMEDIA].⭐️-xpost/16..Te.Busque.(Spanish.Version).mp3: Permission denied

2021-09-30 11:55:19,194 - WARNING - seedsync.Controller.Lftp (MainProcess/ControllerJob) - Ignoring status error (count=2)

2021-09-30 11:55:25,173 - ERROR - seedsync.Controller.Lftp.LftpJobStatusParser (MainProcess/ControllerJob) - LftpJobStateParser error: Unrecognized line 'mirror: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Mi.Plan.(2009).Flac-xpost/[PRiVATE]-[WtFnZb]-[about_album]-[13+15] - "" yEnc 851 (1+1): Permission denied'

2021-09-30 11:55:25,173 - ERROR - seedsync.Controller.Lftp.LftpJobStatusParser (MainProcess/ControllerJob) - Status:

mirror: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Mi.Plan.(2009).Flac-xpost/[PRiVATE]-[WtFnZb]-[about_album]-[13+15] - "" yEnc 851 (1+1): Permission denied 2021-09-30 11:55:25,173 - ERROR - seedsync.ControllerJob (MainProcess/ControllerJob) - Caught exception in job ControllerJob

Traceback (most recent call last): File "/app/python/lftp/job_status_parser.py", line 89, in parse statuses += self.__parse_queue(lines) File "/app/python/lftp/job_status_parser.py", line 482, in __parse_queue raise ValueError("Unrecognized line '{}'".format(lines[0])) ValueError: Unrecognized line 'mirror: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Mi.Plan.(2009).Flac-xpost/[PRiVATE]-[WtFnZb]-[about_album]-[13+15] - "" yEnc 851 (1+1): Permission denied'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/app/python/common/job.py", line 44, in run self.execute() File "/app/python/controller/controller_job.py", line 29, in execute

self.__controller.process() File "/app/python/controller/controller.py", line 202, in process

self.__update_model() File "/app/python/controller/controller.py", line 295, in __update_model

lftp_statuses = self.lftp.status() File "/app/python/lftp/lftp.py", line 322, in status statuses = self.job_status_parser.parse(out) File "/app/python/lftp/job_status_parser.py", line 94, in parse raise LftpJobStatusParserError("Error parsing lftp job status")

lftp.job_status_parser.LftpJobStatusParserError: Error parsing lftp job status

2021-09-30 11:55:25,278 - ERROR - seedsync (MainProcess/MainThread) - Caught exception

Traceback (most recent call last): File "/app/python/lftp/job_status_parser.py", line 89, in parse statuses += self.__parse_queue(lines) File "/app/python/lftp/job_status_parser.py", line 482, in __parse_queue raise ValueError("Unrecognized line '{}'".format(lines[0])) ValueError: Unrecognized line 'mirror: /downloads/#MUSIC-SABnzbd/Nelly.Furtado.-.Mi.Plan.(2009).Flac-xpost/[PRiVATE]-[WtFnZb]-[about_album]-[13+15] - "" yEnc 851 (1+1): Permission denied'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/app/python/seedsync.py", line 164, in run controller_job.propagate_exception() File "/app/python/common/job.py", line 77, in propagate_exception raise exc_info[1].with_traceback(exc_info[2]) File "/app/python/common/job.py", line 44, in run self.execute() File "/app/python/controller/controller_job.py", line 29, in execute

self.__controller.process() File "/app/python/controller/controller.py", line 202, in process

self.__update_model() File "/app/python/controller/controller.py", line 295, in __update_model

lftp_statuses = self.lftp.status() File "/app/python/lftp/lftp.py", line 322, in status statuses = self.job_status_parser.parse(out) File "/app/python/lftp/job_status_parser.py", line 94, in parse raise LftpJobStatusParserError("Error parsing lftp job status")

lftp.job_status_parser.LftpJobStatusParserError: Error parsing lftp job status

2021-09controller-30 11:55:28,491 - INFO - seedsync.Controller (MainProcess/ControllerJob) - Exited `

quadcom commented 3 years ago

What are your connection settings? Can you post a pic here (not your server address and password stuff, the thread counts and stuff on the right side)

casperse commented 3 years ago

I posted The configuration file at The top (All info is There except username and password (-(

quadcom commented 3 years ago

I posted The configuration file at The top (All info is There except username and password (-(

Sorry, missed that

I have tested using different configs but I have learned that anything other than 1 file at a time with 10 connections causes LFTP to bug out.

Here are the settings that I've settled with.

image

casperse commented 3 years ago

Thanks - I just used the "default config" from the docker install (I thought these would be the safe ones) but I have changed the settings now to match yours! Its running but the UI is very unresponsive - maybe I have to many files to sync/DL? If I move away from the UI and go back I have to wait for it to load (Long time) again it seems like its utillizing all power to do the transfers... RAM is just above 2G usage and CPU is really low 15%

casperse commented 3 years ago

SOLVED: I found some comments about Ubuntu and version 20 problems? So I took a chance and took my notes and did it all over again - but this time on Ubuntu 18.04.3 and here everything works! I have the two VM's running side by side and so far I have not ben able to find any configuration or access differences on folder or paths?

One last Q. its now running very fast and without any slow UI or freeze on downloads How many cores and Memory have you assigned for your VM (What is enough?)

quadcom commented 3 years ago

IMO, it doesn't need all that much as the functions are mostly throughput dependant.

On the other hand, I have gone overboard mainly because I have 12 cores in the CPU and 4 VM's of which are a MariaDB, a pihole and a Plex server so mostly low CPU applications (aside from Plex).

4 cores, 4GBon a 20GB HDD. It maxes out the RAM but never crashes or causes any issues with other VM's

Congrats on getting it going. It's a great feeling when you figure shit out and get the lab running.

Now you need to get filebot running.......

casperse commented 3 years ago

Actually I haven't setup the : "Use password-less key-based authentication" does it make any difference? And can you see the Log section? seem to take forever to load

Update: I also had to set 4G and 4 Cores in order for it to run okay I can see that it syncs all the existing files on my existing drive share again, so there must be a DB file where Seedsync safes all transactions to? How does it manage this over time?