search

arabcoders / watchstate

Self-hosted service to sync your plex, jellyfin and emby play state. without relying on 3rd-party external services.
MIT License
411 stars 7 forks source link

Error parsing plex tv library - Malformed UTF-8 characters, possibly incorrectly encoded '}' At position 0. #68

Closed MagnuM2K closed 2 years ago

MagnuM2K commented 2 years ago

Hi

When I try to execute the first command to pull watchstate.. I get the following error. Unable to proceed further. 

docker-compose exec watchstate console state:import -vvrm -s plx
[2022-04-22T01:41:59+00:00] logger.INFO: [M: 3.10M] Preloading all mapper data.
[2022-04-22T01:41:59+00:00] logger.INFO: [M: 7.33M] Finished preloading mapper data.
[2022-04-22T01:41:59+00:00] logger.INFO: [M: 7.33M] Single transaction mode
[2022-04-22T01:41:59+00:00] logger.NOTICE: [M: 9.19M] Importing 'plx' play state changes since beginning.
[2022-04-22T01:41:59+00:00] logger.NOTICE: [M: 9.40M] Waiting on (3) HTTP Requests.
[2022-04-22T01:42:00+00:00] logger.INFO: [M: 9.69M] Parsing Successful plx - (Movies) - (movie:3) response.
[2022-04-22T01:42:01+00:00] logger.INFO: [M: 11.20M] Finished Parsing plx - (Movies) - (movie:3) (1011 objects) response.
[2022-04-22T01:42:01+00:00] logger.INFO: [M: 11.19M] Parsing Successful plx - (Movies 4K) - (movie:2) response.
[2022-04-22T01:42:01+00:00] logger.INFO: [M: 11.21M] Finished Parsing plx - (Movies 4K) - (movie:2) (7 objects) response.
[2022-04-22T01:42:03+00:00] logger.INFO: [M: 11.23M] Parsing Successful plx - (TV Shows) - (episode:4) response.

In Parser.php line 368:

  [JsonMachine\Exception\SyntaxErrorException]                                 
  Malformed UTF-8 characters, possibly incorrectly encoded '}' At position 0.  

Exception trace:
  at /app/vendor/halaxa/json-machine/src/Parser.php:368
 JsonMachine\Parser->error() at /app/vendor/halaxa/json-machine/src/Parser.php:220
 JsonMachine\Parser->getIterator() at /app/src/Libs/Servers/PlexServer.php:631
 App\Libs\Servers\PlexServer->App\Libs\Servers\{closure}() at /app/src/Commands/State/ImportCommand.php:252
 App\Commands\State\ImportCommand->process() at /app/src/Commands/State/ImportCommand.php:96
 App\Commands\State\ImportCommand->App\Commands\State\{closure}() at /app/src/Command.php:84
 App\Command->single() at /app/src/Commands/State/ImportCommand.php:96
 App\Commands\State\ImportCommand->runCommand() at /app/src/Command.php:22
 App\Command->execute() at /app/vendor/symfony/console/Command/Command.php:291
 Symfony\Component\Console\Command\Command->run() at /app/vendor/symfony/console/Application.php:989
 Symfony\Component\Console\Application->doRunCommand() at /app/vendor/symfony/console/Application.php:299
 Symfony\Component\Console\Application->doRun() at /app/vendor/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /app/src/Libs/Initializer.php:117
 App\Libs\Initializer->runConsole() at /app/console:66

state:import [-r|--redirect-logger] [-m|--memory-usage] [-f|--force-full] [--proxy PROXY] [--no-proxy NO-PROXY] [--timeout TIMEOUT] [-s|--servers-filter [SERVERS-FILTER]] [--import-unwatched] [--stats-show] [--stats-filter [STATS-FILTER]] [--mapper-direct] [-c|--config CONFIG]
arabcoders commented 2 years ago

Hmm, it seems to be able to parse your movies libraries fine and choke on your tv library, is your frontend server for plex doing compression? could you try running the following commands and then try to sync

docker exec -ti watchstate console servers:edit --key 'options.client.headers.Accept-Encoding' --set '' -- plx
docker exec -ti watchstate console servers:edit --key 'options.client.http_version' --set '1.0' -- plx
MagnuM2K commented 2 years ago

My server sits behind a reverse proxy, Caddy. But when I tested the server I used both the direct http address and also the https link.

My current set is as below using http and when I ran those commands, I get the same error.

Here is my servers.yaml

plx:
  type: plex
  url: 'http://192.168.2.107:32400'
  token: <removed>
  uuid:  <removed>
  user: <removed>
  import:
    enabled: true
  export:
    enabled: true
  webhook:
    import: true
    push: true
    match:
      user: true
      uuid: true
    token: <removed>
  options:
    client:
      headers:
        Accept-Encoding: ''
      http_version: '1.0'
[2022-04-22T17:10:21+00:00] logger.INFO: [M: 38.71M] Parsing Successful plx - (Movies) - (movie:3) response.
[2022-04-22T17:10:22+00:00] logger.INFO: [M: 38.07M] Finished Parsing plx - (Movies) - (movie:3) (1011 objects) response.
[2022-04-22T17:10:22+00:00] logger.INFO: [M: 38.06M] Parsing Successful plx - (Movies 4K) - (movie:2) response.
[2022-04-22T17:10:22+00:00] logger.INFO: [M: 38.07M] Finished Parsing plx - (Movies 4K) - (movie:2) (7 objects) response.
[2022-04-22T17:10:22+00:00] logger.INFO: [M: 38.06M] Parsing Successful plx - (TV Shows) - (episode:4) response.

In Parser.php line 368:

  [JsonMachine\Exception\SyntaxErrorException]                                 
  Malformed UTF-8 characters, possibly incorrectly encoded '}' At position 0.  

Exception trace:
  at /app/vendor/halaxa/json-machine/src/Parser.php:368
 JsonMachine\Parser->error() at /app/vendor/halaxa/json-machine/src/Parser.php:220
 JsonMachine\Parser->getIterator() at /app/src/Libs/Servers/PlexServer.php:631
 App\Libs\Servers\PlexServer->App\Libs\Servers\{closure}() at /app/src/Commands/State/ImportCommand.php:252
 App\Commands\State\ImportCommand->process() at /app/src/Commands/State/ImportCommand.php:96
 App\Commands\State\ImportCommand->App\Commands\State\{closure}() at /app/src/Command.php:84
 App\Command->single() at /app/src/Commands/State/ImportCommand.php:96
 App\Commands\State\ImportCommand->runCommand() at /app/src/Command.php:22
 App\Command->execute() at /app/vendor/symfony/console/Command/Command.php:291
 Symfony\Component\Console\Command\Command->run() at /app/vendor/symfony/console/Application.php:989
 Symfony\Component\Console\Application->doRunCommand() at /app/vendor/symfony/console/Application.php:299
 Symfony\Component\Console\Application->doRun() at /app/vendor/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /app/src/Libs/Initializer.php:117
 App\Libs\Initializer->runConsole() at /app/console:66

state:import [-r|--redirect-logger] [-m|--memory-usage] [-f|--force-full] [--proxy PROXY] [--no-proxy NO-PROXY] [--timeout TIMEOUT] [-s|--servers-filter [SERVERS-FILTER]] [--import-unwatched] [--stats-show] [--stats-filter [STATS-FILTER]] [--mapper-direct] [-c|--config CONFIG]
arabcoders commented 2 years ago

Alright, could you please try running this command to see plex response, it seems something happening in tv response, 

curl -v -H "Accept: application/json" -H "X-Plex-Token: YOUR_PLEX_TOKEN_HERE" 'http://192.168.2.107:32400/library/sections/4/all?type=4&sort=addedAt:asc&includeGuids=1' 2>&1 | tee response.log

it should save a log in your directory called response.log, please edit the log to remove your plex token, after that please attached it here or send it to me at (ws-contact at arabcoders.net)

i need to see the response headers and body from plex to determine why the json parser is failing on this specific library.

MagnuM2K commented 2 years ago

Ok. Just sent you the response log via email.

arabcoders commented 2 years ago

I've successfully parsed your given response, the only thing i noticed that might be a problem is there is invalid UTF-8 characters in your payload. i can probably fix that, and you are still using the old plex agents. which most likely will cause you problems in the future however they are "kinda" supported right now i'll see what i can do. i'll let you know once i have something hopefully by later today or tomorrow.

arabcoders commented 2 years ago

i pushed possible fix for this issue, in an hour i want you to switch to arabcoders/watchstate:dev-latest pull the new container and restart after that try importing

MagnuM2K commented 2 years ago

Ok. Will try and report back in an hour.

I checked my plex agents and they seem to be using the new agents. There is no option to upgrade like stated in the plx server site. My agents are.. The TV agent is : Plex TV Series and movies is : Plex Movie.

Are these are the correct agents?

Thx

arabcoders commented 2 years ago

Indeed those are correct new agents, however it seems you possibly forgot to run full meta refresh on library. Anyway, the dev build is out try now and report back if it fixes your problem.

MagnuM2K commented 2 years ago

I've refreshed my Plex libraries and downloaded new dev image. But still same.

Also ran db list against plx server, and see a small set of watched status.

Any way to get a count from db and compare plx vs jellyfin?

[2022-04-22T19:10:52+00:00] logger.DEBUG: Ignoring plx - (TV Shows) - (episode:4) - [Star Trek: Voyager - (3x11) - The Q and the Grey]. Not watched.
[2022-04-22T19:10:52+00:00] logger.DEBUG: Ignoring plx - (TV Shows) - (episode:4) - [Star Trek: Voyager - (3x7) - Sacred Ground]. Not watched.
[2022-04-22T19:10:52+00:00] logger.DEBUG: Ignoring plx - (TV Shows) - (episode:4) - [Star Trek: Voyager - (4x6) - The Raven]. Not watched.
[2022-04-22T19:10:52+00:00] logger.DEBUG: Ignoring plx - (TV Shows) - (episode:4) - [Star Trek: Voyager - (4x8) - Year of Hell, Part 1]. Not watched.
[2022-04-22T19:10:52+00:00] logger.DEBUG: Ignoring plx - (TV Shows) - (episode:4) - [Warehouse 13 - (1x5) - Elements]. Not watched.
[2022-04-22T19:10:52+00:00] logger.ERROR: Unable to parse plx - (TV Shows) - (episode:4) response. [(file: /app/vendor/halaxa/json-machine/src/Parser.php), (line: 368), (kind: JsonMachine\Exception\SyntaxErrorException), (error: Malformed UTF-8 characters, possibly incorrectly encoded '}' At position 0.)]
[2022-04-22T19:10:52+00:00] logger.NOTICE: Finished waiting HTTP Requests.
[2022-04-22T19:10:52+00:00] logger.INFO: No changes detected.
┌─────────┬───────┬─────────┬────────┐
│ Type    │ Added │ Updated │ Failed │
├─────────┼───────┼─────────┼────────┤
│ Movie   │ 0     │ 0       │ 0      │
├─────────┼───────┼─────────┼────────┤
│ Episode │ 0     │ 0       │ 0      │
└─────────┴───────┴─────────┴────────┘
arabcoders commented 2 years ago

Ok, i pushed another fix to dev-latest should be done building in <30min after that pull restart and try again. if that also does not work, im afraid im out of ideas it's clearly one an item that is not valid UTF as you can see from the log it can parse some items and then bails.

arabcoders commented 2 years ago

it seems this file "/home/media/TVShows/Show.Title/Season.04/ShowTitle s04e20 - Vis � Vis NTSC DVD x264 DD5.1-JCH.mkv" causing the problems. it's the character between vis and Vis it's incorrectly encoded do you not use media manger like sonarr ?

Star Trek: Voyager S04E20

arabcoders commented 2 years ago

nevermind, please try next build. it seems i messed up.

arabcoders commented 2 years ago

@MagnuM2K the new builds are out, please try and report again pull and restart and try to import, now it should ONLY ignore the malformed item instead of bailing out completely on all other items.

MagnuM2K commented 2 years ago

That seemed to have done the trick. Now that I know which file was causing the issue, I've rename it as well.

Thx!

arabcoders commented 2 years ago

Np. glad it worked for you.