search

ZeroQI / YouTube-Agent.bundle

Plex Metadata Agent for Movies and TV Series libraries
452 stars 43 forks source link

Channel metadata and thumbnail not appearing but episode metadata and thumbnails work fine #92

Open mileyvirus opened 2 years ago

mileyvirus commented 2 years ago

Hello there, excellent work with the agent. Unfortunately it seems there's an issue with the channel metadata and thumbnail not appearing but episode metadata and thumbnails work fine.

Screenshot of "Channel metadata and thumbnail not appearing"

chrome_WhkDbpUtpg

Some channel metadata gets scraped apparently

chrome_Rx73LSphpz

Screenshot of "episode metadata and thumbnails working fine"

chrome_bBSgB3WEXn

Log file with API key redacted

com.plexapp.agents.youtube.log

ZeroQI commented 2 years ago

No search log in logs provided No Plex token ID file Language xn for library = Chinese?

INFO (logkit:16) - Place correct Plex token in X-Plex-Token.id file in logs folder or in PLEX_LIBRARY_URL variable to have a log per library - https://support.plex.tv/hc/en-us/articles/204059436-Finding-your-account-token-X-Plex-TokenHTTP Error 401: Unauthorized [ ] dir: "D:\Media\YouTube\belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]" INFO (logkit:16) - No GUID so random folder

(https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py#L533)

UCXvKUavCtDOlA8bT1i2tI3w is 22 chars after UC and match description

Missing search log in logs so cannot check guide assignment but missing token ID file...

mileyvirus commented 2 years ago

Sorry and thanks for your time. Here's both logs after:

It still isn't working unfortunately…

com.plexapp.agents.youtube.log com.plexapp.system.log

Language xn for library = Chinese?

Language is set to None in Library settings, unsure why the logs state "xn"

If it's any help I'm using yt-dlp 2021.12.27.

Library file tree

.
├── belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].description
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].info.json
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].mkv
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].webp
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].description
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].info.json
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].mkv
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].webp
│  ├── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].description
│  ├── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].info.json
│  └── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].jpg
└── yotobi [UCIPsK5xspHC3-ZFNPTx2X_w]
   ├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].description
   ├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].info.json
   ├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].mkv
   └── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].webp

I tried an old version of YouTube-Agent.bundle I had on an iMac. That one downloads metadata and images for the channels but with wrong image and it sets the title to the filepath.

This old version: YouTube-Agent.bundle.zip

chrome_V3V1eCoZjS

ZeroQI commented 2 years ago

Can access library file now but issue persists. No search was done on that series in logs...

mileyvirus commented 2 years ago

I'm sorry I don't understand if I should search something on Plex or you're referring to the agent not searching?

ZeroQI commented 2 years ago

I need you to match manually the series so I have that in logs, the agent search() funct89n assign a unique I'd and it seems it is what doesn't work here

mileyvirus commented 2 years ago

Hello again, hope you have a good week.

If you meant "Fix match" and then manually matching the YouTube channel, I did just that and the correct metadata for the channel got scraped BUT now the episodes metadata don't get collected.

Here are the logs:

com.plexapp.agents.youtube.log com.plexapp.system.log

cbartondock commented 2 years ago

Just wanted to chime in that I'm also seeing this issue - episode metadata is loading fine but series metadata no longer is. No images for the series and the [RANDOM LETTERS AND NUMBERS] are still in the title of the series. Let me know if you'd like me to get logs also.

mileyvirus commented 2 years ago

Just wanted to chime in that I'm also seeing this issue - episode metadata is loading fine but series metadata no longer is. No images for the series and the [RANDOM LETTERS AND NUMBERS] are still in the title of the series. Let me know if you'd like me to get logs also.

@cbartondock please do! That way ZeroQI can see your logs together with mine as soon as they have time. I'm pasting the instructions to attach the logs if it's of any help:

cbartondock commented 2 years ago

Same error as you looks like, the key line being Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'.

snip

com.plexapp.agents.youtube.log com.plexapp.system.log

cbartondock commented 2 years ago

@mileyvirus Does the X-Plex-Token.id file just have the token string and nothing else in it? Doesn't seem to be working for me.

mileyvirus commented 2 years ago

@cbartondock yes, just the token string and nothing else and no line breaks either. I put mine in two places since I didn't figure which was the right path.

C:\Users\<username>\AppData\Local\Plex Media Server\X-Plex-Token.id
C:\Users\<username>\AppData\Local\Plex Media Server\Logs\X-Plex-Token.id

The content of X-Plex-Token.id looks like this (I anonymized mine for posting it here)

HEYXX9aT8UxW7igoDuCk
cbartondock commented 2 years ago

Okay yeah, I just worked out by looking over __index.py__ that actually the correct path is the first one C:\Users\Chris Barton Dock\AppData\Local\Plex Media Server. Now my logs look better, but still no dice. Going to try debugging __index.py__ further. com.plexapp.agents.youtube.log com.plexapp.system.log .

cbartondock commented 2 years ago

It seems to be grabbing the guid from a random episode of the channel rather than for the channel itself.

ZeroQI commented 2 years ago

"No GUID so random folder" so it considers the folder doesn't have a channel ID indicated... You should output the metadata.id in logs to see if the search part got the channel id, and if not, you might have used the Plex scanner that would delete all in square brackets which is the channel id...

cbartondock commented 2 years ago

Pretty sure this is a bug. It is setting the channel guid to the guid of a random episode. metadata.id is youtube|aPhrTOg1RUk|Contrapoints [UCNvsIonJdJ5E4EXMa65VYpA] when I believe it should be youtube|UCNvsIonJdJ5E4EXMa65VYpA|Contrapoints [UCNvsIonJdJ5E4EXMa65VYpA]. Here is the directory snip3 And here is the Info.xml file that is being set in C:\Users\Chris Barton Dock\AppData\Local\Plex Media Server\Metadata\TV Shows\e\73abc82969112e63ba6300258b3a9a97c05c7a2.bundle\Contents\com.plexapp.agents.youtube: snip2

cbartondock commented 2 years ago

I believe this is occurring as early as media.filename in the Search(...) function.

cbartondock commented 2 years ago

The reason it says No GUID is because the GUID is an episode GUID, which it doesn't recognize since it doesn't start with UC or HC

ZeroQI commented 2 years ago

ASS scanner let the title with square brackets with channel I'd pass and the agent search function assign a forced guid. Plex scanner will drop square brackets and would cause that exact behaviour, so is ASS the scanner? If so, un-assign, re-assign the series and attach the agents logs so they include the search part.

Edit: Please test https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py line 120 should be display_name instead of filename

cbartondock commented 2 years ago

Ah I see, I don't think I am using ASS and I see now that it is a requirement for YouTube-Agent. Weird that it was working before. If you don't mind my asking, the documentation for ASS (what it does and how to use it for YouTube-Agent) is pretty opaque, how do I set-up ASS?

If not no worries, I got it working in the end without ASS by patching __init__.py with the following (added after line 109):

  a_list=[x for x in os.listdir(dir) if x.endswith('.info.json') and re.search(r'\[(.*)\]',x).group(1)[0:2] in ('PL', 'UU', 'FL', 'LP', 'RD','UC','HC')]
  if len(a_list)>0:
    good_guid = re.search(r'\[(.*)\]',a_list[0]).group(1)
    try:     json_video_details = JSON.ObjectFromString(Core.storage.load(json_filename))  #with open(json_filename) as f:  json_video_details = JSON.ObjectFromString(f.read())
    except:  pass
    else:
      results.Append( MetadataSearchResult( id='youtube|{}|{}'.format(good_guid, os.path.basename(dir)), name=displayname, year=Datetime.ParseDate(Dict(json_video_details, 'upload_date')).year, score=100, lang=lang ) )
      Log(u''.ljust(157, '='))
      return

Basically it just forcibly gets the good guid from the correct .info.json

mileyvirus commented 2 years ago

Edit: Please test https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py line 120 should be display_name instead of filename

@ZeroQI I tried that change but it still won't work for me. I have the latest ASS.

If not no worries, I got it working in the end without ASS by patching init.py with the following (added after line 109):

@cbartondock I tried your change but it won't work for me.

I'm a Python 3 dev but opening the Agent's code it looks like it's all Python 2 and it has limited libraries. I'm going to look more into how Plex agents work. I tried the other YouTube agent and that works but with some other issues…

ZeroQI commented 2 years ago

ASS is a requirement for playlist and channel guide in title functionality, as well as numbering playlist correctly...

My agent also read metadata from local Json file if present like the other agent...

Where are the search agent logs after the change? The change will take the ass title with brackets, will make no change with Plex default scanner...

ZeroQI commented 2 years ago

We need code working within Json then working with local Json file so it benefits everybody. I unfortunately don't use Plex anymore, so don't reproduce libraries, and so need the log to do the coding on the phone most of times

sourcecodemage commented 2 years ago

I was able to recreate this issue and resolve it ( at least for myself). Initially I was using the TV scanner, and I inserted a "air date" into the file name to make that scanner happy. I also configured youtube-dl to create an "info.json" file for each episode.

The fix was to change the scanner to the ASS scanner, and remove the "info.json" files from the series folder.

To verify the .json file was the issue, I recreated the library with 2 "series", one with the info.json files and one without. The series without the json file has the correct series title, the other has "foldername + channel id" in the name.

One think I could not do was remove the json file and rescan the library to correct the series name. Perhaps I am being impatient, or I'm not performing the correct steps to fix the series name. Ideas?

ZeroQI commented 2 years ago

Without Jason it works With Jason it gives "foldername + channel id" in the name. When you remove the Json file and rescan (after Plex dance), does it changes anything? Would need agent logs for both situations

sourcecodemage commented 2 years ago

I can't attach files here. I'll open another issue. with logs and additional details.

edit 1: it looks like github doesn't let you attach logs to issues. I'm looking for an alternative way to share the logs.

edit 2: it accepts .log but not archive file like .tgz. I'll post logs and details in another post.

ZeroQI commented 2 years ago

It does but depends the extension, try txt

sourcecodemage commented 2 years ago

Issue

Channel metadata inc poster missing. episode name/metadata is correct. Channel name is the channel folder name.

Example:

Folder is Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], and channel name is Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].

Notable Observations

Environment

Plex Token ID File: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/X-Plex-Token.id

Log Excerpts

Plex token and json from top part of com.plex.agent.youtube.log:

2022-05-20 14:06:57,086 (7f162ad1c808) :  INFO (__init__:549) - 'X-Plex-Token.id' file present
2022-05-20 14:06:57,100 (7f1626e3bb38) :  WARNING (data:179) - Error decoding with simplejson, using demjson instead (this will cause a performance hit) - Expecting value: line 5 column 1 (char 518)

Exception getting hosted resource hashes from com.plex.agent.youtube.log:

2022-05-20 12:53:17,364 (7fc7e8ad8808) :  DEBUG (networking:143) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2022-05-20 12:53:17,367 (7fc7e8ad8808) :  ERROR (networking:196) - Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2022-05-20 12:53:17,369 (7fc7e8ad8808) :  CRITICAL (runtime:1299) - Exception getting hosted resource hashes (most recent call last):
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/runtime.py", line 1293, in get_resource_hashes
    json = self._core.networking.http_request("http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes", timeout=10).content
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 242, in content
    return self.__str__()
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 220, in __str__
    self.load()
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 158, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found

Data for single channel from com.plex.agent.youtube.log:

2022-05-20 14:10:27,045 (7f1626e3bb38) :  DEBUG (networking:143) - Requesting 'http://127.0.0.1:32400/library/metadata/226165/tree'
2022-05-20 14:10:27,053 (7f1626e3bb38) :  INFO (__init__:165) - === update(lang=xn, force=True, movie=False) ===
2022-05-20 14:10:27,053 (7f1626e3bb38) :  INFO (__init__:40) - /mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]/The_Press_vs_Trump_-_Volume_Three_The_Charlottesville_Lie [4jTf3CMbHrk].mp4
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:509) - =============================================================================================================================================================
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:252) - [ ] library:    "YT"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:253) - [ ] root:       "/mnt/smb/youtube2"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:254) - [ ] path:       "Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:255) - [ ] dir:        "/mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:260) - [ ] series_root_folder: "/mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,056 (7f1626e3bb38) :  INFO (__init__:263) - [ ] subfolder_count:    "0"
2022-05-20 14:10:27,057 (7f1626e3bb38) :  INFO (__init__:378) - Grouping folder not found or single folder, root: /mnt/smb/youtube2, path: Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], Grouping folder: Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], subdirs: 0, reverse_path: [u'Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]']
2022-05-20 14:10:27,057 (7f1626e3bb38) :  INFO (__init__:400) - No GUID so random folder
2022-05-20 14:10:27,057 (7f1626e3bb38) :  INFO (__init__:410) - =============================================================================================================================================================

Log file attachments

Plex Media Scanner.log

com.plexapp.agents.youtube.log

ASS Scanner Logs for channel I'm focusing on were empty so I can't attach them. Here are their names.

sourcecodemage commented 2 years ago

ok.I redownloaded my library again with better results. Since I started, I made the following changes 1) I added the Plex Token file 2) (yt-dlp) added --restrict-filename to only allow ascii filenames. 3) (yt-dlp) included the channel name in the filename to remove non-ascii chars from that as well. My channel name had some accented chars in them.

Channel Names , episode names, metadata, and images are all correct now. ( yay!) .

However, I found a new problem. Youtube episodes released on the same day are absent. I had a case where 5 of them had the same modified date and none of them were discovered by the scanner., The Plex scanner log has the write number of episodes, but doesn't add them.

This is my file naming convention, is correct? Or should I change it? I don't use playlist so there isn't a playlist id. -o "%(channel)s [%(channel_id)s]/%(title)s [%(id)s].%(ext)s"

The overall script looks like this ${ytdlp_path} \ --match-filter "is_live != true & was_live != true" \ ${PROXY_CMD} \ --abort-on-error \ --no-warnings \ --restrict-filenames \ --no-progress \ --add-metadata \ --ffmpeg-location "${ffmpeg_path}" \ --recode "mp4" \ --batch-file="${batch_file}" \ --cookies "${cookies_file}" \ --download-archive "${download_archive}" \ -o "%(channel)s [%(channel_id)s]/%(title)s [%(id)s].%(ext)s" \ --exec post_process:"./scripts/move_video_file.sh -f '%(filepath)s' -r '${root_path}'"

I made the mistake of touching all of 24 file in a channel folder, and not only one is shown in Plex. So the scanner seems to be using the modified date and only allowing one episode per day.

I tried a few things, but what got me results was adding MMDDhhmm to the file name. The scanner picks up these files and I see them in Plex. But the youtube agent doesn't fetch the metadata then, so the episode title /etc are wrong. The filename becomes the title.

So I need a file naming format that makes both ASS and Youtube Agent happy.

sourcecodemage commented 2 years ago

I resolved the episode issue with 2 changes

  1. change the youtube type to "youtube2" by changing the channel ID field to [youtube2-id] as per the documentation.
  2. (tricky part) I scripted adding a randomized hour and minute to the "release_date" value from yt-dlp. The value is applied to the video file after copying to the destination folder. 
# generate random hour/minute
HH=$(( 1 + $RANDOM % 23))
MM=$((1 + $RANDOM % 60))
# change modified date using touch command 
touch -m -t "${date_string}${HH}${MM}" "${DESTPATH}${FILENAME}"

ASS will process all of the video files include those with identical release dates.

I'm creating a dev environment to see if I come up with a way to have ASS handle identical release dates on it's own. Moving responsibility for this from the end user to the application.

Btw, ASS/Youtube-Agent are great tools. I threw a donation the dev's way and recommend others do the same.

I believe the root causes for the issue I ran into were the missing Plex token, the non-ascii characters, and duplicate release_dates on video files in the same series.