yamikaitou / YamiCogs

Cogs for Red-Discordbot V3
https://cogs.yamikaitou.dev
Other
7 stars 3 forks source link

Duplicate YouTube Postings & Temporary Solution #25

Closed Wizman87 closed 10 months ago

Wizman87 commented 1 year ago

This issue has been previously mentioned in CBD Cogs' repository. However, I decided to reference it here as well for everyone else that's running into the same issue that I am. I will state too that I prefer and very much enjoy using the YamiCogs version of the Tube cog -- it works very well for both my needs and particular use case. In addition, I also realize there's currently a rework of the Tube cog in progress right now by @yamikaitou. I am both very excited and looking forward to its future release. Anyway, enough of the preface for now and getting back to the main issue at-hand.

When it comes to Tube cog postings of YouTube videos, if the cog happens to have too many assigned YouTube channels to it that it fetches and retrieves new YouTube videos from, the cog's cache will eventually fill the entirety of its default "cache_size" of "500" with previously fetched YouTube videos, and continue to post duplicate URL postings of previously cached YouTube videos both indefinitely and repeatedly every minute into a desired Discord channel. As a result, the cog will continue to post hundreds or even thousands of duplicate YouTube videos every day into a Discord channel. The only workaround, or temporary solution, that I have seen or happened to come across is increasing the "cache_size" value within the "settings.json" configuration file to a much larger, higher value (e.g. "5000"). This fix also happens to have been posted as a comment to a separate unrelated issue on the CBD Cogs' repository here. However, with implementing this fix, it happens to come with a side effect in that it will result in a slightly increased delay to fetching new YouTube videos. In addition, restarting the Discord bot running the cog and issuing a reload of the cog do not refresh the cog's cache of YouTube videos.

I'm uncertain if this would be an easy fix or not.. and I haven't exactly looked into the Python code of the Tube cog myself yet.. though I am sure that I could eventually find a fix to the issue. However, I unfortunately do not currently have enough time on my hands to focus on accomplishing this particular task right now. Anyway, I simply wanted to bring to attention this issue once more, as it's been a major headache and pain in my side for both myself and many others out there for quite some time now. If it's at all possible and not too much to ask, I would hope a commit could possibly be implemented and pushed out in the meantime, at least until the latest rework of the Tube cog has been released.

I would be both deeply and greatly appreciated for anyone's support on this issue. Thank you very much.

yamikaitou commented 1 year ago

The only workaround, or temporary solution, that I have seen or happened to come across is increasing the "cache_size" value within the "settings.json" configuration file to a much larger, higher value (e.g. "5000").

It is better to use the command tube setcache than touching the config file.

Setting the cache higher isn't a fix though, it still happens (at least from what I've been told, I can't reproduce the issue). Cache is also not the only item checked when determining if a video should be posted, the timestamps are also checked.

Wizman87 commented 1 year ago

The botprefix.tube setcache command is not listed within the botprefix.help tube Help Menu, therefore I simply edited the configuration file directly instead. I will be sure to use that command next time though. Thank you very much for that. At this time, the currently listed subcommands described within the Help Menu are: customize, demo, list, rolemention, subscribe, unsubscribe, and update. However, if a botprefix.help tube setcache command is issued, a Help Menu does appear for that specific command.

As for setting the cache higher, you're indeed correct in that it's not a permanent fix or even temporary solution. Over a period of a month, at least in my specific use case and after previously increasing the cache_size value to 3000, the issue reoccurred. I have since updated the cache_size to a higher value of 5000. Unfortunately though, this is not a realistic solution for the long-term. However, it does save me the headache in the short-term of having to frequently edit the settings.json configuration file to remove all of the previously cached entries until it fills up the cache again. I can always clear the entries once more again at a later date and time. Also, I do not believe the code currently checks the timestamps correctly at this time, as otherwise it wouldn't continue to duplicate postings of newly fetched YouTube videos? I will have to take a look at the code in more detail in order to see how exactly it executes this function. For additional clarification, perhaps you could go into more detail on how exactly the code evaluates against these timestamps?

yamikaitou commented 1 year ago

Ah, appears the command is hidden. Did not notice that (or notice I had my bot set to show hidden commands). I'll unhide them later tonight.

If you are able to reproduce the issue easily, please enable the debugger I recently added with tube debugger true (running it without true will return the current state of it as well as the location of the debug logs). Once you get it to reproduce, give it a cycle or 2 and then share the logs with me.

As for more details on the logic, since I didn't actually write the cog, I'm still trying to figure it completely out. The code appears fine at a glance but I could easily be missing something. The way I currently understand it is that it checks the published_parsed value against the stored last_video_time (in config as previous). It is hard for me to diagnose the issue when I can't reproduce it but maybe I'll try your suggestion of lowering my cache size (I've been using 5k for my testing as I was trying to replicate someone elses setup)

Wizman87 commented 1 year ago

I will enable the debugger and report back when the issue is reproduced. I will also attempt lowering my cache_size value back down to 3000 from 5000 to get a report of the logs back to you more quickly. Thank you for the suggestion and explanation on the logic, @yamikaitou. I will be sure to take a look at the logic as well when I have the time.

Wizman87 commented 1 year ago

Duplicate YouTube video postings began a day or two ago, a week or so after enabling the Tube debugger on 2023-09-08. I didn't really notice any errors or warnings related to the duplicate postings. Nearly all of the errors are from issues related to name resolution (DNS). However, even after reconnection to the Discord shard, the duplicate postings continue regardless of whether there are errors or not from name resolution. These errors are simply the only entries that I noticed were written into the log files. The related errors and log entries are written below:

Log entries prior to enabling Tube debugger:

[2023-09-07 01:16:22] [INFO] discord.gateway: Shard ID 0 has connected to Gateway (Session ID: 40a7238dad2f82e94b789b8152c05fbb).
[2023-09-07 03:34:57] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 06:11:14] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 08:16:23] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 08:31:54] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 10:23:17] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 10:54:58] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 12:10:49] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 14:29:19] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 16:02:10] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 18:07:31] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 21:14:53] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 22:33:04] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 40a7238dad2f82e94b789b8152c05fbb.
[2023-09-07 22:56:10] [ERROR] discord.shard: Attempting a reconnect for shard ID 0 in 1.49s
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "uvloop/loop.pyx", line 1528, in getaddrinfo
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/shard.py", line 192, in reidentify
    self.ws = await asyncio.wait_for(coro, timeout=60.0)
  File "/home/dev-redbot/.pyenv/versions/3.9.7/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/gateway.py", line 372, in from_client
    socket = await client.http.ws_connect(str(url))
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/http.py", line 537, in ws_connect
    return await self.__session.ws_connect(url, **kwargs)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 779, in _ws_connect
    resp = await self.request(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host gateway-us-east1-b.discord.gg:443 ssl:default [Temporary failure in name resolution]

[2023-09-07 22:56:32] [ERROR] discord.shard: Attempting a reconnect for shard ID 0 in 3.99s
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "uvloop/loop.pyx", line 1528, in getaddrinfo
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/shard.py", line 210, in reconnect
    self.ws = await asyncio.wait_for(coro, timeout=60.0)
  File "/home/dev-redbot/.pyenv/versions/3.9.7/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/gateway.py", line 372, in from_client
    socket = await client.http.ws_connect(str(url))
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/http.py", line 537, in ws_connect
    return await self.__session.ws_connect(url, **kwargs)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 779, in _ws_connect
    resp = await self.request(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host gateway.discord.gg:443 ssl:default [Temporary failure in name resolution]

[2023-09-07 22:57:41] [ERROR] red.cbd-cogs.tube: Fetch failed for url https://www.youtube.com/feeds/videos.xml?channel_id=UC2t5bjwHdUX4vM2g8TRDq5g:
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "uvloop/loop.pyx", line 1528, in getaddrinfo
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 378, in fetch
    async with session.get(url) as response:
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.youtube.com:443 ssl:default [Temporary failure in name resolution]
[2023-09-07 22:57:41] [ERROR] red.cbd-cogs.tube: Error parsing feed for League of Legends (UC2t5bjwHdUX4vM2g8TRDq5g)
Traceback (most recent call last):
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 291, in _get_new_videos
    cache[sub["id"]] = feedparser.parse(await self.get_feed(sub["id"]))
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/Downloader/lib/feedparser/api.py", line 216, in parse
    data = _open_resource(url_file_stream_or_string, etag, modified, agent, referrer, handlers, request_headers, result)
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/Downloader/lib/feedparser/api.py", line 135, in _open_resource
    return url_file_stream_or_string.encode('utf-8')
AttributeError: 'NoneType' object has no attribute 'encode'
[2023-09-07 22:57:53] [INFO] discord.gateway: Shard ID 0 has connected to Gateway (Session ID: 2727096d847b306d46fd8cc4a3c15247).
[2023-09-08 01:55:46] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 02:20:30] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 03:09:41] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 06:56:39] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 08:15:22] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 09:41:01] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 11:42:37] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 13:02:01] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 14:57:02] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 17:42:03] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.

Log entries after enabling Tube debugger:

[2023-09-07 22:57:53] [INFO] discord.gateway: Shard ID 0 has connected to Gateway (Session ID: 2727096d847b306d46fd8cc4a3c15247).
[2023-09-08 01:55:46] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 02:20:30] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 03:09:41] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 06:56:39] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 08:15:22] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 09:41:01] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 11:42:37] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 13:02:01] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 14:57:02] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 17:42:03] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session 2727096d847b306d46fd8cc4a3c15247.
[2023-09-08 19:05:56] [INFO] red.yamicogs.tube.debugger:
OS version: Debian GNU/Linux 10
User: dev-redbot
Python executable: /home/dev-redbot/.pyenv/versions/venv-redbot/bin/python
Python version: 3.9.7 (64bit)
Pip version: 23.2.1
Red version: 3.5.4
Discord.py version: 2.3.2
https://github.com/yamikaitou/YamiCogs | a411b59bbb7e51ee26a3c62edbc4288c2fa33bc0
Interval: 300 | Cache: 5000
[2023-09-08 19:11:16] [INFO] red.Audio.WS.LLNode: [NODE] | Listener closing: None
[2023-09-08 19:11:16] [INFO] red.Audio.WS.LLNode: Shutdown Lavalink WS.
[2023-09-08 19:11:16] [INFO] red.main: Shutting down with exit code: 26 (RESTART)
[2023-09-08 19:11:16] [INFO] red.main: Please wait, cleaning up a bit more
[2023-09-08 19:11:18] [ERROR] asyncio: Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f14e956de20>

...

Sep 08 19:11:18 debian systemd[1]: red@tubebot.service: Main process exited, code=exited, status=26/n/a
Sep 08 19:11:18 debian systemd[1]: red@tubebot.service: Failed with result 'exit-code'.
Sep 08 19:11:33 debian systemd[1]: red@tubebot.service: Service RestartSec=15s expired, scheduling restart.
Sep 08 19:11:33 debian systemd[1]: red@tubebot.service: Scheduled restart job, restart counter is at 1.
Sep 08 19:11:33 debian systemd[1]: Stopped tubebot redbot.
Sep 08 19:11:33 debian systemd[1]: Started tubebot redbot.

...

[2023-09-08 19:11:33] [INFO] discord.client: logging in using static token
[2023-09-08 19:11:34] [INFO] red: Loading packages...
[2023-09-08 19:11:34] [WARNING] py.warnings: /home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/remindme/__init__.py:17: RuntimeWarning: coroutine 'Red.add_cog' was never awaited
  bot.add_cog(cog)

[2023-09-08 19:11:34] [WARNING] py.warnings: /home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/todo/__init__.py:13: RuntimeWarning: coroutine 'Red.add_cog' was never awaited
  bot.add_cog(Todo(bot))

[2023-09-08 19:11:35] [ERROR] red: Failed to load package Tube
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/redbot/core/bot.py", line 1239, in _pre_connect
    await asyncio.wait_for(self.load_extension(spec), 30)
  File "/home/dev-redbot/.pyenv/versions/3.9.7/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/redbot/core/bot.py", line 1686, in load_extension
    await lib.setup(self)
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/__init__.py", line 6, in setup
    await bot.add_cog(Tube(bot))
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/redbot/core/bot.py", line 1909, in add_cog
    await super().add_cog(cog, guild=guild, guilds=guilds)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/ext/commands/bot.py", line 783, in add_cog
    cog = await cog._inject(self, override=override, guild=guild, guilds=guilds)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/ext/commands/cog.py", line 684, in _inject
    await maybe_coroutine(self.cog_load)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/utils.py", line 693, in maybe_coroutine
    return await value
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 486, in cog_load
    text = text + "\n" + f"{cog.repo.url} | {cog.commit}"
AttributeError: 'NoneType' object has no attribute 'url'

[2023-09-09 17:17:57] [ERROR] red.cbd-cogs.tube: Error parsing feed for League of Legends (UC2t5bjwHdUX4vM2g8TRDq5g)
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "uvloop/loop.pyx", line 1528, in getaddrinfo
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 420, in fetch
    async with session.get(url) as response:
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.youtube.com:443 ssl:default [Temporary failure in name resolution]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 315, in _get_new_videos
    cache[sub["id"]] = feedparser.parse(await self.get_feed(sub["id"]))
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 429, in get_feed
    res = await self.fetch(
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/tubebot/cogs/CogManager/cogs/Tube/tube.py", line 423, in fetch
    await log_exception(f"Fetch failed for url {url}: ", exc_info=e)
NameError: name 'log_exception' is not defined

[2023-09-14 01:41:14] [ERROR] discord.shard: Attempting a reconnect for shard ID 0 in 46.44s
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "uvloop/loop.pyx", line 1528, in getaddrinfo
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/shard.py", line 210, in reconnect
    self.ws = await asyncio.wait_for(coro, timeout=60.0)
  File "/home/dev-redbot/.pyenv/versions/3.9.7/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/gateway.py", line 372, in from_client
    socket = await client.http.ws_connect(str(url))
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/http.py", line 537, in ws_connect
    return await self.__session.ws_connect(url, **kwargs)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 779, in _ws_connect
    resp = await self.request(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host gateway.discord.gg:443 ssl:default [Temporary failure in name resolution]
[2023-09-14 01:42:06] [INFO] discord.gateway: Shard ID 0 has connected to Gateway (Session ID: cd6bb9f78d356cf1cf4ab39112715311).
[2023-09-14 04:44:17] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 06:13:55] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 06:42:41] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 09:13:22] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 10:42:12] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 13:11:04] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 16:41:02] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 19:51:30] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-14 22:03:29] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 00:31:59] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 01:34:16] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 02:40:08] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 04:27:23] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 04:44:13] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 05:42:01] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 05:56:41] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 06:41:59] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 09:15:44] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 09:56:05] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 10:37:38] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 11:42:33] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 12:12:00] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 15:07:57] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 16:37:32] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 17:54:55] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 18:11:22] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 19:43:32] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 20:48:12] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-15 23:57:52] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-16 02:57:24] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-16 06:36:10] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-16 09:42:45] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
[2023-09-16 11:50:39] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session cd6bb9f78d356cf1cf4ab39112715311.
yamikaitou commented 1 year ago

NameError: name 'log_exception' is not defined

This has been fixed

The debugger writes to a different log file, which you can find the location for by running tube debugger. Those are the logs that I need from around the time of duplicate posts

Wizman87 commented 1 year ago

Unfortunately, there does not seem to be any associated errors, warnings, or other related information written within the tube.log file. The only related entries noted are from fetching and processing the YouTube channel feeds.

[2023-09-08 19:05:56] [INFO]:
OS version: Debian GNU/Linux 10
User: dev-redbot
Python executable: /home/dev-redbot/.pyenv/versions/venv-redbot/bin/python
Python version: 3.9.7 (64bit)
Pip version: 23.2.1
Red version: 3.5.4
Discord.py version: 2.3.2
https://github.com/yamikaitou/YamiCogs | a411b59bbb7e51ee26a3c62edbc4288c2fa33bc0
Interval: 300 | Cache: 5000
[2023-09-08 19:10:29] [INFO]: Fetching new videos
[2023-09-08 19:10:29] [INFO]: Processing A (UC-2Y8dQb0S6DtpxNgAKoJKA) for ABC (931860240897564723)
[2023-09-08 19:10:29] [INFO]: Processing B (UCjBp_7RuDBUYbd1LegWEJ8g) for ABC (931860240897564723)
[2023-09-08 19:10:29] [INFO]: Processing C (UCGIY_O-8vW4rfX98KlMkvRg) for ABC (931860240897564723)
[2023-09-08 19:10:30] [INFO]: Processing D (UC2t5bjwHdUX4vM2g8TRDq5g) for DEF (931882125521747992)
[2023-09-08 19:10:30] [INFO]: Processing E (UC6pGDc4bFGD1_36IKv3FnYg) for GHI (1010506404278308875)
[2023-09-08 19:10:30] [INFO]: Processing F (UCV1da9peoqEwqr45bpTJsbQ) for GHI (1010506404278308875)
[2023-09-08 19:10:30] [INFO]: Processing G (UCBSs9x2KzSLhyyA9IKyt4YA) for GHI (1010506404278308875)
[2023-09-08 19:10:31] [INFO]: Processing H (UC14QT5j2nQI8lKBCGtrrBQA) for GHI (1010506404278308875)
[2023-09-08 19:10:31] [INFO]: Processing I (UCDb0peSmF5rLX7BvuTcJfCw) for GHI (1010506404278308875)
Wizman87 commented 1 year ago

I will also add that running the botprefix.tube debugger command does not provide any response or help page from the Discord bot. I had to manually locate the log files for the Tube cog prior to copying and pasting the log entries here.

Edit: This is strange. Please disregard this post as it seems Tube was unloaded somehow by the Discord bot, possibly from a restart? The botprefix.tube debugger command works as intended and described.

Wizman87 commented 1 year ago

I will continue to let Tube run the debugger for a few more cycles before reporting back again, as it seems at some point the Tube cog was unloaded during either an update to it or after a restart of the Discord bot. I am uncertain as to which caused it to do this. With that said however, the Tube cache is already full, so I should be able to see and post additional log entries again here very soon.

yamikaitou commented 1 year ago

Just found an issue where the videos being processed wasn't actually being logged (or at least it wasn't on my bot), so that explains why the log looks useless.

Wizman87 commented 1 year ago

After running Tube for a few cycles, there doesn't seem to be much out of the ordinary within either the core logs or Tube debug logs. The core logs continue to write entries related to domain name resolution and the debug logs continue to fetch the same videos with the same timestamps and write entries related to eligible videos found. Both log entries are written below:

latest.log

[2023-09-17 22:28:02]
OS version: Debian GNU/Linux 10
User: dev-redbot
Python executable: /home/dev-redbot/.pyenv/versions/venv-redbot/bin/python
Python version: 3.9.7 (64bit)
Pip version: 23.2.1
Red version: 3.5.4
Discord.py version: 2.3.2
https://github.com/yamikaitou/YamiCogs | 154ca8a7c03b054421ba1919dcc2379fce9ddbc0
Interval: 300 | Cache: 3000

Tube.log

[2023-09-21 20:17:57] [DEBUG]: Last Video: 2023-09-21 16:00:29
[2023-09-21 20:17:57] [DEBUG]: Last Video: 2023-09-21 16:00:29
[2023-09-21 20:22:55] [INFO]: Fetching new videos
[2023-09-21 20:22:55] [INFO]: Fetching new videos
[2023-09-21 20:22:55] [INFO]: Processing PlayStation (UC-2Y8dQb0S6DtpxNgAKoJKA) for aaa-news (931860240897564723)
[2023-09-21 20:22:55] [INFO]: Processing PlayStation (UC-2Y8dQb0S6DtpxNgAKoJKA) for aaa-news (931860240897564723)
[2023-09-21 20:22:55] [DEBUG]: Last Video: 2023-09-21 15:45:01
[2023-09-21 20:22:55] [DEBUG]: Last Video: 2023-09-21 15:45:01
[2023-09-21 20:22:55] [DEBUG]: Eligible Video Found HzbJsCpSZEE
published: 2023-09-21T15:45:01+00:00
published_parsed: 2023-09-21 16:45:01
updated: 2023-09-21T15:45:01+00:00
updated_parsed: 2023-09-22 01:07:47
Config previous:2023-09-21T15:45:01+00:00
[2023-09-21 20:22:55] [DEBUG]: Eligible Video Found HzbJsCpSZEE
published: 2023-09-21T15:45:01+00:00
published_parsed: 2023-09-21 16:45:01
updated: 2023-09-21T15:45:01+00:00
updated_parsed: 2023-09-22 01:07:47
Config previous:2023-09-21T15:45:01+00:00
[2023-09-21 20:22:55] [INFO]: Processing Xbox (UCjBp_7RuDBUYbd1LegWEJ8g) for aaa-news (931860240897564723)
[2023-09-21 20:22:55] [INFO]: Processing Xbox (UCjBp_7RuDBUYbd1LegWEJ8g) for aaa-news (931860240897564723)
[2023-09-21 20:22:55] [DEBUG]: Last Video: 2023-09-22 01:26:53
[2023-09-21 20:22:55] [DEBUG]: Last Video: 2023-09-22 01:26:53
[2023-09-21 20:22:55] [INFO]: Processing Nintendo (UCGIY_O-8vW4rfX98KlMkvRg) for aaa-news (931860240897564723)
[2023-09-21 20:22:55] [INFO]: Processing Nintendo (UCGIY_O-8vW4rfX98KlMkvRg) for aaa-news (931860240897564723)
[2023-09-21 20:22:55] [DEBUG]: Last Video: 2023-09-22 02:00:02
[2023-09-21 20:22:55] [DEBUG]: Last Video: 2023-09-22 02:00:02
[2023-09-21 20:22:55] [DEBUG]: Eligible Video Found 3WRR0KhaXtI
published: 2023-09-22T02:00:02+00:00
published_parsed: 2023-09-22 03:00:02
updated: 2023-09-22T02:00:02+00:00
updated_parsed: 2023-09-22 04:00:53
Config previous:2023-09-22T02:00:02+00:00
[2023-09-21 20:22:55] [DEBUG]: Eligible Video Found 3WRR0KhaXtI
published: 2023-09-22T02:00:02+00:00
published_parsed: 2023-09-22 03:00:02
updated: 2023-09-22T02:00:02+00:00
updated_parsed: 2023-09-22 04:00:53
Config previous:2023-09-22T02:00:02+00:00

...

[2023-09-22 01:27:55] [INFO]: Processing Nintendo (UCGIY_O-8vW4rfX98KlMkvRg) for aaa-news (931860240897564723)
[2023-09-22 01:27:55] [INFO]: Processing Nintendo (UCGIY_O-8vW4rfX98KlMkvRg) for aaa-news (931860240897564723)
[2023-09-22 01:27:55] [DEBUG]: Last Video: 2023-09-22 02:00:02
[2023-09-22 01:27:55] [DEBUG]: Last Video: 2023-09-22 02:00:02
[2023-09-22 01:27:55] [DEBUG]: Eligible Video Found 3WRR0KhaXtI
published: 2023-09-22T02:00:02+00:00
published_parsed: 2023-09-22 03:00:02
updated: 2023-09-22T02:00:02+00:00
updated_parsed: 2023-09-22 04:00:53
Config previous:2023-09-22T02:00:02+00:00
[2023-09-22 01:27:55] [DEBUG]: Eligible Video Found 3WRR0KhaXtI
published: 2023-09-22T02:00:02+00:00
published_parsed: 2023-09-22 03:00:02
updated: 2023-09-22T02:00:02+00:00
updated_parsed: 2023-09-22 04:00:53
Config previous:2023-09-22T02:00:02+00:00

latest.log

[2023-09-24 01:17:29] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session afb2c085684a378ae7c424031a055cd8.
[2023-09-24 03:36:38] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session afb2c085684a378ae7c424031a055cd8.
[2023-09-24 06:13:43] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session afb2c085684a378ae7c424031a055cd8.
[2023-09-24 06:38:07] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session afb2c085684a378ae7c424031a055cd8.
[2023-09-24 08:52:55] [ERROR] discord.ext.tasks: Unhandled exception in internal background task 'background_get_new_videos'.
Traceback (most recent call last):
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
    await self.coro(*args, **kwargs)
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/girbot/cogs/CogManager/cogs/Tube/tube.py", line 443, in background_get_new_videos
    update = await self._get_new_videos(guild, fetched)
  File "/home/dev-redbot/.local/share/Red-DiscordBot/data/girbot/cogs/CogManager/cogs/Tube/tube.py", line 386, in _get_new_videos
    message = await channel.send(content=description, allowed_mentions=mentions)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/abc.py", line 1561, in send
    data = await state.http.send_message(channel.id, params=params)
  File "/home/dev-redbot/.pyenv/versions/venv-redbot/lib/python3.9/site-packages/discord/http.py", line 743, in request
    raise DiscordServerError(response, data)
discord.errors.DiscordServerError: 503 Service Unavailable (error code: 0): upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2023-09-24 09:29:18] [INFO] discord.gateway: Shard ID 0 has successfully RESUMED session afb2c085684a378ae7c424031a055cd8.

Tube.log

[2023-09-24 08:52:55] [INFO]: Fetching new videos
[2023-09-24 08:52:55] [INFO]: Fetching new videos
[2023-09-24 08:52:55] [INFO]: Processing PlayStation (UC-2Y8dQb0S6DtpxNgAKoJKA) for aaa-news (931860240897564723)
[2023-09-24 08:52:55] [INFO]: Processing PlayStation (UC-2Y8dQb0S6DtpxNgAKoJKA) for aaa-news (931860240897564723)
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 13:00:02
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 13:00:02
[2023-09-24 08:52:55] [INFO]: Processing Xbox (UCjBp_7RuDBUYbd1LegWEJ8g) for aaa-news (931860240897564723)
[2023-09-24 08:52:55] [INFO]: Processing Xbox (UCjBp_7RuDBUYbd1LegWEJ8g) for aaa-news (931860240897564723)
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 13:00:31
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 13:00:31
[2023-09-24 08:52:55] [INFO]: Processing Nintendo (UCGIY_O-8vW4rfX98KlMkvRg) for aaa-news (931860240897564723)
[2023-09-24 08:52:55] [INFO]: Processing Nintendo (UCGIY_O-8vW4rfX98KlMkvRg) for aaa-news (931860240897564723)
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-22 15:00:02
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-22 15:00:02
[2023-09-24 08:52:55] [INFO]: Processing League of Legends (UC2t5bjwHdUX4vM2g8TRDq5g) for league-of-legends (931882125521747992)
[2023-09-24 08:52:55] [INFO]: Processing League of Legends (UC2t5bjwHdUX4vM2g8TRDq5g) for league-of-legends (931882125521747992)
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-21 17:00:13
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-21 17:00:13
[2023-09-24 08:52:55] [INFO]: Processing Crunchyroll Collection (UC6pGDc4bFGD1_36IKv3FnYg) for anime (1010506404278308875)
[2023-09-24 08:52:55] [INFO]: Processing Crunchyroll Collection (UC6pGDc4bFGD1_36IKv3FnYg) for anime (1010506404278308875)
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 15:00:55
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 15:00:55
[2023-09-24 08:52:55] [INFO]: Processing vizmedia (UCV1da9peoqEwqr45bpTJsbQ) for anime (1010506404278308875)
[2023-09-24 08:52:55] [INFO]: Processing vizmedia (UCV1da9peoqEwqr45bpTJsbQ) for anime (1010506404278308875)
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 14:00:34
[2023-09-24 08:52:55] [DEBUG]: Last Video: 2023-09-24 14:00:34
[2023-09-24 08:52:55] [INFO]: Processing Netflix Anime (UCBSs9x2KzSLhyyA9IKyt4YA) for anime (1010506404278308875)
[2023-09-24 08:52:55] [INFO]: Processing Netflix Anime (UCBSs9x2KzSLhyyA9IKyt4YA) for anime (1010506404278308875)
[2023-09-24 08:52:56] [DEBUG]: Last Video: 2023-09-24 13:00:00
[2023-09-24 08:52:56] [DEBUG]: Last Video: 2023-09-24 13:00:00
[2023-09-24 08:52:56] [INFO]: Processing アニプレックス チャンネル (UC14QT5j2nQI8lKBCGtrrBQA) for anime (1010506404278308875)
[2023-09-24 08:52:56] [INFO]: Processing アニプレックス チャンネル (UC14QT5j2nQI8lKBCGtrrBQA) for anime (1010506404278308875)
[2023-09-24 08:52:56] [DEBUG]: Last Video: 2023-09-24 10:29:39
[2023-09-24 08:52:56] [DEBUG]: Last Video: 2023-09-24 10:29:39
[2023-09-24 08:52:56] [DEBUG]: Eligible Video Found XL7xjZdLEak
published: 2023-09-24T10:29:39+00:00
published_parsed: 2023-09-24 11:29:39
updated: 2023-09-24T10:29:39+00:00
updated_parsed: 2023-09-24 13:40:10
Config previous:2023-09-24T10:29:39+00:00
[2023-09-24 08:52:56] [DEBUG]: Eligible Video Found XL7xjZdLEak
published: 2023-09-24T10:29:39+00:00
published_parsed: 2023-09-24 11:29:39
updated: 2023-09-24T10:29:39+00:00
updated_parsed: 2023-09-24 13:40:10
Config previous:2023-09-24T10:29:39+00:00
[2023-09-24 08:53:00] [INFO]: Processing Aniplex USA (UCDb0peSmF5rLX7BvuTcJfCw) for anime (1010506404278308875)
[2023-09-24 08:53:00] [INFO]: Processing Aniplex USA (UCDb0peSmF5rLX7BvuTcJfCw) for anime (1010506404278308875)
[2023-09-24 08:53:01] [DEBUG]: Last Video: 2023-09-23 16:00:06
[2023-09-24 08:53:01] [DEBUG]: Last Video: 2023-09-23 16:00:06
[2023-09-24 08:57:55] [INFO]: Fetching new videos
[2023-09-24 08:57:55] [INFO]: Fetching new videos
Wizman87 commented 1 year ago

I also noticed that the duplicate videos only posted into the same two text channels and not the others (out of 12 text channels). Perhaps this is related to the frequency of posted videos by the respective channels? Eventually, the same duplicate videos that are posted get cycled into different duplicate videos for awhile, and so on.

yamikaitou commented 1 year ago

Can you share all the full logs with me? Feel free to DM it to me on Discord if you'd rather that.

I'm seeing 2 things odd in those snippets. First being the difference between published and published_parsed (same as updated and updated_parsed). I've never used feedparser before, so I'll need to look into why that occurred.

Second being the duplicate log entries. I wonder if this is related to the traceback you provided. I can definitely add in some catches for that.

yamikaitou commented 1 year ago

Oh, forgot to ask, just to make absolutely sure, you aren't running multiple bots from this same datapath, correct?

Wizman87 commented 1 year ago

Oh, forgot to ask, just to make absolutely sure, you aren't running multiple bots from this same datapath, correct?

That's right. I am only running a single bot at this time.

Wizman87 commented 1 year ago

Can you share all the full logs with me? Feel free to DM it to me on Discord if you'd rather that.

I'm seeing 2 things odd in those snippets. First being the difference between published and published_parsed (same as updated and updated_parsed). I've never used feedparser before, so I'll need to look into why that occurred.

Second being the duplicate log entries. I wonder if this is related to the traceback you provided. I can definitely add in some catches for that.

We can chat over Discord if you prefer. I'd rather not post my Discord Tag/ID on here though. If you'd like, you can provide your Discord Tag/ID and I will add you so we can talk some more on there.

As for the snippets, almost everything was included from the log entries. The other entries are more or less the same, though with different timestamps. I made sure to condense only the relevant entries here. Though, if you'd still like the full logs, I can certainly provide them to you over Discord. They're quite large in size, roughly ~2 MB and ~8 MB each, respectively.

I'm not familiar with the differences between the published, published_parsed, updated, and updated_parsed entries either. Again, this is all running from the same Red bot process with the Tube cog loaded. The other loaded cogs are:

17 loaded: audio, cleanup, covid, downloader, giftaway, isitdown, lseen, mcsvr, membercount, remindme, spoilerer, srrecords, sw, todo, Tube, weather, whoplays

I do not believe any of these cogs conflict with the Tube cog.

Please let me know when the catches are added in the code and I'll be sure to update the Tube cog then re-run it for a few cycles with the debugger enabled. Thanks!

yamikaitou commented 1 year ago

Cool, I just wanted to make sure that 2 bots weren't writing to the log at the same time.

I'm in both Red and Cog Support under yamikaitou. Your snippets do help but I want to see the flow of the issue just to see if there is anything odd about it. I can work with what you shared if you'd rather keep it.

Wizman87 commented 1 year ago

Absolutely. I have sent the full logs to you in a DM over Discord. If you happen to come across anything or have any questions for me, please feel free to let me know on here or Discord. Thanks, @yamikaitou!

Wizman87 commented 1 year ago

After running Tube over the past two weeks, I have not encountered any duplicate YouTube postings - yet! I believe the latest commit #4c82a67 did the trick! With that said, I believe we can close the issue as its successfully been resolved. Thank you so much for the support, @yamikaitou!