New YouTube videos don't show up when using the new synchronization functionality

emansom commented 2 years ago

Describe the current behavior

New YouTube videos don't show up when triggered by the channel synchronization scheduler.

Only when manually clicking the synchronize channel button via the UI it fetches the new YouTube videos.

Steps to reproduce

Setup PeerTube using latest develop branch (2022-09-13)
Synchronize a full YouTube channel and wait until all jobs completed
Publish a new video on the YouTube channel
Observe PeerTube never picking it up automatically

Describe the expected behavior

PeerTube picking up new videos from YouTube automatically every hour via the channel synchronization scheduler.

Additional information

PeerTube instance:
- URL: https://pcbu.nl
- Version: latest code in develop branch (2022-09-12)
- NodeJS version: v16.17.0
- Ffmpeg version: 4.3.4-0+deb11u1
Browser name, version and platforms on which you could reproduce the bug: irrelevant
Link to browser console log if relevant: irrelevant
Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/): https://gist.github.com/emansom/8fcdcf5ae31e47c245c798e25b7b2ee3

Chocobozzz commented 2 years ago

Hi,

Please enable debug logs and check debug: Running VideoChannelSyncLatestScheduler.internalExecute is displayed. Then please paste channels sync related logs

emansom commented 2 years ago

Hi,

Please enable debug logs and check debug: Running VideoChannelSyncLatestScheduler.internalExecute is displayed. Then please paste channels sync related logs

Full log: https://gist.github.com/emansom/8fcdcf5ae31e47c245c798e25b7b2ee3

The VideoChannelSyncLatestScheduler fails to synchronize https://www.youtube.com/watch?v=Mnu6dXmhjCg and https://www.youtube.com/watch?v=vQCZUK9zMYQ for PeerTube channel dpc_shorts (PoppenCast Clips on YouTube).

It runs the yt-dlp commands to list data, but never actually downloads nor publishes the videos.

[pcbu.nl:443] 9/14/2022, 2:26:59 PM.530 debug: Run youtube-dl command. {
  "command": "/usr/bin/python3 /data/bin/yt-dlp --dump-json -f bestvideo[vcodec!*=av01][vcodec!*=vp9.2]+bestaudio/best[vcodec!*=av01][vcodec!*=vp9.2]/bestvideo[ext=mp4]+bestaudio[ext=m4a]/best https://www.youtube.com/watch?v=Mnu6dXmhjCg"
}
[pcbu.nl:443] 9/14/2022, 2:26:59 PM.536 debug: Executed SQL request - Executing (default): SELECT "VideoImportModel"."id", "VideoImportModel"."targetUrl", "VideoImportModel"."magnetUri", "VideoImportModel"."torrentName", "VideoImportModel"."state", "VideoImportModel"."error", "VideoImportModel"."userId", "VideoImportModel"."videoId", "VideoImportModel"."videoChannelSyncId", "VideoImportModel"."createdAt", "VideoImportModel"."updatedAt", "Video"."id" AS "Video.id", "Video"."uuid" AS "Video.uuid", "Video"."name" AS "Video.name", "Video"."category" AS "Video.category", "Video"."licence" AS "Video.licence", "Video"."language" AS "Video.language", "Video"."privacy" AS "Video.privacy", "Video"."nsfw" AS "Video.nsfw", "Video"."description" AS "Video.description", "Video"."support" AS "Video.support", "Video"."duration" AS "Video.duration", "Video"."views" AS "Video.views", "Video"."likes" AS "Video.likes", "Video"."dislikes" AS "Video.dislikes", "Video"."remote" AS "Video.remote", "Video"."isLive" AS "Video.isLive", "Video"."url" AS "Video.url", "Video"."commentsEnabled" AS "Video.commentsEnabled", "Video"."downloadEnabled" AS "Video.downloadEnabled", "Video"."waitTranscoding" AS "Video.waitTranscoding", "Video"."state" AS "Video.state", "Video"."publishedAt" AS "Video.publishedAt", "Video"."originallyPublishedAt" AS "Video.originallyPublishedAt", "Video"."channelId" AS "Video.channelId", "Video"."createdAt" AS "Video.createdAt", "Video"."updatedAt" AS "Video.updatedAt" FROM "videoImport" AS "VideoImportModel" INNER JOIN "video" AS "Video" ON "VideoImportModel"."videoId" = "Video"."id" AND "Video"."channelId" = 3 WHERE "VideoImportModel"."targetUrl" = 'https://www.youtube.com/watch?v=vQCZUK9zMYQ' AND "VideoImportModel"."state" IN (1, 6, 2) LIMIT 1;
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.233 debug: Run youtube-dl command. {
  "command": "/usr/bin/python3 /data/bin/yt-dlp --dump-json -f bestvideo[vcodec!*=av01][vcodec!*=vp9.2]+bestaudio/best[vcodec!*=av01][vcodec!*=vp9.2]/bestvideo[ext=mp4]+bestaudio[ext=m4a]/best https://www.youtube.com/watch?v=vQCZUK9zMYQ"
}
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.243 info: Processing after video channel import in job b0f84468-a940-4b79-86ef-e13e013947c9.
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.244 debug: Executed SQL request - Executing (default): SELECT "VideoChannelSyncModel"."id", "VideoChannelSyncModel"."externalChannelUrl", "VideoChannelSyncModel"."videoChannelId", "VideoChannelSyncModel"."state", "VideoChannelSyncModel"."lastSyncAt", "VideoChannelSyncModel"."createdAt", "VideoChannelSyncModel"."updatedAt", "VideoChannel"."id" AS "VideoChannel.id", "VideoChannel"."name" AS "VideoChannel.name", "VideoChannel"."description" AS "VideoChannel.description", "VideoChannel"."support" AS "VideoChannel.support", "VideoChannel"."actorId" AS "VideoChannel.actorId", "VideoChannel"."accountId" AS "VideoChannel.accountId", "VideoChannel"."createdAt" AS "VideoChannel.createdAt", "VideoChannel"."updatedAt" AS "VideoChannel.updatedAt", "VideoChannel->Actor"."id" AS "VideoChannel.Actor.id", "VideoChannel->Actor"."type" AS "VideoChannel.Actor.type", "VideoChannel->Actor"."preferredUsername" AS "VideoChannel.Actor.preferredUsername", "VideoChannel->Actor"."url" AS "VideoChannel.Actor.url", "VideoChannel->Actor"."publicKey" AS "VideoChannel.Actor.publicKey", "VideoChannel->Actor"."privateKey" AS "VideoChannel.Actor.privateKey", "VideoChannel->Actor"."followersCount" AS "VideoChannel.Actor.followersCount", "VideoChannel->Actor"."followingCount" AS "VideoChannel.Actor.followingCount", "VideoChannel->Actor"."inboxUrl" AS "VideoChannel.Actor.inboxUrl", "VideoChannel->Actor"."outboxUrl" AS "VideoChannel.Actor.outboxUrl", "VideoChannel->Actor"."sharedInboxUrl" AS "VideoChannel.Actor.sharedInboxUrl", "VideoChannel->Actor"."followersUrl" AS "VideoChannel.Actor.followersUrl", "VideoChannel->Actor"."followingUrl" AS "VideoChannel.Actor.followingUrl", "VideoChannel->Actor"."remoteCreatedAt" AS "VideoChannel.Actor.remoteCreatedAt", "VideoChannel->Actor"."serverId" AS "VideoChannel.Actor.serverId", "VideoChannel->Actor"."createdAt" AS "VideoChannel.Actor.createdAt", "VideoChannel->Actor"."updatedAt" AS "VideoChannel.Actor.updatedAt", "VideoChannel->Actor->Server"."id" AS "VideoChannel.Actor.Server.id", "VideoChannel->Actor->Server"."host" AS "VideoChannel.Actor.Server.host", "VideoChannel->Actor->Server"."redundancyAllowed" AS "VideoChannel.Actor.Server.redundancyAllowed", "VideoChannel->Actor->Server"."createdAt" AS "VideoChannel.Actor.Server.createdAt", "VideoChannel->Actor->Server"."updatedAt" AS "VideoChannel.Actor.Server.updatedAt", "VideoChannel->Actor->Avatars"."id" AS "VideoChannel.Actor.Avatars.id", "VideoChannel->Actor->Avatars"."filename" AS "VideoChannel.Actor.Avatars.filename", "VideoChannel->Actor->Avatars"."height" AS "VideoChannel.Actor.Avatars.height", "VideoChannel->Actor->Avatars"."width" AS "VideoChannel.Actor.Avatars.width", "VideoChannel->Actor->Avatars"."fileUrl" AS "VideoChannel.Actor.Avatars.fileUrl", "VideoChannel->Actor->Avatars"."onDisk" AS "VideoChannel.Actor.Avatars.onDisk", "VideoChannel->Actor->Avatars"."type" AS "VideoChannel.Actor.Avatars.type", "VideoChannel->Actor->Avatars"."actorId" AS "VideoChannel.Actor.Avatars.actorId", "VideoChannel->Actor->Avatars"."createdAt" AS "VideoChannel.Actor.Avatars.createdAt", "VideoChannel->Actor->Avatars"."updatedAt" AS "VideoChannel.Actor.Avatars.updatedAt" FROM "videoChannelSync" AS "VideoChannelSyncModel" INNER JOIN "videoChannel" AS "VideoChannel" ON "VideoChannelSyncModel"."videoChannelId" = "VideoChannel"."id" INNER JOIN "actor" AS "VideoChannel->Actor" ON "VideoChannel"."actorId" = "VideoChannel->Actor"."id" LEFT OUTER JOIN "server" AS "VideoChannel->Actor->Server" ON "VideoChannel->Actor"."serverId" = "VideoChannel->Actor->Server"."id" LEFT OUTER JOIN "actorImage" AS "VideoChannel->Actor->Avatars" ON "VideoChannel->Actor"."id" = "VideoChannel->Actor->Avatars"."actorId" AND "VideoChannel->Actor->Avatars"."type" = 1 WHERE "VideoChannelSyncModel"."id" = 2;
[pcbu.nl:443] 9/14/2022, 2:27:01 PM.247 info: Finished synchronizing "dpc_shorts" successfully. {
  "successes": 0
}

Chocobozzz commented 2 years ago

Thanks, it should be fixed by https://github.com/Chocobozzz/PeerTube/commit/3204f4d17dec79413a94eae8a71b72e0c7ec7ba5

emansom commented 2 years ago

@Chocobozzz That fixed it! Thanks! 👍🏻

Chocobozzz commented 2 years ago

I made a mistake in the previous commit, that should be fixed by https://github.com/Chocobozzz/PeerTube/commit/9a3a23a834679e072c15960903e05e667ef14481

Chocobozzz / PeerTube