Youtube not pulling all videos

[Legalien]

Hey, I registered on Github to thank you guys for this wonderful... what to call it... plugin, solution. I needed it for something for Twitter and it works great. Then I also figured out I can use it for something for Youtube, but I ran into a problem.

I have a channel with 18 videos. When I use Rss Bridge it pulls a max of 14 videos :/

I tried looking through the YoutubeBridge file, changing "$item_count <= 15" to 20, and also changing these to 20 - "'exampleValue' => '15'," in two places.

I saved settings, went to my Rss Bridge homepage again, tried creating feed with Username and with ID, and both times only 14 out of the 18 videos were shown.

I also saw this in the YoutubeBridge file:

"* WARNING: to parse big playlists (over ~90 videos), you need to edit simple_html_dom.php:

• change: define('MAX_FILE_SIZE', 600000);
• into: define('MAX_FILE_SIZE', 900000); (or more)"

So I went looking through the simple_html_dom.php file, but didn't see anything that made sense.

I also searched and looked through the wiki here and the threads that were created around Youtube, but still came up empty. Unfortunately it seems that the RSS Bridge basically pulls the same amount of posts that this URL would create -

https://www.youtube.com/feeds/videos.xml?channel_id=XXX

But I'm sure there's a way to have RSS bridge pull all the posts from the channel. The channel I'm talking about only has 18 videos. It's not too big.

Frankly, I would even be willing to insert the last 4 videos in the feed manually, just as long as they stay in the feed and the feed then gets updated when new videos are posted.

Any way to make RSS Bridge pull all the videos from a Youtube Channel (Username or User ID, all the same)? And if not, any way I can edit the existing feed and manually add the videos that are not pulled, but then to make the feed Remember them while still updating when new videos are published on the channel?

Thanks :)

[logmanoriginal]

Hey, I registered on Github to thank you guys for this wonderful... what to call it... plugin, solution.

Welcome to GitHub and welcome to RSS-Bridge! As anything else on GitHub, RSS-Bridge is a project :wink:

I tried looking through the YoutubeBridge file, changing "$item_count <= 15" to 20, and also changing these to 20 - "'exampleValue' => '15'," in two places.

exampleValue is only used for display on the main page of RSS-Bridge, it has no effect on the actual results. Its purpose is to suggest a value to users who don't know what to insert into that field. They are displayed as greyed-out text:

$item_count <= 15 is a condition that only applies to playlists: https://github.com/RSS-Bridge/rss-bridge/blob/c4896c7791cd76dd9905ad8ee64cb3ca2b340d1f/bridges/YoutubeBridge.php#L226-L233

(first line is the condition for playlists)

I have a channel with 18 videos. When I use Rss Bridge it pulls a max of 14 videos :/

I also searched and looked through the wiki here and the threads that were created around Youtube, but still came up empty. Unfortunately it seems that the RSS Bridge basically pulls the same amount of posts that this URL would create -

https://www.youtube.com/feeds/videos.xml?channel_id=XXX

You are right, the bridge uses the YouTube feed by default (because it's fast), which contains exactly as many videos as YouTube includes with it. Unless your channel uploads more than 14 videos before you update your feed, this shouldn't be an issue. And even then you could just update your feed more often.

That being said, you can disable that behavior by commenting out line 210 (this may or may not work, depending on the user): https://github.com/RSS-Bridge/rss-bridge/blob/c4896c7791cd76dd9905ad8ee64cb3ca2b340d1f/bridges/YoutubeBridge.php#L208-L211

EDIT: You might have to insert some data to $url_feed because of later conditions. Anything works (i.e. $url_feed = 'bla';).

Frankly, I would even be willing to insert the last 4 videos in the feed manually, just as long as they stay in the feed and the feed then gets updated when new videos are posted.

Why would you do that?

A feed is supposed to notify users about new content. For YouTube that means new videos should appear in your feed. It is not designed to give you the full history of someones channel (nor is it supposed to).

I also saw this in the YoutubeBridge file:

"* WARNING: to parse big playlists (over ~90 videos), you need to edit simple_html_dom.php:

• change: define('MAX_FILE_SIZE', 600000);

• into: define('MAX_FILE_SIZE', 900000); (or more)"

This is meant for playlists only. Some playlists are too large to be downloaded in one go (if bigger than MAX_FILE_SIZE). Changing the parameter allows RSS-Bridge to download bigger files. I don't think this has ever been used (at least I'm not aware of it).

Any way to make RSS Bridge pull all the videos from a Youtube Channel (Username or User ID, all the same)?

This is not what RSS-Bridge is meant to do. This would also take a very long time for large channels (speaking about channels with thousands of videos). Are you sure that a feed is what you are searching for?

For this you should really take a look at the official YouTube API.

You could also take a look at the alternative YouTube bridge I developed some time ago (haven't been using it for some time, so it might be broken): https://github.com/RSS-Bridge/rss-bridge/wiki/Extended-Bridges

This bridge is actually using the YouTube API and requires an API Key, which you can acquire from Google for free (at the expense of your data of course).

And if not, any way I can edit the existing feed and manually add the videos that are not pulled, but then to make the feed Remember them while still updating when new videos are published on the channel?

The contents returned by RSS-Bridge are either XML (Atom and MRss), Plaintext or JSON. There is no build in solution to do what you want to do, but you can always change the content afterwards. Then again, why would you do that?

[Legalien]

Hey man, thanks for the comprehensive reply.

The reason I want to do this is basically for autosharing my videos on a few social networks (my own pages). To keep old videos alive. Right now I have it set up using that Youtube RSS Feed so that each time I publish a new video on Youtube it gets shared on a few social pages I have.

But when I found RSS Bridge I got the idea, if I could have it display all of my videos (currently 18, and I post 1-2 videos per month so it would grow in time) then I could set it up in the same way to re-share my old videos on my social pages. I would have it share 1 video every 4-5 days, because I don't want to post too much, then you get into spam territory and you put your Facebook/Twitter/etc page at risk, plus you risk annoying the people that are following you by posting the same stuff too much.

Another little problem I would have is that the method I use to repost my content uses (I believe) the Timestamps (or GUID I think it's also called) in order to identify the unique posts that haven't been shared yet. So right now with the Youtube feed, it shows 14 videos. If I set it up for re-sharing, once it posts all the 14 videos it will stop. But with RSS Bridge I hoped because the cached page gets deleted and a new page is created (as far as I understand it), the new page would have new/unique timestamps and whatnot, and then it would just continue to work. Of course, I would have to figure out how to have the cache for my Youtube feed be changed at a specific time (ex, every 30 days or 60 days).

How I do it for some websites I have is I create a Feedburner feed, I set things up for reposting my stuff using the feed, and when it goes through all the posts (all the timestamps, or GUID), then I go and Delete the Feedburner feed and I create a new one with the same content. This updates the timestamps, or GUID (I'm not sure what to call it) and then I reset it and it works again until it reposts all the posts. There's a bit of manual work involved, I have to check if it's finished and then do the process of deleting feedburner feed and creating a new one and restarting the process, but, it works :)

About the other API solution. Yeah, thanks, but I don't wanna use the Youtube API :)

I tried removing the line 210 like you said, and called a feed, and it showed a feed but again, only 14 videos are displayed. Now, I'm not sure if it showed me the cached version or if it created a new version (does it create a new version when I run it through RSS Bridge again? I think it does).

You said that there's a conditional for Playlists, $item_count <= 15. I created a Playlist with my 18 videos and changed this conditional to 50, and ran the playlist through RSS Bridge. It returned a feed with 10 videos. Is there any way to have a Playlist RSS Feed with all the 18 videos in my playlist?

[em92]

@Legalien

• provide the links with your bridge instance
• enable DEBUG mode (create file called DEBUG file in rss-bridge root directory). This will disable cache.

[Legalien]

Hey Em92,

Okay. I think this is the bridge instance - https://ninomaluri.com/rss-bridge-master/?action=display&bridge=Youtube&p=PL8xx00rOxoCOM5yS3Q2ivDT_oZ_6JsPLK&duration_min=&duration_max=&format=Atom

I created a file called DEBUG (capital letters, exactly as you wrote it, no .php at the end) in the root directory of my rss-bridge.

What now? :)

[em92]

Return back $item_count <= 15 and edit 10 value here to 50: https://github.com/RSS-Bridge/rss-bridge/blob/87d218296e513c08a1a3ff71062b46a9bdcb1d1a/bridges/YoutubeBridge.php#L130

That $item_count <= 15 is NOT applied for limiting video items. It applies to where to fetch data from.

[Legalien]

Broo! BOSS!!! That worked! Yes! 🥇

I'll increase that to 50, it will keep me going for a long time.

And you know, I don't know if you read all the stuff I wrote previously as to why I need this, but it just so happens that there's an option in Youtube to automatically add new videos to a Playlist when the video title includes a keyword (you choose the keyword). And it just so happens that in each of my video titles I include a specific keyword that I don't think anyone ever uses. So now what I'm hoping will happen is, Youtube will add all my new uploads to my playlist automatically, and Rss Bridge will update the feed automatically, and so my feed will always contain all my previous videos AND stay updated with new videos automatically. BOSS MODE! 👍

Now let me just ask you.

1) Can I/Should I delete the DEBUG file you told me to previously create? 2) In the YoutubeBridge file there's this - const CACHE_TIMEOUT = 10800; // 3h - Does this mean that my feed will be Updated every 3 hours? Is this where you control it? 3) And when the feed gets updated, what exactly happens? Does it REMOVE the cached feed and creates a brand new one? Or does it simply update the existing feed with new content (if it finds some).

Sorry for all the questions, I'm just trying to understand exactly how it works.

p.s I have shared RSS Bridge on my Twitter because it says in "How can I contribute" that sharing RSS Bridge with friends helps. I'm really grateful for your help.

[Legalien]

Okay, so I deleted the DEBUG file you told me to create. I think that's okay.

About my other 2 questions.

I would assume that for question 2 the feed does get updated every 3 hours by default. The reason I ask is because I tested this with a Twitter feed and it's not working for Twitter, the Twitter feed gets updated randomly whenever it wants, so I'm wondering if there's a different place where I can set how often a feed is updated.

And for question 3 I'd still like to know the answer. When a feed gets updated does it simply add new content on top of the old, or does it re-crawl a page and "creates a brand new feed" every time?

Thank you :)

[Legalien]

Okay I figured out the answer to question 2. I know how to make the feeds update as often as I'd like them to 👍 👍 👍

That just leaves question 3. I hope one of you guys can let me know real quick.

Thanks again :)

[logmanoriginal]

• Can I/Should I delete the DEBUG file you told me to previously create?

That file enables debug mode which is very useful in testing RSS-Bridge. In debug mode RSS-Bridge actually ignores caching and returns a new feed on each request (which might get you blacklisted on certain servers).

You should indeed delete that file to reduce bandwidth and keep your instance more responsive.

• In the YoutubeBridge file there's this - const CACHE_TIMEOUT = 10800; // 3h - Does this mean that my feed will be Updated every 3 hours? Is this where you control it?

When you request a feed for the first time, RSS-Bridge downloads data from YouTube, builds the feed and returns it to you. A copy of the data is placed on the cache.

For any subsequent request (no matter how often you do it), RSS-Bridge returns a copy of the previous data from cache.

After 3 hours RSS-Bridge removes the data from cache and downloads new data from YouTube.

If you want to adjust the timeout, there is a parameter available. See https://github.com/rss-bridge/rss-bridge/wiki/Custom-Configuration#custom_timeout

• And when the feed gets updated, what exactly happens? Does it REMOVE the cached feed and creates a brand new one? Or does it simply update the existing feed with new content (if it finds some).

Yes, if you update your feed after CACHE_TIMEOUT has elapsed, RSS-Bridge will delete the cached data and create a brand new one (by downloading from YouTube again). The items which were part of the previous feed, however, will have the same GUID, so you can detect if new items were added.

[Legalien]

Okay, that's awesome.

Just to be clear. When you say "if you update your feed after CACHE_TIMEOUT has elapsed," do you mean when I actually go to my RSS Bridge and run it again? Or do you mean, like, "when the feed gets updated either by me running it or by my server automatically updating it"?

Sorry, I just want to be clear.

And one last question, I promise. Is there an edit I can make so that when a feed is updated the GUID's change?

And I'll tell you why I need this. The method I use for autosharing my old Youtube videos to Twitter uses GUIDs. When I create a feed of, say 10 videos, once the 10 videos get shared it stops. But if I could have my Youtube feed update every 30 days for example (using CACHE_TIMEOUT) but have it change the GUIDs when it updates, that would be the best thing in the world! Seriously.

Is that doable?

[logmanoriginal]

Just to be clear. When you say "if you update your feed after CACHE_TIMEOUT has elapsed," do you mean when I actually go to my RSS Bridge and run it again? Or do you mean, like, "when the feed gets updated either by me running it or by my server automatically updating it"?

Sorry, I just want to be clear.

No problem.

RSS-Bridge only works if you actually go to RSS-Bridge and run it again. It will never do anything automatically.

And one last question, I promise. Is there an edit I can make so that when a feed is updated the GUID's change?

The GUID is based on the URL of the video. Since the URL points to the same location every time, the GUID doesn't change. This is actually how RSS and Atom feeds are designed.

Show solution anyway

This is highly discourages (because technically your feeds are no valid feeds anymore), but you could change how the GUID is generated (i.e. using the current timestamp instead of the URI). For the MRSS format this line needs to be changed for that (make sure to set `isPermalink="false"`!): https://github.com/RSS-Bridge/rss-bridge/blob/c4896c7791cd76dd9905ad8ee64cb3ca2b340d1f/formats/MrssFormat.php#L69

> And I'll tell you why I need this. The method I use for autosharing my old Youtube videos to Twitter uses GUIDs. When I create a feed of, say 10 videos, once the 10 videos get shared it stops. But if I could have my Youtube feed update every 30 days for example (using **CACHE_TIMEOUT**) but **have it change the GUIDs when it updates**, that would be the best thing in the world! Seriously. > > Is that doable? I'm not sure if it is a good idea to re-post old videos over and over again. That being said, the way you intend to use feeds is not how they are meant to be used. Maybe you could just generate a public feed for anyone interested to subscribe to? Or include the feed (or multiple types of feeds) on your personal website? These are just some wild ideas that come to mind. Anyway, RSS-Bridge only allows for a cache timeout of maximum 24 hours. If you need a slower update rate, I suggest you only request it every 30 days.

[Legalien]

No problem.

RSS-Bridge only works if you actually go to RSS-Bridge and run it again. It will never do anything automatically.

Um, not to disagree but, it does update the feeds automatically. No? :)

English is my 3rd language so maybe I'm getting lost in translation here.

The GUID is based on the URL of the video. Since the URL points to the same location every time, the GUID doesn't change. This is actually how RSS and Atom feeds are designed. Show solution anyway

This is highly discourages (because technically your feeds are no valid feeds anymore), but you could change how the GUID is generated (i.e. using the current timestamp instead of the URI). For the MRSS format this line needs to be changed for that (make sure to set isPermalink="false"!):

rss-bridge/formats/MrssFormat.php

Line 69 in c4896c7

{$itemUri}

Oh brother, that is awesome. I'll definitely try to implement this and see if it works.

And the timestamps would be different every time a feed is generated, which I think would work for what I need it just fine :)

I'm not sure if it is a good idea to re-post old videos over and over again. That being said, the way you intend to use feeds is not how they are meant to be used. Maybe you could just generate a public feed for anyone interested to subscribe to? Or include the feed (or multiple types of feeds) on your personal website? These are just some wild ideas that come to mind.

Anyway, RSS-Bridge only allows for a cache timeout of maximum 24 hours. If you need a slower update rate, I suggest you only request it every 30 days.

Oh reposting videos is not a problem. I mean, unless you repost too many and/or too fast, it's no problem. There are quite a few Wordpress plugins for example that allow you to repost your old blog posts on the various social networks. This keeps the old blog posts "alive" and brings traffic to them, which is good :)

Of course I like to take it easy, to not annoy people with too many posts or reposts, and to not endanger my social media - Twitter, Facebook, etc - profiles and pages. If you post too much, too fast, it's a risk and you could get blocked or banned. And that's valid even if you were to post manually. So I post sporadically.

Really? That's okay, I'll put it on my to-do list to go and refresh the feed once per month or when I need to. But so... is there a way to increase the maximum cache timeout and make it longer than 24 hours? I had to ask haha :)

[logmanoriginal]

Um, not to disagree but, it does update the feeds automatically. No? :)

No, it doesn't. You must request it.

But so... is there a way to increase the maximum cache timeout and make it longer than 24 hours? I had to ask haha :)

Search for purgeCache. For example: https://github.com/RSS-Bridge/rss-bridge/blob/c4896c7791cd76dd9905ad8ee64cb3ca2b340d1f/index.php#L223

[Legalien]

No, it doesn't. You must request it.

Wait a minute. In my RSS Bridge I've set up a Twitter feed. And I set it in IFTTT to receive an email every time the feed gets updated with new content. And it seems to work. The Twitter feed on my RSS Bridge gets "Automatically" updated at some point with new content (assuming there are new posts made on the Twitter page).

Are you talking in technical terms, like the feed doesn't get magically updated, but it must be requested either manually or by my server. But the server does it, so ultimately the update happens automatically.

What am I not understanding here?

I'm unclear on what mechanism is used for this update to happen. Is it cron? I don't know how it happens, but it seems to work.

I'm also unclear, and I would like to know, how to control how often my server/RSS bridge checks for new updates for any given bridge/feed. I thought it was CACHE_TIMEOUT =, but it doesn't seem to be it. The feeds get updated whenever they want. Can this be controlled?

Say I want my RSS Bridge to check for new updates on Twitter once every 24 hours. Can I set this somewhere?

Search for purgeCache. For example: https://github.com/RSS-Bridge/rss-bridge/blob/c4896c7791cd76dd9905ad8ee64cb3ca2b340d1f/index.php#L223

Thank you for that.

[Legalien]

I mean, unless you mean to tell me that RSS Bridge only creates Static feeds that never get updated unless the user goes and runs/re-creates the feed again manually through RSS Bridge in order to get a fresh/updated feed?

But what good would that do? No way. I'm 99% sure that feeds do get updated on their own with new content. If I create a feed for a Twitter profile, when that Twitter profile posts new tweets, the RSS Bridge Twitter feed I created will display the new tweets on it ...eventually...

That ...eventually... though, I have no idea when that is, for any bridge. And I looked closely through some of the bridges and I looked through the Wiki and I've been trying to understand what defines how often new/fresh/updated data is pulled for any of the bridges.

[logmanoriginal]

Can I assume these questions were answered in #892?

[Legalien]

Hey, sorry for the delay. Yes, indeed, all good, thank you :)

I will close this one.

[squromiv]

You could also take a look at the alternative YouTube bridge I developed some time ago (haven't been using it for some time, so it might be broken): https://github.com/RSS-Bridge/rss-bridge/wiki/Extended-Bridges

Sorry for raising the old topic, but is it possible now somehow to get all videos from channel into RSS, using RSS-Bridge? With or without youtube API.