Please provide an up-to-date example of PubSubhubbub

sluongng commented 6 years ago

According to https://developers.google.com/youtube/v3/guides/push_notifications you guys do support it. But most of the code I can find around github or tutorial online are widely outdated.

Plus the configuration is a mix-bag between setting up environment for callback support to working with the web interface on https://pubsubhubbub.appspot.com/subscribe

Many library also claim that their implementation are not compatitble with latest specification.

Please help provide a working tutorial/example code. Preferable in GO/Python or NodeJS. Also if there is a reliable way of testing this code, instead of having to wait around for a channel to upload a video, it would be very nice.

Thanks!

MrBoombastic commented 5 years ago

Nothing changed after year. That's sad.

whydinkov commented 4 years ago

@sluongng, @MrBoombastic Hi guys I had the same problem for the past several weeks and I guess I was able to fix it locally, so wanted to share my approach. To begin with I also share the same frustration towards the lack of documentation.

I tried two approaches - NodeJS (via koa2) didn't worked (wasn't able to read the body correctly) and pythonic very simple quite straightforwad approach, which worked (at least locally).

So the steps are:

1. Create a new flask app

pip install flask
touch app.py

app.py:

from flask import Flask
from flask import request

app = Flask(__name__)

# irrelevant to the question method, just for testing purposes
@app.route("/", methods=['GET', 'POST'])
def ping():
    return 'pong'

@app.route("/feed", methods=['GET', 'POST'])
def feed():
    challenge = request.args.get('hub.challenge')

    if (challenge):
        return challenge

    print(request.data)  # binary literal with xml payload

    return '', 204

flask run

2. Create a ngrok tunnel (https://ngrok.com/docs)

For me it looked something like that

3. Make a subscription in https://pubsubhubbub.appspot.com/subscribe

https://www.youtube.com/xml/feeds/videos.xml?channel_id={{CHANNEL_ID}} where {{CHANNEL_ID}} is YouTube channel id

4. Test subscription

5. Analyze report

if you're returning hub.challenge then server authorizes you for 5 days as it sends another checks from time to time
if your service return 204 for POST then youtube server will mark it as successful receive

6. Debugging

as mentioned above you can always subscribe to a big channel (e.g. CNN) and wait a day or two to generate 5-10 requests and debug it, but it is quite slow. In my case I just used a simple quite short (5-10 seconds) video and uploaded it multiple times to my personal channel. It is not simultaneous, yet way faster then waiting 6-12 hours for somebody. It was 5-10 minutes while video has been processed, etc.

7. Further improvements

subscription part are simple POST requests so this can be automized as well (for multiple channels)
there are some node-js implementation for PSHB subscriptions, but they are not documented as well

MrBoombastic commented 4 years ago

Good job, but PSHB should be simple. Now scraping XML for channel is faster and easier than this. Anyway, thank you!

whydinkov commented 4 years ago

@MrBoombastic what do you mean by

scraping xml for channel

MrBoombastic commented 4 years ago

Each channel has xml file with 10 latest videos.

whydinkov commented 4 years ago

Gotcha! You mean this RSS feed https://www.youtube.com/feeds/videos.xml?channel_id= which is still limiting, but yeah you're right - it's easier than figuring out how a 15 years old technology works without any decent documentation/example.

shan1iu commented 4 years ago

@yoandinkov I tried to do this in Express.js but there is nothing return from the req or res object.

whydinkov commented 4 years ago

@shan1iu can you share your project/give more information what have you done and what haven't worked exactly. Also do you get the right results from step 4 (testing the subscription)?

mmkhitaryan commented 4 years ago

What limits does PubSubhubbub have? How many channels can I listen to using it?

Ulusamay commented 4 years ago

i creating api in python to deploy in heroku and use the pubsubhubub

malaire commented 4 years ago

@mmkhitaryan I'd like to know that also - I asked a question at StackOverflow but no answer yet: https://stackoverflow.com/questions/63726197/is-there-usage-limit-for-youtube-push-notifications

AXVin commented 4 years ago

Hey there! I am currently having problems determining whether a pubsubhubbub notification that i recieved is a new upload or just a title/description change. Can anybody help me with that? I tried comparing the published and updated value but both are never same 😖

whydinkov commented 4 years ago

@malaire - what's your count that you're trying it with?

whydinkov commented 4 years ago

@AXVin - actually it's hard to say, I guess, I do use following approach:

if a video's release date was less than an hour in the time the event appears I treat it as "new video" and store it in my database. in my database, I have a unique constraint on video_id to avoid duplicates

malaire commented 4 years ago

@yoandinkov I have been testing with just 2 subscriptions for now, but YouTube push notifications are not working at all - I have received ZERO notifications while both channels have uploaded videos. I'm now waiting for YouTube to fix this bug before I start testing with more subscriptions.

(Subscriber Diagnostics confirms that PubSubHubbub has received nothing from YouTube, so it's YouTube bug, not PubSubHubbub bug.)

AXVin commented 4 years ago

@malaire Do you have the GET method handled for the callback? You must return the hub.challenge query param inside that otherwise it will not be verified correctly and you will never recieve any updates ever. (Because, i do get the notifications)

malaire commented 4 years ago

@AXVin My code is working correctly. Subscriber Diagnostics says "State: verified", and no errors of any kind, but "Content received: n/a" and "Content delivered: n/a".

And manually checking the Topic URL confirms that uploads are happening so there should be push notifications.

whydinkov commented 4 years ago

@malaire I can confirm that this service works correctly for me. My suggestion is either upload your own video in your own channel (so you know exactly what and when is being uploaded) or use some big channel (as CNN for example).

Another suggestion is to refer to my previous post (from 02.12.2019) especially point 5. What about other properties (except State) do you have timestamps for all others as well (like subscription, etc.)?

malaire commented 4 years ago

@yoandinkov I did test with my own channel where I uploaded video - no notification. And timestamps were correct also.

I'm not interested in big channels but in small channels.

I made issue here (without timestamps as I didn't remember them anymore, but I can add those once I finish currently ongoing 10-day test without any notifications): https://issuetracker.google.com/issues/169067950

JMTK commented 4 years ago

I also tried doing this for live broadcasts. I can confirm the live broadcast shows up in the XML videos feed but the pubsubhubbub just doesn't seem to work. I did a similar flow for Twitch webhooks and it worked fine so I'm not sure where the disconnect is. There's seemingly no way to debug things programmatically so it just "doesn't work" from my end.

hanzlahabib commented 3 years ago

@AXVin - actually it's hard to say, I guess, I do use the following approach:

if a video's release date was less than an hour in the time the event appears I treat it as "new video" and store it in my database. in my database, I have a unique constraint on video_id to avoid duplicates

I created flask API using your snippet it displays get request on terminal and black page on pubsubhubbub

127.0.0.1 - - [21/Nov/2020 10:46:19] "[37mGET /feed?hub.topic=https://www.youtube.com/xml/feeds/videos.xml%3Fchannel_id%3DUCqD5M4_-cpDOSGiKq3iyGHg&hub.challenge=10170485706286032479&hub.mode=subscribe&hub.lease_seconds=432000 HTTP/1.1[0m" 204 -

and when i try to diagnose it displays these details

Any idea why this is soo?

pham commented 3 years ago

Here's how I got it to work with NodeJS:

const express = require('express');
const bodyParser = require('body-parser');
const app = express();
require('body-parser-xml')(bodyParser);

app.get('/test', ({ query: { 'hub.challenge': challenge } }, res) => {
  console.log(challenge);
  res.status(200).end(challenge)
});

app.post(`/test`, bodyParser.xml(), ({ body: {feed} }, res) => {
  console.log(feed);
  res.status(204).end();
});

app.listen(1234);

PSHB sends a GET method with a few queries, the one you're interested in is hub.challenge. Response back with that code (sub/unsub).

POST method is used to send the XML. I use body-parser-xml but you can use anything for this.

Menelion commented 3 years ago

Hi guys, thanks for the solution! Sorry, this question might be stupid, so please bear with me — I'm really new to YouTube APIs.
As far as I can see, you get the activities per channel. But what I'm really missing is the email notifications about new videos I'm subscribed to. I mean, before August 13th 2020 you could adjust it so that when a new video is uploaded to any channel you're subscribed to, you get an email notification.
Could PSHB help me to accomplish that task? Thanks.

MrBoombastic commented 3 years ago

@Menelion In theory - yes, you can, if you have your own mailing service etc. In practice - PSHB is really piece of garbage to test and handle. As I said about year ago in this thread, it's easier to use XML endpoint than trying to set up PubSub. Maybe you will have more luck?

Menelion commented 3 years ago

@MrBoombastic Thanks for your reply! However, I don't see any way of getting the single feed for all of my subscriptions in XML, either. See this, for example. Any ideas?

MrBoombastic commented 3 years ago

@Menelion As somebody said in the answer in your linked question, you can manually export all your subscribed channels here: https://www.youtube.com/subscription_manager, convert downloaded XML to something easy to work with like JSON and ask Youtube's XML endpoint about each of channels listed in your JSON file. Yeah, if you have, for example, 200 channels subscribed, you have to make 200 requests to YouTube. But don't worry - it's not very limited, so you shouldn't expect Captchas or other blocks.

andrei-polukhin commented 3 years ago

Hey there! I am currently having problems determining whether a pubsubhubbub notification that i recieved is a new upload or just a title/description change. Can anybody help me with that? I tried comparing the published and updated value but both are never same 😖

@AXVin It seems this feature of poorly documented, so I created an issue on their official issue tracker.

Edit: Google has already given this issue to an assignee.

zainulhassan815 commented 2 years ago

I made it work with Firebase Cloud Functions. Here is the github repo you can refer to: https://github.com/zainulhassan815/youtube-pubsub-firebase

The cloud function verifies and also handles incoming feed. When it recieves data, it gets the updated video id, make a request to youtube api to get updated data and then stores it in firestore.

CarlosMunozT commented 2 years ago

@AXVin - actually it's hard to say, I guess, I do use the following approach: if a video's release date was less than an hour in the time the event appears I treat it as "new video" and store it in my database. in my database, I have a unique constraint on video_id to avoid duplicates

I created flask API using your snippet it displays get request on terminal and black page on pubsubhubbub
127.0.0.1 - - [21/Nov/2020 10:46:19] "�[37mGET /feed?hub.topic=https://www.youtube.com/xml/feeds/videos.xml%3Fchannel_id%3DUCqD5M4_-cpDOSGiKq3iyGHg&hub.challenge=10170485706286032479&hub.mode=subscribe&hub.lease_seconds=432000 HTTP/1.1�[0m" 204 -
and when i try to diagnose it displays these details

Any idea why this is soo?

@hanzlahabib can you fix your problem? I have the same problem, don't receive the notification (post) from pubsub

tree1891 commented 1 year ago

I have save problem. Is Topic url https://www.youtube.com/feeds/videos.xml or https://www.youtube.com/xml/feeds/videos.xml?

wsysuper commented 1 year ago

Is Topic url (1) https://www.youtube.com/feeds/videos.xml or (2) https://www.youtube.com/xml/feeds/videos.xml?

I have the same question. I can only find the url (1) from a youtube channel page src, and it seems work for many rss feed applicaitions; while the youtube data api page claims only the url (2), but it doesn't seem to work with any public rss feed applications.

Can anyone give some explainations or advices?

tree1891 commented 1 year ago

@wsysuper Topic url is https://www.youtube.com/xml/feeds/videos.xml

loademon commented 1 year ago

@tree1891 did you solve the problem

tree1891 commented 1 year ago

@loademon yes, it does work well.

loademon commented 1 year ago

@loademon yes, it does work well.

how, can u explain

loademon commented 1 year ago

The problem was solved when I ran it with Ngrok account. (Mismatch error)

I logged in with my account key, not without an account.

NeloBlivion commented 1 year ago

Has anyone had any luck with other pubsubhubbub endpoints? We know https://www.youtube.com/xml/feeds/videos.xml is the only one documented, however there's also https://www.youtube.com/xml/feeds/playlists.xml and https://www.youtube.com/xml/feeds/channels.xml which both return supposedly valid feeds, just not sure on the params they require...

kevincox commented 7 months ago

Hi everyone. I saw this issue and figured I would help out. YouTube doesn't really support PubSubHubbub or WebSub. It supports a protocol that is sort of based on these protocols. It also has some major issues. For example the push doesn't contain all of the information from the feed as would be required by the protocol, but instead basically just the video IDs and some fake information. Then you need to fetch the information yourself. This can be done by the YouTube API (if you have access) or by manually fetching the feed. But if you fetch the feed be careful as the feed seems to be cached even when it shouldn't be, so you will frequently get a notification via WebSub long before the item is actually available in the feed. So you basically need to fall back to polling anyways.

I wrote more detail about this a while ago. I just gave it a re-read and it is still accurate. (I run a feed-reader service which subscribes to YouTube feeds via their WebSub-like API.) Between this doc and the WebSub specification there should be enough information to subscribe via "YouTube flavoured PubSubHubbub".

https://kevincox.ca/2021/12/16/youtube-websub/

I'm happy to answer any questions not answered in either the spec or article.

lionderful commented 6 months ago

I start to touch this issue a few days ago. The most difficult thing is "there's seemingly no way to debug things programmatically". I can receive the feeds whether State is verified, or unverified, using callback.php. This code, $feed=file_get_contents("php://input") can work, we can log those feeds. But how to parse this xml feed in "php://input"? This is a challenge for php becuase of XXE（XML External Entity attack）. after php 8.0, libxml_disable_entity_loader, This function has been DEPRECATED , DOMDocument, SimpleXML, XMLReader and xml_parse() could not load 'php://input' now. $xmlDoc->loadXML($file_get_contents('php://input')), this will not work as expect . how to parse this xml feed? Any one can help me ? -------update---------------- After hard work with github for a few days, my code seems works with php8.1, although I could not fully understand. libxml_disable_entity_loader (false); $xmlfile = file_get_contents('php://input'); $xmlDoc = new DOMDocument(); if ($xmlDoc->loadXML($xmlfile, LIBXML_NOENT | LIBXML_DTDLOAD)) { $entrys = $xmlDoc->getElementsByTagName('entry'); }

meyvan commented 1 month ago

following

youtube / api-samples