Add web-scraping alternative to already retrievable data of YouTube Data API v3

[Benjamin-Loison]

As mentioned in #87, being able to retrieve from YouTube operational API YouTube Data API v3 already retrievable data isn't a priority. However in the future it may be interesting, as an alternative to the no-key service for people having exceeded their quota (especially if they have their own instance of the YouTube operational API).

[Benjamin-Loison]

As current approach is to have one YouTube UI webpage scraped by YouTube operational API endpoint with a given part, maybe merging playlists and playlistItems makes sense.

[Benjamin-Loison]

Concerning videos tab, someone did a proposition.

[zakrian07]

@Benjamin-Loison yes this is me discuss months ago. Can we complete this let me know if I can help you to add this, as I'm using self deploy your source code and I have scrape lot of data and my project is viral I need use proxies and get this done.

[Benjamin-Loison]

@zakrian07

You have not answered my question:

I do not understand your feature request. It seems you want the YouTube operational API to be able to parse https://www.youtube.com/channel/CHANNEL_ID/videos. YouTube Data API v3 meets your needs, no?

[zakrian07]

Yes target is to get a channel videos. If you can add videos part it would be great as you already scrape clips and all other tabs info. Scrapping videos is same, like 90 percent, like clips very minor work will be for you to add videos as part of channel API. Thoughts?

On Sat, Dec 23, 2023 at 3:36 PM Benjamin Loison @.***> wrote:

@zakrian07 https://github.com/zakrian07

You have not answered my question:

I do not understand your feature request. It seems you want the YouTube operational API to be able to parse https://www.youtube.com/channel/CHANNEL_ID/videos. YouTube Data API v3 meets your needs https://stackoverflow.com/a/74579030, no?

— Reply to this email directly, view it on GitHub https://github.com/Benjamin-Loison/YouTube-operational-API/issues/91#issuecomment-1868265008, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPI5XL56MAOEYJKROEJJSLYK2X2TAVCNFSM6AAAAAASQFGQW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGI3DKMBQHA . You are receiving this because you were mentioned.Message ID: @.***>

[Benjamin-Loison]

@zakrian07 In my opinion you still have not answered the main part of my question:

It seems you want the YouTube operational API to be able to parse https://www.youtube.com/channel/CHANNEL_ID/videos. YouTube Data API v3 meets your needs, no?

Hence I am rephrasing it again:

What makes this method: https://stackoverflow.com/a/74579030 using YouTube Data API v3 not solving your problem?

[zakrian07]

Yes, it works! As you know the API have a daily limit of 10 k requests. My projects is viral I want prefer scrape methodology same like you did for clips. I was using your API endpoint sometimes it blocks my IP and it causes downtime on my app.

On Sat, Dec 23, 2023 at 3:57 PM Benjamin Loison @.***> wrote:

@zakrian07 https://github.com/zakrian07 In my opinion you still have not answered the main part of my question:

It seems you want the YouTube operational API to be able to parse https://www.youtube.com/channel/CHANNEL_ID/videos. YouTube Data API v3 meets your needs https://stackoverflow.com/a/74579030, no?

Hence I am rephrasing it again:

What makes this method: https://stackoverflow.com/a/74579030 using YouTube Data API v3 not solving your problem?

— Reply to this email directly, view it on GitHub https://github.com/Benjamin-Loison/YouTube-operational-API/issues/91#issuecomment-1868268517, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPI5XIJREGTSQ4NYV6UKVDYK22JPAVCNFSM6AAAAAASQFGQW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGI3DQNJRG4 . You are receiving this because you were mentioned.Message ID: @.***>

[Benjamin-Loison]

@zakrian07 By your API endpoint do you mean the official instance no-key endpoint or any web-scraping endpoint? I.e. does the URL limiting you starts with https://yt.lemnoslife.com/noKey/ or not? You can directly share an example of limiting URL to make sure.

Currently I guess that

your API endpoint sometimes it blocks my IP

refer to this issue.

If you are limited by another endpoint than the official instance no-key endpoint (https://yt.lemnoslife.com/noKey/), then please try it.

[zakrian07]

Thanks for your explanation. I can share log with you and my IP as well. Really appreciate your help.

On Sat, Dec 23, 2023 at 4:16 PM Benjamin Loison @.***> wrote:

@zakrian07 https://github.com/zakrian07 By your API endpoint do you mean the official instance https://yt.lemnoslife.com/ no-key endpoint or any web-scraping endpoint? I.e. does the URL limiting you starts with https://yt.lemnoslife.com/noKey/ or not? You can directly share an example of limiting URL to make sure.

Currently I guess that

your API endpoint sometimes it blocks my IP

refer to this issue https://github.com/Benjamin-Loison/YouTube-operational-API/issues/11.

If you are limited by another endpoint than the no-key endpoint, then please try it.

— Reply to this email directly, view it on GitHub https://github.com/Benjamin-Loison/YouTube-operational-API/issues/91#issuecomment-1868271983, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPI5XIG5JHYV47KWFZRBBTYK24QPAVCNFSM6AAAAAASQFGQW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGI3TCOJYGM . You are receiving this because you were mentioned.Message ID: @.***>

[Benjamin-Loison]

@zakrian07 As you have not answered any of my questions, share (if necessary, privately on Matrix or Discord) your logs and IP then.

[zakrian07]

Yes https://yt.lemnoslife.com/noKey/ https://yt.lemnoslife.com/noKey/videos?part=snippet&id=VIDEO_ID this seems to be quota limited and it blocks my IP and this is why my app gets down.

Based on Channels: list: https://yt.lemnoslife.com/channels?part=status,upcomingEvents,shorts,community,channels,about,approval,playlists,snippet,membership&cId=C_ID&id=CHANNEL_ID&handle=HANDLE&forUsername=USERNAME&order=viewCount(&pageToken=PAGE_TOKEN)

I'm wondering if you can add videos in part same as clips. We can scrape data without using https://yt.lemnoslife.com/noKey/ https://yt.lemnoslife.com/noKey/videos?part=snippet&id=VIDEO_ID or official YouTube API. It will be more convient. Let me know if I answered your questions.

On Sat, Dec 23, 2023 at 4:39 PM Muhammad Umer @.***> wrote:

Thanks for your explanation. I can share log with you and my ip as well Really appreciate your help

On Sat, Dec 23, 2023 at 4:16 PM Benjamin Loison @.***> wrote:

@zakrian07 https://github.com/zakrian07 By your API endpoint do you mean the official instance https://yt.lemnoslife.com/ no-key endpoint or any web-scraping endpoint? I.e. does the URL limiting you starts with https://yt.lemnoslife.com/noKey/ or not? You can directly share an example of limiting URL to make sure.

Currently I guess that

your API endpoint sometimes it blocks my IP

refer to this issue https://github.com/Benjamin-Loison/YouTube-operational-API/issues/11.

If you are limited by another endpoint than the no-key endpoint, then please try it.

— Reply to this email directly, view it on GitHub https://github.com/Benjamin-Loison/YouTube-operational-API/issues/91#issuecomment-1868271983, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPI5XIG5JHYV47KWFZRBBTYK24QPAVCNFSM6AAAAAASQFGQW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGI3TCOJYGM . You are receiving this because you were mentioned.Message ID: @.***>

[Benjamin-Loison]

@zakrian07

https://yt.lemnoslife.com/noKey/ is currently powered by apache2 on a free Oracle Ubuntu VPS having its iptables cleared (I say so for possible reproductibility), hence as far as I know there no IP blocking at all. Do you receive any answer from the official instance HTTPS server when you consider being IP blocked? If so, please share it (privately if necessary). (1)

I monitor every minute the correct behavior of the official instance no-key endpoint:

• here are its logs
• here is its source code

From the logs it does not seem that they were major downtime recently. Could you precise up to the minute when your app was down the last time? (2)

I clearly understand your request concerning having a web-scraping alternative to the official instance no-key endpoint but from the pieces of information I have on my side I currently do not notice any major issue with the official instance no-key endpoint, so please answer my questions (1) and (2) to let me know if I am wrong. Because without me being convinced of any major issue with the official instance no-key endpoint there is no strong need for an alternative.

[zakrian07]

this is last active commit on main branch I have build docker image as well ee7ee425bdd2dca5432d9131a46a2834eb8b1c31.

[zakrian07]

I have tried to explain as per my knowledge looks like I'm not able to express what I need actually https://yt.lemnoslife.com/channels?part=about&id=UCeVMnSShP_Iviwkknt83cww

[Benjamin-Loison]

It seems that your issue is about updating your YouTube operational API relying on Docker. I am not keen on that topic but assuming you started your instance with docker-compose up, then stopping it with Ctrl + C and running docker image rm youtube-operational-api -f, finally running again docker-compose up and continuing with the new image (see below) updated my instance on my side, hence theoretically solved your issue:

ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y

[zakrian07]

I will try to delete all volumes and delete image and fresh docker build. Will let you know then.

On Wed, Dec 27, 2023 at 2:10 AM Benjamin Loison @.***> wrote:

It seems that your issue is about updating your YouTube operational API relying on Docker. I am not keen on that topic but assuming you started your instance with docker-compose up https://github.com/Benjamin-Loison/YouTube-operational-API/blob/ee7ee425bdd2dca5432d9131a46a2834eb8b1c31/README.md?plain=1#L93, then stopping it with Ctrl + C and running docker image rm youtube-operational-api -f, finally running again docker-compose up and continuing with the new image (see below) updated my instance on my side, hence theoretically solved your issue:

ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y

— Reply to this email directly, view it on GitHub https://github.com/Benjamin-Loison/YouTube-operational-API/issues/91#issuecomment-1869773755, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPI5XP442W7O45R5L4GG6LYLM4MNAVCNFSM6AAAAAASQFGQW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRZG43TGNZVGU . You are receiving this because you were mentioned.Message ID: @.***>

[zakrian07]

https://youtubedata.alaskaenterprises.com/channels?part=about&id=UCeVMnSShP_Iviwkknt83cww Thank you so much for the correction the issue has been resolved I was just deleting images, but deleting the volume and fresh image pull fixed the issue.

[Benjamin-Loison]

https://discord.com/channels/933841502155706418/933841503103627316/1249913353661579325