Benjamin-Loison / YouTube-operational-API

YouTube operational API works when YouTube Data API v3 fails.
397 stars 50 forks source link

Get precise creation date of community post #295

Open Benjamin-Loison opened 3 months ago

Benjamin-Loison commented 3 months ago

As requested on Discord.

https://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo:

image

Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo does not seem base64 encoded.

There is no tooltip and the source code of the shown date does not seem interesting:

<yt-formatted-string id="published-time-text" link-inherit-color="" class="style-scope ytd-backstage-post-renderer" has-link-only_=""><a class="yt-simple-endpoint style-scope yt-formatted-string" spellcheck="false" href="/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" dir="auto">1 month ago</a></yt-formatted-string>

I have an idea that I have not investigated yet that is investigate everything encoded notably in Protobuf, notably for continuation tokens.

The whole community post code: ```html
Benjamin
Benjamin 1 month ago
a9qxjTfH
```

AIdro_k6E7UTsF655DeERRis1BYvl8uSsF973x_Fdovm3dYObcg6BnyGOi5TiYl_QS1VWQCHTQ and AIdro_k6E7UTsF655DeERRis1BYvl8uSsF973x are not base 64 encoded otherwise I manually checked all this code.

Have to invesitgate both the precise community post webpage and the community tab webpage.

What difference with above URL and https://www.youtube.com/channel/UC2ChxHEZCmK5Nj4JB649iKA/community?lb=Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo one?

curl -I 'https://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo'
Output: ``` HTTP/2 200 content-type: text/html; charset=utf-8 x-content-type-options: nosniff cache-control: no-cache, no-store, max-age=0, must-revalidate pragma: no-cache expires: Mon, 01 Jan 1990 00:00:00 GMT date: Sun, 04 Aug 2024 11:47:19 GMT content-length: 507365 strict-transport-security: max-age=31536000 x-frame-options: SAMEORIGIN permissions-policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factors=*, ch-ua-platform=*, ch-ua-platform-version=* cross-origin-opener-policy: same-origin-allow-popups; report-to="youtube_main" origin-trial: AmhMBR6zCLzDDxpW+HfpP67BqwIknWnyMOXOQGfzYswFmJe+fgaI6XZgAzcxOrzNtP7hEDsOo1jdjFnVr2IdxQ4AAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTc1ODA2NzE5OSwiaXNTdWJkb21haW4iOnRydWV9 report-to: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]} p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=fr for more info." server: ESF x-xss-protection: 0 set-cookie: YSC=86fipQsYId0; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none set-cookie: __Secure-YEC=CgtaX1lfQk5mSUlTSSjH1L21BjIiCgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgQQ%3D%3D; Domain=.youtube.com; Expires=Wed, 03-Sep-2025 11:47:18 GMT; Path=/; Secure; HttpOnly; SameSite=lax set-cookie: VISITOR_PRIVACY_METADATA=CgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgQQ%3D%3D; Domain=.youtube.com; Expires=Wed, 03-Sep-2025 11:47:19 GMT; Path=/; Secure; HttpOnly; SameSite=none set-cookie: VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 08-Nov-2021 11:47:19 GMT; Path=/; Secure; HttpOnly; SameSite=none alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 ```
curl -I 'https://www.youtube.com/channel/UC2ChxHEZCmK5Nj4JB649iKA/community?lb=Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo'
Output: ``` HTTP/2 200 content-type: text/html; charset=utf-8 x-content-type-options: nosniff cache-control: no-cache, no-store, max-age=0, must-revalidate pragma: no-cache expires: Mon, 01 Jan 1990 00:00:00 GMT date: Sun, 04 Aug 2024 11:47:50 GMT content-length: 519755 strict-transport-security: max-age=31536000 x-frame-options: SAMEORIGIN permissions-policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factors=*, ch-ua-platform=*, ch-ua-platform-version=* origin-trial: AmhMBR6zCLzDDxpW+HfpP67BqwIknWnyMOXOQGfzYswFmJe+fgaI6XZgAzcxOrzNtP7hEDsOo1jdjFnVr2IdxQ4AAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTc1ODA2NzE5OSwiaXNTdWJkb21haW4iOnRydWV9 cross-origin-opener-policy: same-origin-allow-popups; report-to="youtube_main" report-to: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]} p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=fr for more info." server: ESF x-xss-protection: 0 set-cookie: YSC=FenNnZMWRv4; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none set-cookie: __Secure-YEC=Cgs4cHRrREthTVQybyjm1L21BjIiCgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgZQ%3D%3D; Domain=.youtube.com; Expires=Wed, 03-Sep-2025 11:47:49 GMT; Path=/; Secure; HttpOnly; SameSite=lax set-cookie: VISITOR_PRIVACY_METADATA=CgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgZQ%3D%3D; Domain=.youtube.com; Expires=Wed, 03-Sep-2025 11:47:50 GMT; Path=/; Secure; HttpOnly; SameSite=none set-cookie: VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 08-Nov-2021 11:47:50 GMT; Path=/; Secure; HttpOnly; SameSite=none alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 ```

As there is the same UI I guess that it is done the same way.

Maybe could also leverage comments but as there is no guarantee to have any maybe it is not leverageable.

Maybe there is some webpage meta-data letting us know that the webpage has not changed since a given date but I doubt so due to UI changes.

Benjamin-Loison commented 3 months ago
curl -s 'https://www.youtube.com/channel/UC2ChxHEZCmK5Nj4JB649iKA/community?lb=Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo' > a
getJSONPathFromKey a | grep 'a9qxjTfH'
208 /contents/twoColumnBrowseResultsRenderer/tabs/0/tabRenderer/content/sectionListRenderer/contents/0/itemSectionRenderer/contents/0/backstagePostThreadRenderer/post/backstagePostRenderer/contentText/runs/0/text a9qxjTfH
 48 /microformat/microformatDataRenderer/description a9qxjTfH
jq .microformat a
Output: ```json { "urlCanonical": "https://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo", "title": "Post de Benjamin", "description": "a9qxjTfH", "thumbnail": { "thumbnails": [ { "url": "https://yt3.googleusercontent.com/ytc/AIdro_k6E7UTsF655DeERRis1BYvl8uSsF973x_Fdovm3dYObcg6BnyGOi5TiYl_QS1VWQCHTQ=s200-c-k-c0x00ffffff-no-rj?days_since_epoch=19939", "width": 200, "height": 200 } ] }, "siteName": "YouTube", "appName": "YouTube", "androidPackage": "com.google.android.youtube", "iosAppStoreId": "544007664", "iosAppArguments": "https://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo", "ogType": "yt-fb-app:channel", "urlApplinksWeb": "https://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo?feature=applinks", "urlApplinksIos": "vnd.youtube://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo?feature=applinks", "urlApplinksAndroid": "vnd.youtube://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo?feature=applinks", "urlTwitterIos": "vnd.youtube://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo?feature=twitter-deep-link", "urlTwitterAndroid": "vnd.youtube://www.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo?feature=twitter-deep-link", "twitterCardType": "summary", "twitterSiteHandle": "@YouTube", "schemaDotOrgType": "http://schema.org/http://schema.org/YoutubeChannelV2", "noindex": false, "unlisted": false, "familySafe": true, "availableCountries": [ "CV", "ID", "MX", "SC", "MP", "MS", "SV", "YE", "GD", "UG", "IR", "BJ", "GH", "IM", "FM", "GB", "SJ", "FI", "JP", "CF", "ME", "MG", "LA", "BE", "ZW", "NU", "MH", "KR", "MZ", "CW", "GN", "VE", "BY", "CY", "MU", "BF", "CR", "MF", "FK", "JO", "MR", "LU", "SS", "PL", "ES", "ET", "DO", "TG", "CD", "VN", "AI", "DZ", "BV", "CX", "CH", "IS", "EH", "NF", "RS", "SD", "SN", "TJ", "SL", "YT", "AT", "CC", "SM", "PG", "EG", "CA", "KH", "UM", "MN", "CN", "HR", "EC", "OM", "PM", "CL", "IN", "MY", "TN", "TM", "JM", "NO", "PN", "GY", "MO", "VU", "ER", "GP", "QA", "AW", "IT", "AR", "LC", "HT", "UY", "GL", "NZ", "LR", "FR", "VA", "VI", "TZ", "PW", "FO", "BT", "GW", "TK", "KI", "NR", "PS", "ZA", "GA", "BR", "NL", "LT", "NI", "BQ", "DM", "MK", "GS", "BW", "TW", "LY", "TC", "NE", "SY", "SI", "PE", "GQ", "MD", "MQ", "GE", "PY", "IQ", "GM", "UZ", "SB", "BG", "PR", "SH", "CM", "MW", "ST", "PA", "VC", "BH", "BS", "TV", "DE", "NC", "AL", "GF", "JE", "DJ", "RU", "CZ", "AE", "NG", "BD", "BM", "BA", "GT", "PT", "BN", "NP", "VG", "SZ", "AU", "TO", "SA", "CK", "BL", "TF", "TH", "MA", "AX", "CI", "HU", "CG", "KW", "FJ", "KZ", "BZ", "GG", "IE", "IL", "IO", "KG", "SG", "TT", "HM", "BO", "SX", "LI", "ZM", "PK", "GU", "TR", "AO", "SE", "LK", "RW", "TD", "NA", "PH", "PF", "TL", "ML", "EE", "HN", "LB", "CU", "AF", "GI", "BB", "RO", "GR", "BI", "MT", "UA", "RE", "AM", "AS", "SR", "KE", "AZ", "LS", "CO", "MV", "HK", "KN", "US", "AQ", "AD", "MC", "SK", "DK", "KM", "KY", "LV", "SO", "MM", "AG", "KP", "WF", "WS" ], "linkAlternates": [ { "hrefUrl": "https://m.youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" }, { "hrefUrl": "android-app://com.google.android.youtube/http/youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" }, { "hrefUrl": "ios-app://544007664/http/youtube.com/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" } ] } ```

In theory can lower bound by the channel creation date in About tab.

I believe that days_since_epoch=19939 is channel thumbnail related but let us check.

https://www.epochconverter.com/seconds-days-since-y0#d1970

So it correponds to today...

import datetime

print(datetime.datetime(1970, 1, 1, 0, 0) + datetime.timedelta(19939))
jq '.contents.twoColumnBrowseResultsRenderer.tabs[0].tabRenderer.content.sectionListRenderer.contents[0].itemSectionRenderer.contents[0]' a
Output: ```json { "backstagePostThreadRenderer": { "post": { "backstagePostRenderer": { "postId": "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo", "authorText": { "runs": [ { "text": "Benjamin", "navigationEndpoint": { "clickTrackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "commandMetadata": { "webCommandMetadata": { "url": "/@Benjamin-xq", "webPageType": "WEB_PAGE_TYPE_CHANNEL", "rootVe": 3611, "apiUrl": "/youtubei/v1/browse" } }, "browseEndpoint": { "browseId": "UC2ChxHEZCmK5Nj4JB649iKA", "canonicalBaseUrl": "/@Benjamin-xq" } } } ], "accessibility": { "accessibilityData": { "label": "Benjamin" } } }, "authorThumbnail": { "thumbnails": [ { "url": "//yt3.googleusercontent.com/ytc/AIdro_k6E7UTsF655DeERRis1BYvl8uSsF973x_Fdovm3dYObcg6BnyGOi5TiYl_QS1VWQCHTQ=s32-c-k-c0x00ffffff-no-rj-mo", "width": 32, "height": 32 }, { "url": "//yt3.googleusercontent.com/ytc/AIdro_k6E7UTsF655DeERRis1BYvl8uSsF973x_Fdovm3dYObcg6BnyGOi5TiYl_QS1VWQCHTQ=s48-c-k-c0x00ffffff-no-rj-mo", "width": 48, "height": 48 }, { "url": "//yt3.googleusercontent.com/ytc/AIdro_k6E7UTsF655DeERRis1BYvl8uSsF973x_Fdovm3dYObcg6BnyGOi5TiYl_QS1VWQCHTQ=s76-c-k-c0x00ffffff-no-rj-mo", "width": 76, "height": 76 } ], "accessibility": { "accessibilityData": { "label": "Benjamin" } } }, "authorEndpoint": { "clickTrackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "commandMetadata": { "webCommandMetadata": { "url": "/@Benjamin-xq", "webPageType": "WEB_PAGE_TYPE_CHANNEL", "rootVe": 3611, "apiUrl": "/youtubei/v1/browse" } }, "browseEndpoint": { "browseId": "UC2ChxHEZCmK5Nj4JB649iKA", "canonicalBaseUrl": "/@Benjamin-xq" } }, "contentText": { "runs": [ { "text": "a9qxjTfH" } ] }, "expandButton": { "buttonRenderer": { "style": "STYLE_TEXT", "size": "SIZE_DEFAULT", "text": { "accessibility": { "accessibilityData": { "label": "Lire la suite" } }, "simpleText": "Lire la suite" }, "accessibility": { "label": "Lire la suite" }, "trackingParams": "CBgQr9gCIhMI5riXoJvbhwMV9sJJBx3MEy4d" } }, "collapseButton": { "buttonRenderer": { "style": "STYLE_TEXT", "size": "SIZE_DEFAULT", "text": { "accessibility": { "accessibilityData": { "label": "Moins" } }, "simpleText": "Moins" }, "accessibility": { "label": "Moins" }, "trackingParams": "CBcQsNgCIhMI5riXoJvbhwMV9sJJBx3MEy4d" } }, "publishedTimeText": { "runs": [ { "text": "il y a 1 mois", "navigationEndpoint": { "clickTrackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "commandMetadata": { "webCommandMetadata": { "url": "/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo", "webPageType": "WEB_PAGE_TYPE_CHANNEL", "rootVe": 3611, "apiUrl": "/youtubei/v1/browse" } }, "browseEndpoint": { "browseId": "UC2ChxHEZCmK5Nj4JB649iKA", "params": "Egljb21tdW5pdHnKASeyASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_qAgQQARgB", "canonicalBaseUrl": "/post/Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" } } } ] }, "voteStatus": "INDIFFERENT", "actionButtons": { "commentActionButtonsRenderer": { "likeButton": { "toggleButtonRenderer": { "style": { "styleType": "STYLE_TEXT" }, "size": { "sizeType": "SIZE_DEFAULT" }, "isToggled": false, "isDisabled": false, "defaultIcon": { "iconType": "LIKE" }, "accessibility": { "label": "Aimer ce post" }, "trackingParams": "CBYQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=", "defaultTooltip": "J'aime", "toggledTooltip": "Je n'aime plus", "toggledStyle": { "styleType": "STYLE_DEFAULT_ACTIVE" }, "defaultNavigationEndpoint": { "clickTrackingParams": "CBYQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=", "commandMetadata": { "webCommandMetadata": { "url": "https://accounts.google.com/ServiceLogin?service=youtube&uilel=3&passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26app%3Ddesktop%26hl%3Dfr%26next%3D%252F&hl=fr", "webPageType": "WEB_PAGE_TYPE_UNKNOWN", "rootVe": 83769 } }, "signInEndpoint": { "nextEndpoint": { "clickTrackingParams": "CBYQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=", "commandMetadata": { "webCommandMetadata": { "url": "/", "webPageType": "WEB_PAGE_TYPE_BROWSE", "rootVe": 3854, "apiUrl": "/youtubei/v1/browse" } }, "browseEndpoint": { "browseId": "FEwhat_to_watch" } } } }, "accessibilityData": { "accessibilityData": { "label": "Aimer ce post" } }, "toggledAccessibilityData": { "accessibilityData": { "label": "Je n'aime plus" } } } }, "dislikeButton": { "toggleButtonRenderer": { "style": { "styleType": "STYLE_TEXT" }, "size": { "sizeType": "SIZE_DEFAULT" }, "isToggled": false, "isDisabled": false, "defaultIcon": { "iconType": "DISLIKE" }, "accessibility": { "label": "Cliquez sur \"Je n'aime pas\" pour ce post" }, "trackingParams": "CBUQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=", "defaultTooltip": "Je n'aime pas", "toggledTooltip": "Supprimer la mention Je n'aime pas", "toggledStyle": { "styleType": "STYLE_DEFAULT_ACTIVE" }, "defaultNavigationEndpoint": { "clickTrackingParams": "CBUQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=", "commandMetadata": { "webCommandMetadata": { "url": "https://accounts.google.com/ServiceLogin?service=youtube&uilel=3&passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26app%3Ddesktop%26hl%3Dfr%26next%3D%252F&hl=fr", "webPageType": "WEB_PAGE_TYPE_UNKNOWN", "rootVe": 83769 } }, "signInEndpoint": { "nextEndpoint": { "clickTrackingParams": "CBUQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=", "commandMetadata": { "webCommandMetadata": { "url": "/", "webPageType": "WEB_PAGE_TYPE_BROWSE", "rootVe": 3854, "apiUrl": "/youtubei/v1/browse" } }, "browseEndpoint": { "browseId": "FEwhat_to_watch" } } } }, "accessibilityData": { "accessibilityData": { "label": "Cliquez sur \"Je n'aime pas\" pour ce post" } }, "toggledAccessibilityData": { "accessibilityData": { "label": "Supprimer la mention Je n'aime pas" } } } }, "trackingParams": "CBQQtXUiEwjmuJegm9uHAxX2wkkHHcwTLh0=", "style": "COMMENT_ACTION_BUTTON_STYLE_TYPE_DESKTOP_TOOLBAR" } }, "actionMenu": { "menuRenderer": { "items": [ { "menuServiceItemRenderer": { "text": { "accessibility": { "accessibilityData": { "label": "Signaler" } }, "simpleText": "Signaler" }, "icon": { "iconType": "FLAG" }, "serviceEndpoint": { "clickTrackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "showEngagementPanelEndpoint": { "identifier": { "tag": "PAabuse_report" }, "globalConfiguration": { "params": "qgcoCAMSJFVna3g0eldfWjZRZUtWU25SelBtbkY1cEFkRElKNHRCaWtTbw%3D%3D" }, "engagementPanelPresentationConfigs": { "engagementPanelPopupPresentationConfig": { "popupType": "PANEL_POPUP_TYPE_DIALOG" } } } }, "trackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d" } } ], "trackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "accessibility": { "accessibilityData": { "label": "Menu d'actions" } }, "menuPopupAccessibility": { "label": "Liste des actions du menu" } } }, "trackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "surface": "BACKSTAGE_SURFACE_TYPE_STREAM", "loggingDirectives": { "trackingParams": "CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d", "visibility": { "types": "12" }, "enableDisplayloggerExperiment": true } } }, "trackingParams": "CBIQzL8CGAAiEwjmuJegm9uHAxX2wkkHHcwTLh0=", "useUpdatedRepostUi": false, "loggingDirectives": { "trackingParams": "CBIQzL8CGAAiEwjmuJegm9uHAxX2wkkHHcwTLh0=", "visibility": { "types": "12" }, "enableDisplayloggerExperiment": true } } } ```
echo -n 'CBMQ9LwCIhMI5riXoJvbhwMV9sJJBx3MEy4d' | base64 -d | protoc --decode_raw
Output: ``` 1: 19 2: 40564 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```
date -d @1722770237
Sun Aug  4 01:17:17 PM CEST 2024

is today.

echo -n 'CBgQr9gCIhMI5riXoJvbhwMV9sJJBx3MEy4d' | base64 -d | protoc --decode_raw
Output: ``` 1: 24 2: 44079 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```

so the same timestamp.

echo -n 'CBcQsNgCIhMI5riXoJvbhwMV9sJJBx3MEy4d' | base64 -d | protoc --decode_raw
Output: ``` 1: 23 2: 44080 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```

same timestamp.

echo -n 'Egljb21tdW5pdHnKASeyASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_qAgQQARgB' | base64 -d
    community�'�$Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSbase64: invalid input

does not seem interesting even with one or two appending = it does not solve the error, hence no Protobuf decoding seems possible.

import blackboxprotobuf
import base64
import json

data = base64.b64decode('Egljb21tdW5pdHnKASeyASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_qAgQQARgB', altchars = '-_')
message, typedef = blackboxprotobuf.decode_message(data)
print(json.dumps(message, indent = 4))
Output: ```json { "2": "community", "25": { "22": "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" }, "45": { "2": 1, "3": 1 } } ```

Source: menmob/innertube-documentation/issues/1#issuecomment-1688923923

echo -n 'Egljb21tdW5pdHnKASeyASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_qAgQQARgB' | base64url -d | protoc --decode_raw
Output: ``` 2: "community" 25 { 22: "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" } 45 { 2: 1 3: 1 } ```
echo -n 'CBYQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=' | base64 -d | protoc --decode_raw
Output: ``` 1: 22 2: 9880 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```

same timestamp.

echo -n 'CBUQmE0iEwjmuJegm9uHAxX2wkkHHcwTLh0=' | base64 -d | protoc --decode_raw
Output: ``` 1: 21 2: 9880 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```

same timestamp.

echo -n 'CBQQtXUiEwjmuJegm9uHAxX2wkkHHcwTLh0=' | base64 -d | protoc --decode_raw
Output: ``` 1: 20 2: 15029 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```

same timestamp.

echo -n 'qgcoCAMSJFVna3g0eldfWjZRZUtWU25SelBtbkY1cEFkRElKNHRCaWtTbw==' | base64 -d | protoc --decode_raw
Output: ``` 117 { 1: 3 2: "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" } ```

not interesting.

echo -n 'CBIQzL8CGAAiEwjmuJegm9uHAxX2wkkHHcwTLh0=' | base64 -d | protoc --decode_raw
Output: ``` 1: 18 2: 40908 3: 0 4 { 1: 1722770237283430 2: 0x0749c2f6 3: 0x1d2e13cc } ```

same timestamp.

So all tracking seem uninteresting.

Benjamin-Loison commented 3 months ago

https://www.youtube.com/youtubei/v1/browse continuation:

echo -n '4qmFsgL_ARIYVUMyQ2h4SEVaQ21LNU5qNEpCNjQ5aUtBGuIBRWdsamIyMXRkVzVwZEhtNEFRREtBU2V5QVNSVloydDROSHBYWDFvMlVXVkxWbE51VW5wUWJXNUdOWEJCWkVSSlNqUjBRbWxyVTJfcUFnUVFBUmdCa2dNQXFnTmJJa2N3QU5nQkFlb0JKRlZuYTNnMGVsZGZXalpSWlV0V1UyNVNlbEJ0YmtZMWNFRmtSRWxLTkhSQ2FXdFRiX0lCR0ZWRE1rTm9lRWhGV2tOdFN6Vk9halJLUWpZME9XbExRVUlRWTI5dGJXVnVkSE10YzJWamRHbHZidklHQkFvQ1NnQSUzRA==' | base64url -d | protoc --decode_raw
Output: ``` 80226972 { 2: "UC2ChxHEZCmK5Nj4JB649iKA" 3: "Egljb21tdW5pdHm4AQDKASeyASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_qAgQQARgBkgMAqgNbIkcwANgBAeoBJFVna3g0eldfWjZRZUtWU25SelBtbkY1cEFkRElKNHRCaWtTb_IBGFVDMkNoeEhFWkNtSzVOajRKQjY0OWlLQUIQY29tbWVudHMtc2VjdGlvbvIGBAoCSgA%3D" } ```
echo -n 'Egljb21tdW5pdHm4AQDKASeyASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_qAgQQARgBkgMAqgNbIkcwANgBAeoBJFVna3g0eldfWjZRZUtWU25SelBtbkY1cEFkRElKNHRCaWtTb_IBGFVDMkNoeEhFWkNtSzVOajRKQjY0OWlLQUIQY29tbWVudHMtc2VjdGlvbvIGBAoCSgA=' | base64url -d | protoc --decode_raw
Output: ``` 2: "community" 23: 0 25 { 22: "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" } 45 { 2: 1 3: 1 } 50: "" 53 { 4 { 6: 0 27: 1 29: "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" 30: "UC2ChxHEZCmK5Nj4JB649iKA" } 8: "comments-section" } 110 { 1 { 9: "" } } ```
Benjamin-Loison commented 3 months ago

Concerning the response:

b.json

jq keys b
[
  "frameworkUpdates",
  "onResponseReceivedEndpoints",
  "responseContext",
  "trackingParams"
]

I checked in the following frameworkUpdates and onResponseReceivedEndpoints.

/frameworkUpdates/entityBatchUpdate/timestamp:

date -d @1722772289
Sun Aug  4 01:51:29 PM CEST 2024
echo -n '4qmFsgK1ARIYVUMyQ2h4SEVaQ21LNU5qNEpCNjQ5aUtBGpgBRWdsamIyMXRkVzVwZEhtcUExOGlTVEFBZUFMSUFRRHFBU1JWWjJ0NE5IcFhYMW8yVVdWTFZsTnVVbnBRYlc1R05YQkJaRVJKU2pSMFFtbHJVMl95QVJoVlF6SkRhSGhJUlZwRGJVczFUbW8wU2tJMk5EbHBTMEU0QVVJUVkyOXRiV1Z1ZEhNdGMyVmpkR2x2YmclM0QlM0Q=' | base64url -d | protoc --decode_raw
Output: ``` 80226972 { 2: "UC2ChxHEZCmK5Nj4JB649iKA" 3: "Egljb21tdW5pdHmqA18iSTAAeALIAQDqASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_yARhVQzJDaHhIRVpDbUs1Tmo0SkI2NDlpS0E4AUIQY29tbWVudHMtc2VjdGlvbg%3D%3D" } ```
echo -n 'Egljb21tdW5pdHmqA18iSTAAeALIAQDqASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_yARhVQzJDaHhIRVpDbUs1Tmo0SkI2NDlpS0E4AUIQY29tbWVudHMtc2VjdGlvbg==' | base64url -d | protoc --decode_raw
Output: ``` 2: "community" 53 { 4 { 6: 0 15: 2 25: 0 29: "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" 30: "UC2ChxHEZCmK5Nj4JB649iKA" } 7: 1 8: "comments-section" } ```
echo -n '4qmFsgK1ARIYVUMyQ2h4SEVaQ21LNU5qNEpCNjQ5aUtBGpgBRWdsamIyMXRkVzVwZEhtcUExOGlTVEFCZUFMSUFRRHFBU1JWWjJ0NE5IcFhYMW8yVVdWTFZsTnVVbnBRYlc1R05YQkJaRVJKU2pSMFFtbHJVMl95QVJoVlF6SkRhSGhJUlZwRGJVczFUbW8wU2tJMk5EbHBTMEU0QVVJUVkyOXRiV1Z1ZEhNdGMyVmpkR2x2YmclM0QlM0Q=' | base64url -d | protoc --decode_raw
Output: ``` 80226972 { 2: "UC2ChxHEZCmK5Nj4JB649iKA" 3: "Egljb21tdW5pdHmqA18iSTABeALIAQDqASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_yARhVQzJDaHhIRVpDbUs1Tmo0SkI2NDlpS0E4AUIQY29tbWVudHMtc2VjdGlvbg%3D%3D" } ```
echo -n 'Egljb21tdW5pdHmqA18iSTABeALIAQDqASRVZ2t4NHpXX1o2UWVLVlNuUnpQbW5GNXBBZERJSjR0QmlrU2_yARhVQzJDaHhIRVpDbUs1Tmo0SkI2NDlpS0E4AUIQY29tbWVudHMtc2VjdGlvbg==' | base64url -d | protoc --decode_raw
Output: ``` 2: "community" 53 { 4 { 6: 1 15: 2 25: 0 29: "Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo" 30: "UC2ChxHEZCmK5Nj4JB649iKA" } 7: 1 8: "comments-section" } ```
Benjamin-Loison commented 3 months ago

Next page of community posts:

curl 'https://www.youtube.com/youtubei/v1/browse?prettyPrint=false' --compressed -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'Referer: https://www.youtube.com/@Benjamin-xq/community' -H 'Content-Type: application/json' -H 'X-Goog-EOM-Visitor-Id: CgttRHpBU09DUHVHQSi_1r21BjIiCgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgDw%3D%3D' -H 'X-Youtube-Bootstrap-Logged-In: false' -H 'X-Youtube-Client-Name: 1' -H 'X-Youtube-Client-Version: 2.20240731.04.00' -H 'Origin: https://www.youtube.com' -H 'DNT: 1' -H 'Sec-GPC: 1' -H 'Connection: keep-alive' -H 'Cookie: SOCS=CAESNQgDEitib3FfaWRlbnRpdHlmcm9udGVuZHVpc2VydmVyXzIwMjQwNzMwLjA1X3AwGgJlbiACGgYIgIm7tQY; YSC=mCkhd8P4OAM; __Secure-YEC=CgttRHpBU09DUHVHQSi_1r21BjIiCgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgDw%3D%3D; VISITOR_PRIVACY_METADATA=CgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgDw%3D%3D; PREF=f4=4000000&f6=40000000&tz=Europe.Paris' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: same-origin' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw '{"context":{"client":{"hl":"en","gl":"FR","remoteHost":"2a01:cb04:609:5d00:188:4bc8:4e3d:699c","deviceMake":"","deviceModel":"","visitorData":"CgttRHpBU09DUHVHQSi_1r21BjIiCgJGUhIcEhgSFhMLFBUWFwwYGRobHB0eHw4PIBAREiEgDw%3D%3D","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0,gzip(gfe)","clientName":"WEB","clientVersion":"2.20240731.04.00","osName":"X11","osVersion":"","originalUrl":"https://www.youtube.com/channel/UC2ChxHEZCmK5Nj4JB649iKA/community?lb=Ugkx4zW_Z6QeKVSnRzPmnF5pAdDIJ4tBikSo","screenPixelDensity":2,"platform":"DESKTOP","clientFormFactor":"UNKNOWN_FORM_FACTOR","configInfo":{"appInstallData":"CL_WvbUGENqgsQUQ9quwBRDvzbAFEK-hsQUQqLewBRCx3LAFEMn3rwUQpaWxBRDd6P4SEKXC_hIQt--vBRCDuf8SEJCSsQUQ1KGvBRCmk7EFEKrYsAUQlpWwBRDrmbEFEN_1sAUQyeawBRDviLEFEO6irwUQ1YiwBRC2sf8SEJKdsQUQ6sOvBRDBpbEFEMr5sAUQsO6wBRC9tq4FENuvrwUQ0PqwBRCinbEFEMefsQUQsKqxBRD0q7AFENCNsAUQ1-mvBRComrAFEKaasAUQj8SwBRDwnLAFEJ7QsAUQlqOxBRCI468FEMSSsQUQ_IWwBRC9mbAFEKiTsQUQmZixBRC36v4SEOPRsAUQ65OuBRDW3bAFEInorgUQ4bz_EhD_iLEFEI7asAUQooGwBRCU_rAFENqlsQUQydewBRDM364FEOLUrgUQvoqwBRCokrEFENnJrwUQzdewBRCNzLAFEI2UsQUQnaawBRDr6P4SELn4sAUQ0-GvBRDb_rciEJrwrwUQi8-wBRD68LAFEJSJsQUQ4eywBRCIh7AFEOX0sAUQppKxBRCPlLEFEJuosQUQ-oKxBSogQ0FNU0VoVUpvTDJ3RE5Ia0J2UHQ4UXVCOXdFZEJ3PT0%3D"},"screenDensityFloat":2,"userInterfaceTheme":"USER_INTERFACE_THEME_DARK","timeZone":"Europe/Paris","browserName":"Firefox","browserVersion":"128.0","acceptHeader":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8","deviceExperimentId":"ChxOek01T1RJMU1EWXpNalV6TXpZNE56WXhOZz09EL_WvbUGGL_WvbUG","screenWidthPoints":1128,"screenHeightPoints":315,"utcOffsetMinutes":120,"mainAppWebInfo":{"graftUrl":"https://www.youtube.com/@Benjamin-xq/community","pwaInstallabilityStatus":"PWA_INSTALLABILITY_STATUS_UNKNOWN","webDisplayMode":"WEB_DISPLAY_MODE_BROWSER","isWebNativeShareAvailable":false}},"user":{"lockedSafetyMode":false},"request":{"useSsl":true,"internalExperimentFlags":[],"consistencyTokenJars":[]},"clickTracking":{"clickTrackingParams":"CBoQuy8YACITCK2muby224cDFdEZ8QUdTOw0ew=="},"adSignalsInfo":{"params":[{"key":"dt","value":"1722772287758"},{"key":"flash","value":"0"},{"key":"frm","value":"0"},{"key":"u_tz","value":"120"},{"key":"u_his","value":"4"},{"key":"u_h","value":"752"},{"key":"u_w","value":"1128"},{"key":"u_ah","value":"712"},{"key":"u_aw","value":"1128"},{"key":"u_cd","value":"24"},{"key":"bc","value":"31"},{"key":"bih","value":"315"},{"key":"biw","value":"1128"},{"key":"brdim","value":"0,0,0,0,1128,0,1128,684,1128,315"},{"key":"vis","value":"1"},{"key":"wgl","value":"true"},{"key":"ca_type","value":"image"}]}},"continuation":"4qmFsgKNARIYVUMyQ2h4SEVaQ21LNU5qNEpCNjQ5aUtBGlhFZ2xqYjIxdGRXNXBkSG1xQXlnS0pGRXlhRU5TUms1WVRsWnNUMDFJYUU5VmFrSlhWVWRHZEdWSVRtbE5WVnBGVWxWR1FpZ0s4Z1lFQ2dKS0FBJTNEJTNEmgIWYmFja3N0YWdlLWl0ZW0tc2VjdGlvbg%3D%3D"}'
echo -n '4qmFsgKNARIYVUMyQ2h4SEVaQ21LNU5qNEpCNjQ5aUtBGlhFZ2xqYjIxdGRXNXBkSG1xQXlnS0pGRXlhRU5TUms1WVRsWnNUMDFJYUU5VmFrSlhWVWRHZEdWSVRtbE5WVnBGVWxWR1FpZ0s4Z1lFQ2dKS0FBJTNEJTNEmgIWYmFja3N0YWdlLWl0ZW0tc2VjdGlvbg==' | base64url -d | protoc --decode_raw
Output: ``` 80226972 { 2: "UC2ChxHEZCmK5Nj4JB649iKA" 3: "Egljb21tdW5pdHmqAygKJFEyaENSRk5YTlZsT01IaE9VakJXVUdGdGVITmlNVVpFUlVGQigK8gYECgJKAA%3D%3D" 35: "backstage-item-section" } ```
echo -n 'Egljb21tdW5pdHmqAygKJFEyaENSRk5YTlZsT01IaE9VakJXVUdGdGVITmlNVVpFUlVGQigK8gYECgJKAA==' | base64url -d | protoc --decode_raw
Output: ``` 2: "community" 53 { 1: "Q2hCRFNXNVlOMHhOUjBWUGFteHNiMUZERUFB" 5: 10 } 110 { 1 { 9: "" } } ```
echo -n 'Q2hCRFNXNVlOMHhOUjBWUGFteHNiMUZERUFB' | base64url -d
ChBDSW5YN0xNR0VPamxsb1FDEAA
echo -n 'Q2hCRFNXNVlOMHhOUjBWUGFteHNiMUZERUFB' | base64url -d | base64 -d
CInX7LMGEOjlloQCbase64: invalid input
echo -n 'ChBDSW5YN0xNR0VPamxsb1FDEAA=' | base64 -d | protoc --decode_raw
1: "CInX7LMGEOjlloQC"
2: 0
echo -n 'CInX7LMGEOjlloQC' | base64 -d | protoc --decode_raw
1: 1719348105
2: 545633000
date -d @1719348105
Tue Jun 25 10:41:45 PM CEST 2024

looks interesting.

The question is is the same timestamp always returned and where does it come from? The same timestamp is always returned.

getJSONPathFromKey community.html token
202 /contents/twoColumnBrowseResultsRenderer/tabs/2/tabRenderer/content/sectionListRenderer/contents/0/itemSectionRenderer/contents/10/continuationItemRenderer/continuationEndpoint/continuationCommand/token 4qmFsgKVARIYVUMyQ2h4SEVaQ21LNU5qNEpCNjQ5aUtBGmBFZ2xqYjIxdGRXNXBkSG00QVFDU0F3Q3FBeWdLSkZFeWFFTlNSazVZVGxac1QwMUlhRTlWYWtKWFZVZEdkR1ZJVG1sTlZWcEZVbFZHUWlnSzhnWUpDZ2RLQUtJQkFnZ0KaAhZiYWNrc3RhZ2UtaXRlbS1zZWN0aW9u
369 /header/pageHeaderRenderer/content/pageHeaderViewModel/description/descriptionPreviewViewModel/rendererContext/commandContext/onTap/innertubeCommand/showEngagementPanelEndpoint/engagementPanel/engagementPanelSectionListRenderer/content/sectionListRenderer/contents/0/itemSectionRenderer/contents/0/continuationItemRenderer/continuationEndpoint/continuationCommand/token 4qmFsgJgEhhVQzJDaHhIRVpDbUs1Tmo0SkI2NDlpS0EaRDhnWXJHaW1hQVNZS0pEWTRORE5qTWpJd0xUQXdNREF0TW1KbE5TMDROV1JsTFdZME1ETXdORE5rTmpVMU9BJTNEJTNE
/contents/twoColumnBrowseResultsRenderer/tabs/2/tabRenderer/content/sectionListRenderer/contents/0/itemSectionRenderer/contents/10/continuationItemRenderer/continuationEndpoint/continuationCommand/token

Can we somehow leverage a dichotomy to request results after a given datetime?

https://www.youtube.com/post/Ugwu1_hVsgkc3x-6V9B4AaABCQ

curl https://www.youtube.com/youtubei/v1/browse -H 'Content-Type: application/json' --data-raw '{"context": {"client": {"clientName": "WEB", "clientVersion": "2.20240731.04.00"}}, "continuation": "4qmFsgKNARIYVUNReEpzQWxxbUJQQWJSXzBzeURpOW1nGlhFZ2xqYjIxdGRXNXBkSG1xQXlnS0pGRXlZelZTUmxKelZXMXNhbEo2YkVOVmFrSlhWRzFHY2sxWVVtRk5iVTVTVVZWRlBTZ284Z1lFQ2dKS0FBJTNEJTNEmgIWYmFja3N0YWdlLWl0ZW0tc2VjdGlvbg%3D%3D"}'
import requests
import blackboxprotobuf
import base64

def getBase64Protobuf(message, typedef):
    data = blackboxprotobuf.encode_message(message, typedef)
    return base64.b64encode(data).decode('ascii')

message = {
    '1': 1611247060,
}

typedef = {
    '1': {
        'type': 'int'
    },
}

one = getBase64Protobuf(message, typedef)

message = {
    '1': one,
}

typedef = {
    '1': {
        'type': 'string'
    },
}

one = base64.b64encode(getBase64Protobuf(message, typedef).encode('ascii'))

message = {
    '2': 'community',
    '53': {
        '1': one,
    },
}

typedef = {
    '2': {
        'type': 'string'
    },
    '53': {
        'type': 'message',
        'message_typedef': {
            '1': {
                'type': 'string'
            },
        },
    },
}

three = getBase64Protobuf(message, typedef)

message = {
    '80226972': {
        '2': 'UCQxJsAlqmBPAbR_0syDi9mg',
        '3': three,
    }
}

typedef = {
    '80226972': {
        'type': 'message',
        'message_typedef': {
            '2': {
                'type': 'string'
            },
            '3': {
                'type': 'string'
            },
        },
        'field_order': [
            '2',
            '3',
        ]
    }
}

continuation = getBase64Protobuf(message, typedef)

json_data = {
    'context': {
        'client': {
            'clientName': 'WEB',
            'clientVersion': '2.20240731.04.00',
        },
    },
    'continuation': continuation,
}

response = requests.post('https://www.youtube.com/youtubei/v1/browse', headers=headers, json=json_data)
print('There is a lot going on behind the shiny veil of innovation and flagship phones.' in response.text)
Benjamin-Loison commented 3 months ago

browse.json

jq keys browse.json 
[
  "metadata",
  "microformat",
  "onResponseReceivedEndpoints",
  "responseContext",
  "trackingParams"
]
echo -n 'Egljb21tdW5pdHnKASeyASRVZ2t4ajZZUE5qcUFkdTFnOHgyVFBSYnk2dTc2S08tTkJwQ27qAgQQARgB' | base64 -d | protoc --decode_raw
Output: ``` 2: "community" 25 { 22: "Ugkxj6YPNjqAdu1g8x2TPRby6u76KO-NBpCn" } 45 { 2: 1 3: 1 } ```
echo -n 'qgcoCAMSJFVna3hqNllQTmpxQWR1MWc4eDJUUFJieTZ1NzZLTy1OQnBDbg==' | base64 -d | protoc --decode_raw
Output: ``` 117 { 1: 3 2: "Ugkxj6YPNjqAdu1g8x2TPRby6u76KO-NBpCn" } ```
echo -n 'Egljb21tdW5pdHnKASeyASRVZ2t4aXFvWUJ1aE5KTG9HN1Y0RDlDWWpjMC1KVGYxZE1SZVHqAgQQARgB' | base64 -d | protoc --decode_raw
Output: ``` 2: "community" 25 { 22: "UgkxiqoYBuhNJLoG7V4D9CYjc0-JTf1dMReQ" } 45 { 2: 1 3: 1 } ```
echo -n 'qgcoCAMSJFVna3hpcW9ZQnVoTkpMb0c3VjREOUNZamMwLUpUZjFkTVJlUQ==' | base64 -d | protoc --decode_raw
Output: ``` 117 { 1: 3 2: "UgkxiqoYBuhNJLoG7V4D9CYjc0-JTf1dMReQ" } ```

how many entries have been returned in fact?

I verified metadata, microformat and onResponseReceivedEndpoints.

Benjamin-Loison commented 3 months ago

community.json

jq keys community.json
[
  "contents",
  "header",
  "metadata",
  "microformat",
  "responseContext",
  "topbar",
  "trackingParams"
]
echo -n 'EgdzdHJlYW1z8gYECgJ6AA==' | base64 -d | protoc --decode_raw
Output: ``` 2: "streams" 110 { 1 { 15: "" } } ```
echo -n 'Egljb21tdW5pdHnyBgQKAkoA' | base64 -d | protoc --decode_raw
Output: ``` 2: "community" 110 { 1 { 9: "" } } ```
echo -n 'Egljb21tdW5pdHnyBgkKB0oAogECCAE=' | base64 -d | protoc --decode_raw
Output: ``` 2: "community" 110 { 1 { 9: "" 20 { 1: 1 } } } ```
echo -n 'Egljb21tdW5pdHnKASeyASRVZ2t4UzMwOHprX1ZsRDJxUU9GcmxDR0xKZDVZYnRhWUJiOUbqAgQQARgB' | base64 -d | protoc --decode_raw
Output: ``` 2: "community" 25 { 22: "UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F" } 45 { 2: 1 3: 1 } ```
echo -n 'qgcoCAMSJFVna3hTMzA4emtfVmxEMnFRT0ZybENHTEpkNVlidGFZQmI5Rg==' | base64 -d | protoc --decode_raw
Output: ``` 117 { 1: 3 2: "UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F" } ```
echo -n 'QUFFLUhqbnBRYUZGS1lCdmxzNlh1SjFJbkxCN3JxN25Zd3xBQ3Jtc0tudzZ6U0FFZWktZW1qaFg5VVZ1b0Q5ZU5pNEZaMUF3SzR3YlFwMHVhRmFhMTVuMEZXbDh1c0JsYnY1UzR1YnBMRVdvVHdrYmpXUW9CeVFNLVlkNlJUUVRxY0U3bE9SOEQtd0RGaF9EWHg3RThHOG9kWDB6Rk1XRVlOX0tLRlJOdGxubTl0QzRPLWFNNFdFVElwZmVJTGd0a1Y4WEQ0R2o2OF8tOUdiMU1ZcHp2LXZUWjE2MjAwdUtzMWlqNjkzYmtxbDUydmw=' | base64 -d
AAE-HjnpQaFFKYBvls6XuJ1InLB7rq7nYw|ACrmsKnw6zSAEei-emjhX9UVuoD9eNi4FZ1AwK4wbQp0uaFaa15n0FWl8usBlbv5S4ubpLEWoTwkbjWQoByQM-Yd6RTQTqcE7lOR8D-wDFh_DXx7E8G8odX0zFMWEYN_KKFRNtlnm9tC4O-aM4WETIpfeILgtkV8XD4Gj68_-9Gb1MYpzv-vTZ16200uKs1ij693bkql52vl

Unable to decode further with base64{,url}.

echo -n '4qmFsgJgEhhVQzJDaHhIRVpDbUs1Tmo0SkI2NDlpS0EaRDhnWXJHaW1hQVNZS0pEWTRNMk5qT1dWbUxUQXdNREF0TWpBMllpMDVORFl3TFRFMFl6RTBaV1kwTURVMll3JTNEJTNE' | base64 -d | protoc --decode_raw
Output: ``` 80226972 { 2: "UC2ChxHEZCmK5Nj4JB649iKA" 3: "8gYrGimaASYKJDY4M2NjOWVmLTAwMDAtMjA2Yi05NDYwLTE0YzE0ZWY0MDU2Yw%3D%3D" } ```
echo -n '8gYrGimaASYKJDY4M2NjOWVmLTAwMDAtMjA2Yi05NDYwLTE0YzE0ZWY0MDU2Yw==' | base64 -d | protoc --decode_raw
Output: ``` 110 { 3 { 19 { 1: "683cc9ef-0000-206b-9460-14c14ef4056c" } } } ```
echo -n 'EgZ0b3BiYXIg9QEoAQ==' | base64 -d | protoc --decode_raw
Output: ``` 2: "topbar" 4: 245 5: 1 ```

I checked first returned item, contents, header (seems not community post focused), metadata and microformat.

Benjamin-Loison commented 3 months ago
import requests
import blackboxprotobuf
import base64

def getBase64Protobuf(message, typedef):
    data = blackboxprotobuf.encode_message(message, typedef)
    return base64.b64encode(data).decode('ascii')

def getCommunity(timestamp):
    message = {
        '1': timestamp,
    }

    typedef = {
        '1': {
            'type': 'int'
        },
    }

    one = getBase64Protobuf(message, typedef)

    message = {
        '1': one,
    }

    typedef = {
        '1': {
            'type': 'string'
        },
    }

    one = base64.b64encode(getBase64Protobuf(message, typedef).encode('ascii'))

    message = {
        '2': 'community',
        '53': {
            '1': one,
        },
    }

    typedef = {
        '2': {
            'type': 'string'
        },
        '53': {
            'type': 'message',
            'message_typedef': {
                '1': {
                    'type': 'string'
                },
            },
        },
    }

    three = getBase64Protobuf(message, typedef)

    message = {
        '80226972': {
            '2': 'UCgvqvBoSHB1ctlyyhoHrGwQ',
            '3': three,
        }
    }

    typedef = {
        '80226972': {
            'type': 'message',
            'message_typedef': {
                '2': {
                    'type': 'string'
                },
                '3': {
                    'type': 'string'
                },
            },
            'field_order': [
                '2',
                '3',
            ]
        }
    }

    continuation = getBase64Protobuf(message, typedef)

    json_data = {
        'context': {
            'client': {
                'clientName': 'WEB',
                'clientVersion': '2.20240731.04.00',
            },
        },
        'continuation': continuation,
    }

    response = requests.post('https://www.youtube.com/youtubei/v1/browse', json = json_data)
    return response.json()

community = getCommunity(1722782039)
print('ce dimanche pour cause de vacances en famille' in str(community))
Benjamin-Loison commented 3 months ago

https://www.youtube.com/post/UgkxOpujSABK9-1yzNZHml1PkEmExobp1s8Z

Benjamin-Loison commented 3 months ago

As if use future timestamp such as 1732782039 it still returns True, I believe that this value is an upperbound of results returned.

We would like a code supporting all community posts, not all except the most recent one etc.

I start feeling it is not deterministic.

Benjamin-Loison commented 3 months ago
timestamp = 1722782903
TO_REMOVE = 1
while True:
    community = getCommunity(timestamp)
    isIn = 'ce dimanche pour cause de vacances en famille' in str(community)
    print(f'{timestamp=} {isIn=}')
    if not isIn:
        break
    timestamp -= TO_REMOVE
timestamp=1722782903 isIn=True
timestamp=1722782902 isIn=True
...
timestamp=1722782896 isIn=True
timestamp=1722782895 isIn=False

but with TO_REMOVE = 10:

timestamp=1722782903 isIn=True
timestamp=1722782893 isIn=True
timestamp=1722782883 isIn=True
timestamp=1722782873 isIn=True
timestamp=1722782863 isIn=False
Benjamin-Loison commented 3 months ago

In fact if do not break then get:

isIn=True except sometimes isIn=False ``` timestamp=1722782903 isIn=True timestamp=1722782902 isIn=True ... timestamp=1722782896 isIn=True timestamp=1722782895 isIn=False timestamp=1722782894 isIn=True ... timestamp=1722782880 isIn=True timestamp=1722782879 isIn=False timestamp=1722782878 isIn=True ... ```

let us return the first community post content to maybe better understand.

Benjamin-Loison commented 3 months ago
communityPostMessage = community['contents']['twoColumnBrowseResultsRenderer']['tabs'][5]['tabRenderer']['content']['sectionListRenderer']['contents'][0]['itemSectionRenderer']['contents'][0]['backstagePostThreadRenderer']['post']['backstagePostRenderer']['contentText']['runs'][0]['text']

is not relevant as not from AJAX.

Benjamin-Loison commented 3 months ago
timestamp Python decrease by 1 ```python timestamp = 1722782903 TO_REMOVE = 1 while True: community = getCommunity(timestamp) communityPostMessage = community['continuationContents']['itemSectionContinuation']['contents'][0]['backstagePostThreadRenderer']['post']['backstagePostRenderer']['contentText']['runs'][0]['text'] isIn = 'ce dimanche pour cause de vacances en famille' in communityPostMessage print(f'{timestamp=} {isIn=}') #if not isIn: # break timestamp -= TO_REMOVE ```
Output resulting in KeyError: 'backstagePostThreadRenderer' ``` timestamp=1722782903 isIn=True timestamp=1722782902 isIn=True ... timestamp=1722782896 isIn=True Traceback (most recent call last): File "", line 101, in communityPostMessage = community['continuationContents']['itemSectionContinuation']['contents'][0]['backstagePostThreadRenderer']['post']['backstagePostRenderer']['contentText']['runs'][0]['text'] KeyError: 'backstagePostThreadRenderer' ```
Benjamin-Loison commented 3 months ago
timestamp Python decrease by 100 ```python timestamp = 1722782903 TO_REMOVE = 100 while True: #print(f'{timestamp=}') community = getCommunity(timestamp) content = community['continuationContents']['itemSectionContinuation']['contents'][0] timestamp -= TO_REMOVE if 'messageRenderer' in content and content['messageRenderer']['text']['runs'][0]['text'] == "This channel hasn't posted yet": continue communityPostMessage = content['backstagePostThreadRenderer']['post']['backstagePostRenderer']['contentText']['runs'][0]['text'] isIn = 'ce dimanche pour cause de vacances en famille' in communityPostMessage #print(f'{isIn=}') print(f'{timestamp=} {isIn=}') #if not isIn: # break ```
Output mentioning isIn=Trues then isIn=Falses ``` timestamp=1722782803 isIn=True timestamp=1722782703 isIn=True ... timestamp=1722777503 isIn=True timestamp=1722777203 isIn=True timestamp=1722777103 isIn=False timestamp=1722776903 isIn=False ... timestamp=1722774803 isIn=False timestamp=1722774703 isIn=False ```
Benjamin-Loison commented 3 months ago

After about 10 executions I confirm that This channel hasn't posted yet seems deterministic.

Benjamin-Loison commented 3 months ago

Recently I tried automatically create community posts but do not remember where I keep track of that. I remember having achieved an algorithm blocked due to bot verification after a few posts. I stopped investigating mentioning that it takes too much of my time but I do not quickly find it on Discord.

Benjamin-Loison commented 3 months ago

An approximative timestamp is when switching from is in to is not more in, no matter if most recent or not comment.

Benjamin-Loison commented 3 months ago

So:

timestamp timestamp + 1 possible next timestamp
False False True
False True True
True False True
True True True

According to my understanding, booleans standing for is in.

Dichotomy between channel creation and now.

Benjamin-Loison commented 3 months ago

Maybe the most simple is to guess all community post creation time.

Even if start from most recent comment and decrease exponentially time, cannot know if missed one.

Benjamin-Loison commented 3 months ago

Let us assume multiple community posts to start.

Benjamin-Loison commented 3 months ago

Related to #257.

Benjamin-Loison commented 3 months ago
community['backstagePostThreadRenderer']['post']['backstagePostRenderer']['postId']

does not seem useful.

Benjamin-Loison commented 3 months ago
YOUTUBE_OPERATIONAL_API_INSTANCE_URL = 'http://localhost/YouTube-operational-API'
communityPostIds = []
params = {
    'part': 'community',
    'handle': '@Amixem',
}
while True:
    data = requests.get(YOUTUBE_OPERATIONAL_API_INSTANCE_URL + '/channels', params).json()
    #print(json.dumps(data, indent = 4))
    item = data['items'][0]
    for communityPost in item['community']:
        communityPostIds += [communityPost['id']]
    if not 'nextPageToken' in item:
        break
    params['pageToken'] = item['nextPageToken']

currentTimestamp = time.time()
timestamp = math.ceil(currentTimestamp)
toRemove = 1
while True:
    community = getCommunity(timestamp)
    communityDatetime = datetime.fromtimestamp(timestamp)
    print(communityDatetime)
    content = community['continuationContents']['itemSectionContinuation']['contents'][0]
    if 'messageRenderer' in content and content['messageRenderer']['text']['runs'][0]['text'] == "This channel hasn't posted yet":
        timestamp -= 1
        continue
    if community['continuationContents']['itemSectionContinuation']['contents'][0]['backstagePostThreadRenderer']['post']['backstagePostRenderer']['postId'] != 'UgkxOpujSABK9-1yzNZHml1PkEmExobp1s8Z':
        print(datetime.fromtimestamp(currentTimestamp) - communityDatetime)
        break
    timestamp -= toRemove
    toRemove *= 2
Benjamin-Loison commented 3 months ago

https://www.youtube.com/@Benjamin-xq/community

Benjamin-Loison commented 3 months ago
print(json.dumps(getCommunity(1000)['continuationContents']['itemSectionContinuation']['contents'][0], indent = 4))
Output mentioning This channel hasn't posted yet ```json { "messageRenderer": { "text": { "runs": [ { "text": "This channel hasn't posted yet" } ] }, "trackingParams": "CAMQljsYACITCOmSvfHi24cDFTs88QUdsbkNjw==" } } ```
Benjamin-Loison commented 3 months ago
First try using channel handle and linear shift: ```python import requests import blackboxprotobuf import base64 import time import math from datetime import datetime, timedelta import re from enum import Enum, auto import time def getBase64Protobuf(message, typedef): data = blackboxprotobuf.encode_message(message, typedef) return base64.b64encode(data).decode('ascii') def getCommunity(timestamp): message = { '1': timestamp, } typedef = { '1': { 'type': 'int' }, } one = getBase64Protobuf(message, typedef) message = { '1': one, } typedef = { '1': { 'type': 'string' }, } one = base64.b64encode(getBase64Protobuf(message, typedef).encode('ascii')) message = { '2': 'community', '53': { '1': one, }, } typedef = { '2': { 'type': 'string' }, '53': { 'type': 'message', 'message_typedef': { '1': { 'type': 'string' }, }, }, } three = getBase64Protobuf(message, typedef) message = { '80226972': { '2': 'UCgvqvBoSHB1ctlyyhoHrGwQ', '3': three, } } typedef = { '80226972': { 'type': 'message', 'message_typedef': { '2': { 'type': 'string' }, '3': { 'type': 'string' }, }, 'field_order': [ '2', '3', ] } } continuation = getBase64Protobuf(message, typedef) json_data = { 'context': { 'client': { 'clientName': 'WEB', 'clientVersion': '2.20240731.04.00', }, }, 'continuation': continuation, } response = requests.post('https://www.youtube.com/youtubei/v1/browse', json = json_data) return response.json() CHANNEL_HANDLE = '@Amixem' def getApi(url, params): return requests.get(YOUTUBE_OPERATIONAL_API_INSTANCE_URL + f'/{url}', params).json() params = { 'part': 'about', 'handle': CHANNEL_HANDLE, } channelJoinedDateTime = getApi('channels', params)['items'][0]['about']['stats']['joinedDate'] HOURS_AGO = re.compile('(\d) hours ago') MONTHS_AGO = re.compile('(\d) months ago') class Approximation(Enum): UPPER = auto() LOWER = auto() # To avoid possible time shift issue in community post date string and pagination. MOST_TIME_SHIFT = timedelta(days = 1) def getTimeDelta(timeDeltaStr, approximation): hoursAgoMatch = HOURS_AGO.match(timeDeltaStr) relativeOffsetUnit = -1 if approximation is Approximation.UPPER else 1 relativeOffset = relativeOffsetUnit * MOST_TIME_SHIFT if hoursAgoMatch is not None: myTimedelta = timedelta(hours = int(hoursAgoMatch[1]) + relativeOffsetUnit) monthsAgoMatch = MONTHS_AGO.match(timeDeltaStr) #if monthsAgoMatch is not None: # Between 28 and 31. # myTimedelta = timedelta(days = 31 * int(monthsAgoMatch[1]) + relativeOffsetUnit) return int((myTimedelta + relativeOffset).total_seconds()) YOUTUBE_OPERATIONAL_API_INSTANCE_URL = 'http://localhost/YouTube-operational-API' communityPosts = [] params = { 'part': 'community', 'handle': CHANNEL_HANDLE, } while True: data = getApi('channels', params) item = data['items'][0] for communityPost in item['community']: communityPosts += [{ 'id': communityPost['id'], 'date': communityPost['date'], }] if not 'nextPageToken' in item: break params['pageToken'] = item['nextPageToken'] currentTimestamp = time.time() currentTimestampCeil = math.ceil(currentTimestamp) def printBound(type_, bound): print(f'{type_} bound: {datetime.fromtimestamp(bound)}') def getCommunityPost(timestamp, approximation): shift = 0 while True: community = getCommunity(timestamp + (1 if approximation is Approximation.UPPER else -1) * shift) content = community['continuationContents']['itemSectionContinuation']['contents'][0] if 'messageRenderer' in content and content['messageRenderer']['text']['runs'][0]['text'] == "This channel hasn't posted yet": #if shift == 1: # print('2 community posts missing in a row!') #print(f'{shift=}') shift += 1 continue return community['continuationContents']['itemSectionContinuation']['contents'][0]['backstagePostThreadRenderer']['post']['backstagePostRenderer']['postId'] def printCommunityPost(type_, communityPost): print(f'{type_} community post: {communityPost}') ## def dichotomy(upper, lower): for type_, bound in [('Upper', upper['timestamp']), ('Lower', lower['timestamp'])]: printBound(type_, bound) upperCommunityPost = getCommunityPost(upper['timestamp'], Approximation.UPPER) lowerCommunityPost = getCommunityPost(lower['timestamp'], Approximation.LOWER) for type_, communityPost in [('Upper', upperCommunityPost), ('Lower', lowerCommunityPost)]: printCommunityPost(type_, communityPost) print() middle = (upper['timestamp'] + lower['timestamp']) // 2 communityPost = getCommunityPost(middle, Approximation.UPPER) printBound('Middle', middle) printCommunityPost('Middle', communityPost) print('Decrease upper' if communityPost == upper["id"] else 'Increase lower') (upper if communityPost == upper['id'] else lower)['timestamp'] = middle print() #time.sleep(1) dichotomy(upper, lower) dichotomy( { 'id': communityPosts[0]['id'], 'timestamp': currentTimestampCeil, }, { 'id': communityPosts[1]['id'], 'timestamp': math.floor(currentTimestamp) - getTimeDelta(communityPosts[0]['date'], Approximation.LOWER), } ) ```
Benjamin-Loison commented 3 months ago

https://www.youtube.com/post/UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI

date -d @1722783810
Sun Aug  4 05:03:30 PM CEST 2024
Second try using channel id and exponential shift ```python import requests import blackboxprotobuf import base64 import time import math from datetime import datetime, timedelta import re from enum import Enum, auto import time CHANNEL_ID = 'UC2ChxHEZCmK5Nj4JB649iKA' YOUTUBE_OPERATIONAL_API_INSTANCE_URL = 'https://yt.lemnoslife.com' def getBase64Protobuf(message, typedef): data = blackboxprotobuf.encode_message(message, typedef) return base64.b64encode(data).decode('ascii') def getCommunity(timestamp): message = { '1': timestamp, } typedef = { '1': { 'type': 'int' }, } one = getBase64Protobuf(message, typedef) message = { '1': one, } typedef = { '1': { 'type': 'string' }, } one = base64.b64encode(getBase64Protobuf(message, typedef).encode('ascii')) message = { '2': 'community', '53': { '1': one, }, } typedef = { '2': { 'type': 'string' }, '53': { 'type': 'message', 'message_typedef': { '1': { 'type': 'string' }, }, }, } three = getBase64Protobuf(message, typedef) message = { '80226972': { '2': CHANNEL_ID, '3': three, } } typedef = { '80226972': { 'type': 'message', 'message_typedef': { '2': { 'type': 'string' }, '3': { 'type': 'string' }, }, 'field_order': [ '2', '3', ] } } continuation = getBase64Protobuf(message, typedef) json_data = { 'context': { 'client': { 'clientName': 'WEB', 'clientVersion': '2.20240731.04.00', }, }, 'continuation': continuation, } response = requests.post('https://www.youtube.com/youtubei/v1/browse', json = json_data) return response.json() def getApi(url, params): return requests.get(YOUTUBE_OPERATIONAL_API_INSTANCE_URL + f'/{url}', params).json() params = { 'part': 'about', 'id': CHANNEL_ID, } channelJoinedDateTime = getApi('channels', params)['items'][0]['about']['stats']['joinedDate'] HOURS_AGO = re.compile('(\d) hour(s|) ago') DAYS_AGO = re.compile('(\d) day(s|) ago') MONTHS_AGO = re.compile('(\d) month(s|) ago') class Approximation(Enum): UPPER = auto() LOWER = auto() # To avoid possible time shift issue in community post date string and pagination. MOST_TIME_SHIFT = timedelta(days = 1) def getTimeDelta(timeDeltaStr, approximation): hoursAgoMatch = HOURS_AGO.match(timeDeltaStr) relativeOffsetUnit = -1 if approximation is Approximation.UPPER else 1 relativeOffset = relativeOffsetUnit * MOST_TIME_SHIFT if hoursAgoMatch is not None: myTimedelta = timedelta(hours = int(hoursAgoMatch[1]) + relativeOffsetUnit) daysAgoMatch = DAYS_AGO.match(timeDeltaStr) if daysAgoMatch is not None: myTimedelta = timedelta(days = int(daysAgoMatch[1]) + relativeOffsetUnit) monthsAgoMatch = MONTHS_AGO.match(timeDeltaStr) #if monthsAgoMatch is not None: # Between 28 and 31. # myTimedelta = timedelta(days = 31 * int(monthsAgoMatch[1]) + relativeOffsetUnit) return int((myTimedelta + relativeOffset).total_seconds()) communityPosts = [] params = { 'part': 'community', 'id': CHANNEL_ID, } while True: data = getApi('channels', params) item = data['items'][0] for communityPost in item['community']: communityPosts += [{ 'id': communityPost['id'], 'date': communityPost['date'], }] if not 'nextPageToken' in item: break params['pageToken'] = item['nextPageToken'] currentTimestamp = time.time() currentTimestampCeil = math.ceil(currentTimestamp) def printBound(type_, bound): print(f'{type_} bound: {datetime.fromtimestamp(bound)}') def getCommunityPost(timestamp, approximation): shift = 1 while True: community = getCommunity(timestamp + (1 if approximation is Approximation.UPPER else -1) * shift) content = community['continuationContents']['itemSectionContinuation']['contents'][0] if 'messageRenderer' in content and content['messageRenderer']['text']['runs'][0]['text'] == "This channel hasn't posted yet": #if shift == 1: # print('2 community posts missing in a row!') #print(f'{shift=}') shift *= 2 continue return community['continuationContents']['itemSectionContinuation']['contents'][0]['backstagePostThreadRenderer']['post']['backstagePostRenderer']['postId'] def printCommunityPost(type_, communityPost): print(f'{type_} community post: {communityPost}') ## def dichotomy(upper, lower): for type_, bound in [('Upper', upper['timestamp']), ('Lower', lower['timestamp'])]: printBound(type_, bound) print('Seconds in range', upper['timestamp'] - lower['timestamp']) upperCommunityPost = getCommunityPost(upper['timestamp'], Approximation.UPPER) lowerCommunityPost = getCommunityPost(lower['timestamp'], Approximation.LOWER) for type_, communityPost in [('Upper', upperCommunityPost), ('Lower', lowerCommunityPost)]: printCommunityPost(type_, communityPost) print() middle = (upper['timestamp'] + lower['timestamp']) // 2 communityPost = getCommunityPost(middle, Approximation.UPPER) printBound('Middle', middle) printCommunityPost('Middle', communityPost) print('Decrease upper' if communityPost == upper["id"] else 'Increase lower') (upper if communityPost == upper['id'] else lower)['timestamp'] = middle print() time.sleep(5) dichotomy(upper, lower) dichotomy( { 'id': communityPosts[0]['id'], 'timestamp': currentTimestampCeil, }, { 'id': communityPosts[1]['id'], 'timestamp': math.floor(currentTimestamp) - getTimeDelta(communityPosts[0]['date'], Approximation.LOWER), } ) ```
Example output: ``` Upper bound: 2024-08-04 20:14:07 Lower bound: 2024-08-03 16:14:06 Seconds in range 100801 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 06:14:06 Middle community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Increase lower Upper bound: 2024-08-04 20:14:07 Lower bound: 2024-08-04 06:14:06 Seconds in range 50401 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 13:14:06 Middle community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Increase lower Upper bound: 2024-08-04 20:14:07 Lower bound: 2024-08-04 13:14:06 Seconds in range 25201 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 16:44:06 Middle community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Increase lower Upper bound: 2024-08-04 20:14:07 Lower bound: 2024-08-04 16:44:06 Seconds in range 12601 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 18:29:06 Middle community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Decrease upper Upper bound: 2024-08-04 18:29:06 Lower bound: 2024-08-04 16:44:06 Seconds in range 6300 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 17:36:36 Middle community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Decrease upper Upper bound: 2024-08-04 17:36:36 Lower bound: 2024-08-04 16:44:06 Seconds in range 3150 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 17:10:21 Middle community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Decrease upper Upper bound: 2024-08-04 17:10:21 Lower bound: 2024-08-04 16:44:06 Seconds in range 1575 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 16:57:13 Middle community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Increase lower Upper bound: 2024-08-04 17:10:21 Lower bound: 2024-08-04 16:57:13 Seconds in range 788 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 17:03:47 Middle community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Decrease upper Upper bound: 2024-08-04 17:03:47 Lower bound: 2024-08-04 16:57:13 Seconds in range 394 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 17:00:30 Middle community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Increase lower Upper bound: 2024-08-04 17:03:47 Lower bound: 2024-08-04 17:00:30 Seconds in range 197 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 17:02:08 Middle community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Increase lower Upper bound: 2024-08-04 17:03:47 Lower bound: 2024-08-04 17:02:08 Seconds in range 99 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxS308zk_VlD2qQOFrlCGLJd5YbtaYBb9F Middle bound: 2024-08-04 17:02:57 Middle community post: UgkxDyufNHpui0LjnrMzN7FKdBhivqVH2bhX Increase lower Upper bound: 2024-08-04 17:03:47 Lower bound: 2024-08-04 17:02:57 Seconds in range 50 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxDyufNHpui0LjnrMzN7FKdBhivqVH2bhX Middle bound: 2024-08-04 17:03:22 Middle community post: Ugkx0W2h6WitH_Wgrc1PzJOC4H3wF4S9RaBc Increase lower Upper bound: 2024-08-04 17:03:47 Lower bound: 2024-08-04 17:03:22 Seconds in range 25 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: Ugkx0W2h6WitH_Wgrc1PzJOC4H3wF4S9RaBc Middle bound: 2024-08-04 17:03:34 Middle community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Decrease upper Upper bound: 2024-08-04 17:03:34 Lower bound: 2024-08-04 17:03:22 Seconds in range 12 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: Ugkx0W2h6WitH_Wgrc1PzJOC4H3wF4S9RaBc Middle bound: 2024-08-04 17:03:28 Middle community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Increase lower Upper bound: 2024-08-04 17:03:34 Lower bound: 2024-08-04 17:03:28 Seconds in range 6 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: UgkxyMVojoV7RJJSlwerTim3Viv6ru1YuUMU Middle bound: 2024-08-04 17:03:31 Middle community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Increase lower Upper bound: 2024-08-04 17:03:34 Lower bound: 2024-08-04 17:03:31 Seconds in range 3 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Middle bound: 2024-08-04 17:03:32 Middle community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Increase lower Upper bound: 2024-08-04 17:03:34 Lower bound: 2024-08-04 17:03:32 Seconds in range 2 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Middle bound: 2024-08-04 17:03:33 Middle community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Increase lower Upper bound: 2024-08-04 17:03:34 Lower bound: 2024-08-04 17:03:33 Seconds in range 1 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Middle bound: 2024-08-04 17:03:33 Middle community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Increase lower Upper bound: 2024-08-04 17:03:34 Lower bound: 2024-08-04 17:03:33 Seconds in range 1 Upper community post: UgkxyQX6cJcKVEPItVA3rjwpwMUK6YejXTsI Lower community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Middle bound: 2024-08-04 17:03:33 Middle community post: Ugkx2ogkY_YjBtLZ2lxkIvQoaezp7MIRWtLz Increase lower ```

So a precision of 3 seconds, while knowing that manually clicked on POST, have not taken into account sending the request and YouTube treating the community post creation, it seems quite perfect.

ghost commented 3 months ago

True boss

Benjamin-Loison commented 3 months ago

Following my Discord message about me testing on macOS.

Related to Benjamin_Loison/MacOS/issues/1.

See Benjamin_Loison/blackboxprotobuf/issues/1.

TLDR: use Pypi: bbpb.

Investigation details: I updated macOS thanks to VNC: ![image](https://github.com/user-attachments/assets/735244af-59ad-4edb-8549-5f20aed39415) `requirements.txt`: ``` requests blackboxprotobuf ``` ```zsh python3 -m pip install -r requirements.txt ``` ```zsh python3 get_most_recent_community_post_precise_creation_date.py ```
Output: ``` /Users/m1/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( Upper bound: 2024-08-06 00:15:31 Lower bound: 2024-08-03 00:15:30 Seconds in range 259201 Traceback (most recent call last): File "/Users/m1/get_most_recent_community_post_precise_creation_date.py", line 203, in dichotomy( File "/Users/m1/get_most_recent_community_post_precise_creation_date.py", line 187, in dichotomy upperCommunityPost = getCommunityPost(upper['timestamp'], Approximation.UPPER) File "/Users/m1/get_most_recent_community_post_precise_creation_date.py", line 168, in getCommunityPost community = getCommunity(timestamp + (1 if approximation is Approximation.UPPER else -1) * shift) File "/Users/m1/get_most_recent_community_post_precise_creation_date.py", line 29, in getCommunity one = getBase64Protobuf(message, typedef) File "/Users/m1/get_most_recent_community_post_precise_creation_date.py", line 15, in getBase64Protobuf data = blackboxprotobuf.encode_message(message, typedef) File "/Users/m1/Library/Python/3.9/lib/python/site-packages/blackboxprotobuf/lib/interface.py", line 70, in encode_message return blackboxprotobuf.lib.types.length_delim.encode_message(value, message_type) File "/Users/m1/Library/Python/3.9/lib/python/site-packages/blackboxprotobuf/lib/types/length_delim.py", line 56, in encode_message if info['name'] == field_number and field_number != '': KeyError: 'name' ```
Related to [Benjamin_Loison/requests/issues/1](https://codeberg.org/Benjamin_Loison/requests/issues/1). [Pypi: blackboxprotobuf](https://pypi.org/project/blackboxprotobuf/) [ydkhatri/blackboxprotobuf](https://github.com/ydkhatri/blackboxprotobuf) [ydkhatri/blackboxprotobuf/blob/a67018eae032097e1fc153d69d4ae577046077c7/blackboxprotobuf/lib/interface.py#L70](https://github.com/ydkhatri/blackboxprotobuf/blob/a67018eae032097e1fc153d69d4ae577046077c7/blackboxprotobuf/lib/interface.py#L70) [ydkhatri/blackboxprotobuf/blob/a67018eae032097e1fc153d69d4ae577046077c7/blackboxprotobuf/lib/types/length_delim.py#L56](https://github.com/ydkhatri/blackboxprotobuf/blob/a67018eae032097e1fc153d69d4ae577046077c7/blackboxprotobuf/lib/types/length_delim.py#L56)
Benjamin-Loison commented 2 months ago

Note that #257 pagination seems to provide precise date of first or last community post of each page. Maybe by finding the precise date of each first page community posts, thanks to pagination can get the ones of all.