xenova / chat-downloader

A simple tool used to retrieve chat messages from livestreams, videos, clips and past broadcasts. No authentication needed!

https://chat-downloader.readthedocs.io/

MIT License

953 stars 133 forks source link

[BUG] Downloader thinks it has finished retrieving chat on the second request on member-only streams #76

Closed ghost closed 3 years ago

ghost commented 3 years ago

Basic information

Program version: 0.0.8
Python version: 3.6.9
Operating system: Ubuntu

Describe the bug

I'm trying to get the livechat from an ongoing member stream. The downloader outputs to stderr [INFO] Finished retrieving chat. after the second message batch, then the program closes.

I'm not yet sure if member-only is the cause, I'll keep trying on subsequent member-only streams I have access to.

Command/Code used

If running from the command line, provide the following:

The command used (including the verbose tag, -v):
```
chat_downloader "$URL" --cookies cookies.txt -v
```

Output from the above command:


[DEBUG] Python version: 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0]
[DEBUG] Program version: 0.0.8
[DEBUG] Initialisation parameters: {'cookies': 'tl_bot/cookies.txt', 'proxy': None, 'hea
ders': None}
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'buffer_size': 4096, 'message_receive_timeout': 0.1, 'chat_
type': 'live', 'format_file': None, 'format': 'youtube', 'message_types': None, 'message
_groups': ['messages'], 'exit_on_debug': False, 'pause_on_debug': False, 'logging': 'deb
ug', 'max_messages': None, 'timeout': None, 'retry_timeout': None, 'max_attempts': 15, '
end_time': None, 'start_time': None, 'url': 'https://youtu.be/33LME76okKw', 'inactivity_
timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=33LME76okKw HTTP/1.1" 200 None
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_mes
sages at 0x7f5b27cf94c0>, 'title': '【 メン限 】手術後初のピアノのお時間なのら🎹🎵【姫森
ルーナ/ホロライブ】', 'duration': 0, 'is_live': True, 'start_time': 1614006016703036.0,
'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x7f5b27d3a860>,
'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x7f5b29816950>}
[INFO] Retrieving chat for "【 メン限 】手術後初のピアノのお時間なのら🎹🎵【姫森ルーナ/
ロライブ】".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZ
VTmhPVmsxTjJkbVpWa3dXbkp2WDI1dlNGSldjbTUzRWdzek0weE5SVGMyYjJ0TGR4b1Q2cWpkdVFFTkNnc3pNMHh
OUlRjMmIydExkeUFCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None

messages omitted

[DEBUG] Total number of messages: 75 [DEBUG] Sleeping for 10000ms. [DEBUG] Continuation: 0ofMyAOkARpYQ2lrcUp3b1lWVU5oT1ZrMU4yZG1aVmt3V25KdlgyNXZTRkpXY201M0 Vnc3pNMHhOUlRjMmIydExkeG9UNnFqZHVRRU5DZ3N6TTB4TlJUYzJiMnRMZHlBQijv_Jbl5v3uAjAAOABAAkoVCA AQABgBIABQ9pfe5eb97gJYA3gAUKSeveXm_e4CWPnkpZ34_O4CggECCAGIAQCgAYyI5OXm_e4C [DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaS yAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None [INFO] Finished retrieving chat. [DEBUG] Session closed.

3 to 15 messages



## Expected behavior
Downloader keeps outputting new messages as long as the stream is live.

xenova commented 3 years ago

That is quite strange. YouTube recently updated their API endpoint. Member-only streams were working before then, so my best guess is that the parameters sent to get the next continuation is missing. This also aligns with the fact that the first retrieval is successful, but the next crashes.

continuation_params = {
    'context': {
        'client': {
            'visitorData': visitor_data,
            'userAgent': self.get_session_headers('User-Agent'),
            'clientName': 'WEB',
            'clientVersion': '2.{}.01.00'.format(datetime.today().strftime('%Y%m%d'))
        }
    }
}

As an example, this request may be missing some authentication cookies. I will continue to investigate, but this might help you debug as well. Thanks for filing the issue! Hopefully we can get this sorted out as soon as possible. :)

xenova commented 3 years ago

Could you try download the source code, and remove lines 1041 and 1042 in https://github.com/xenova/chat-downloader/blob/master/chat_downloader/sites/youtube.py

https://github.com/xenova/chat-downloader/blob/0f1b1326eec9fb45031d7fd58e0a4a9dd2297d5d/chat_downloader/sites/youtube.py#L1041-L1042

This data passed might be corrupting the request when cookies are used. The program seems to work fine without them.

ghost commented 3 years ago

Hey, I'll do that next time I'm in a member stream.

I'm also going to check what happens for archived member streams as soon as I'm capable.

I'll also look at the network inspector and see if I find anything.

ghost commented 3 years ago

Haven't had problems with archived member streams so I'll look into it further next time I catch one live.

xenova commented 3 years ago

Hey, I'll do that next time I'm in a member stream.

I'm also going to check what happens for archived member streams as soon as I'm capable.

I'll also look at the network inspector and see if I find anything.

That would be amazing! Thank you!

xenova commented 3 years ago

Haven't had problems with archived member streams so I'll look into it further next time I catch one live.

Oh wow... okay, that's interesting. I wonder what the difference is? I'll see if I can add some debugging info which may be able to help.

ghost commented 3 years ago

Another member stream is happening right now so I can report that same problem occurs, even after removing the two lines you suggested.

Here a request taken from the network inspector. I wouldn't know what to make of it right now, not yet familiar with the way this API works.

curl 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8' \
  -H 'authority: www.youtube.com' \
  -H 'x-origin: https://www.youtube.com' \
  -H 'authorization: SAPISIDHASH ###' \
  -H 'content-type: application/json' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36' \
  -H 'x-youtube-client-name: 1' \
  -H 'x-youtube-client-version: 2.20210221.08.00' \
  -H 'x-goog-authuser: 0' \
  -H 'x-goog-pageid: 118190112336746196515' \
  -H 'x-goog-visitor-id: CgtFNVhMV00xXzlOSSjphtSBBg%3D%3D' \
  -H 'accept: */*' \
  -H 'origin: https://www.youtube.com' \
  -H 'x-client-data: CJW2yQEIpLbJAQipncoBCJesygEI+MfKAQikzcoBCMvPygEI3NXKAQjih8sBCKicywEIxZzLAQjjnMsBCKidywEY2cPKAQ==' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-fetch-mode: same-origin' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://www.youtube.com/live_chat?continuation=0ofMyANxGlhDaWtxSndvWVZVTmhPVmsxTjJkbVpWa3dXbkp2WDI1dlNGSldjbTUzRWd0M1dIWjBlVlZLY2pscVRSb1Q2cWpkdVFFTkNndDNXSFowZVZWS2NqbHFUU0FCMAGCAQIIBIgBAaABm7TY1o6A7wKyAQA%253D' \
  -H 'accept-language: en-GB,en;q=0.9,en-US;q=0.8,fr;q=0.7' \
  -H 'cookie: CONSENT=###; SID=###; __Secure-3PSID=###; HSID=###; SSID=###; APISID=###; SAPISID=###; __Secure-3PAPISID=###; YSC=###; VISITOR_INFO1_LIVE=###; PREF=###; LOGIN_INFO=###; wide=###; SIDCC=###; __Secure-3PSIDCC=###' \
  --data-raw '{"context":{"client":{"hl":"en-GB","gl":"FR","remoteHost":"###","deviceMake":"","deviceModel":"","visitorData":"CgtFNVhMV00xXzlOSSjphtSBBg%3D%3D","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36,gzip(gfe)","clientName":"WEB","clientVersion":"2.20210221.08.00","osName":"X11","osVersion":"","originalUrl":"https://www.youtube.com/live_chat?continuation=0ofMyANxGlhDaWtxSndvWVZVTmhPVmsxTjJkbVpWa3dXbkp2WDI1dlNGSldjbTUzRWd0M1dIWjBlVlZLY2pscVRSb1Q2cWpkdVFFTkNndDNXSFowZVZWS2NqbHFUU0FCMAGCAQIIBIgBAaABm7TY1o6A7wKyAQA%253D","platform":"DESKTOP","gfeFrontlineInfo":"vip=142.250.179.78,server_port=443,client_port=7366,tcp_connection_request_count=0,header_order=HUARELC,gfe_version=2.708.2,ssl,ssl_info=QUIC:RNA:T,tlsext=S,sni=www.youtube.com,pn=quic,rtt_source=tcp,rtt=615,srtt=615,bw=4277,client_protocol=h2,client_transport=quic,gfe=acparh8.prod.google.com,pzf=n/a,vip_region=default,asn=25117,cc=FR,eid=aQM1YJmHLoj78wfo4YHwBg,scheme=https","clientFormFactor":"UNKNOWN_FORM_FACTOR","timeZone":"Europe/Paris","browserName":"Chrome","browserVersion":"88.0.4324.182","screenWidthPoints":425,"screenHeightPoints":565,"screenPixelDensity":1,"screenDensityFloat":1,"utcOffsetMinutes":60,"userInterfaceTheme":"USER_INTERFACE_THEME_LIGHT","connectionType":"CONN_CELLULAR_3G","mainAppWebInfo":{"graftUrl":"https://www.youtube.com/live_chat?continuation=0ofMyANxGlhDaWtxSndvWVZVTmhPVmsxTjJkbVpWa3dXbkp2WDI1dlNGSldjbTUzRWd0M1dIWjBlVlZLY2pscVRSb1Q2cWpkdVFFTkNndDNXSFowZVZWS2NqbHFUU0FCMAGCAQIIBIgBAaABm7TY1o6A7wKyAQA%253D","webDisplayMode":"WEB_DISPLAY_MODE_BROWSER"}},"user":{"lockedSafetyMode":false},"request":{"useSsl":true,"internalExperimentFlags":[],"consistencyTokenJars":[]},"clickTracking":{"clickTrackingParams":"IhMI2IHS2o6A7wIVSj3xBR3nFAEf"},"clientScreenNonce":"MC45MzQwMjU4MDM5NDc4Mzcx","adSignalsInfo":{"params":[{"key":"dt","value":"1614087019199"},{"key":"flash","value":"0"},{"key":"frm","value":"1"},{"key":"u_tz","value":"60"},{"key":"u_his","value":"10"},{"key":"u_java","value":"false"},{"key":"u_h","value":"1080"},{"key":"u_w","value":"1920"},{"key":"u_ah","value":"1063"},{"key":"u_aw","value":"1920"},{"key":"u_cd","value":"24"},{"key":"u_nplug","value":"3"},{"key":"u_nmime","value":"4"},{"key":"bc","value":"31"},{"key":"bih","value":"941"},{"key":"biw","value":"355"},{"key":"brdim","value":"679,36,679,36,1920,17,1227,1026,425,565"},{"key":"vis","value":"1"},{"key":"wgl","value":"true"},{"key":"ca_type","value":"image"}]}},"continuation":"0ofMyAOsARpYQ2lrcUp3b1lWVU5oT1ZrMU4yZG1aVmt3V25KdlgyNXZTRkpXY201M0VndDNXSFowZVZWS2NqbHFUUm9UNnFqZHVRRU5DZ3QzV0haMGVWVktjamxxVFNBQiiV97LnjoDvAjAAOABAAkoVCAAQABgBIABQw5nU2o6A7wJYA3gAUJ7l6OeOgO8CWOPiz9H3_e4CggECCASIAQCaAQIIAKABz96K6I6A7wKyAQA%3D","webClientInfo":{"isDocumentHidden":false}}' \
  --compressed

xenova commented 3 years ago

Thank you! This will definitely help in debugging.

Could you perhaps clone the latest version of the https://github.com/xenova/chat-downloader/tree/docs branch by:

git clone https://github.com/xenova/chat-downloader.git
cd chat-downloader
python setup.py install
python -m chat_downloader -v "$URL" --cookies cookies.txt

This should show some additional debugging information as well as why nothing is returned.

ghost commented 3 years ago

$ ./chat_downloader https://youtu.be/wXvtyUJr9jM --cookies cookies.txt -v
[DEBUG] Python version: 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0]
[DEBUG] Program version: 0.0.8
[DEBUG] Initialisation parameters: {'cookies': '/home/alpha/bots/tl_bot/cookies.txt', 'proxy': None, 'headers
': None}
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'buffer_size': 4096, 'message_receive_timeout': 0.1, 'chat_type': 'live', 'forma
t_file': None, 'format': 'youtube', 'message_types': None, 'message_groups': ['messages'], 'exit_on_debug': F
alse, 'pause_on_debug': False, 'logging': 'debug', 'max_messages': None, 'timeout': None, 'retry_timeout': No
ne, 'max_attempts': 15, 'end_time': None, 'start_time': None, 'url': 'https://youtu.be/wXvtyUJr9jM', 'inactiv
ity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=wXvtyUJr9jM HTTP/1.1" 200 None
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x7f4e18f635
70>, 'title': '【 メンバー限定 】作戦会議！ルーナイト集合なのら(・o・🍬)【姫森ルーナ/ホロライブ】', 'duration
': 0, 'is_live': True, 'start_time': 1614085205393908.0, 'site': <chat_downloader.sites.youtube.YouTubeChatDo
wnloader object at 0x7f4e197de9b0>, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x7f4e1a
837ea0>}
[INFO] Retrieving chat for "【 メンバー限定 】作戦会議！ルーナイト集合なのら(・o・🍬)【姫森ルーナ/ホロライブ
】".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTmhPVmsxTjJkbVpWa3dX
bkp2WDI1dlNGSldjbTUzRWd0M1dIWjBlVlZLY2pscVRSb1Q2cWpkdVFFTkNndDNXSFowZVZWS2NqbHFUU0FCMAGCAQIIAQ%3D%3D HTTP/1.1
" 200 None
# 75 messages
[DEBUG] Total number of messages: 75
[DEBUG] Sleeping for 10000ms.
[DEBUG] Continuation: 0ofMyAOkARpYQ2lrcUp3b1lWVU5oT1ZrMU4yZG1aVmt3V25KdlgyNXZTRkpXY201M0VndDNXSFowZVZWS2NqbHFUUm9UNnFqZHVRRU5DZ3QzV0haMGVWVktjamxxVFNBQiiA3oz8l4DvAjAAOABAAkoVCAAQABgBIABQn6Xm_JeA7wJYA3gAUOKuxfyXgO8CWOPiz9H3_e4CggECCAGIAQCgAfvf6fyXgO8C
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[INFO] Finished retrieving chat.
[DEBUG] Session closed.
# 7 messages

I've also run git checkout docs before building. Just from a quick check I don't think there's any more information... (I might not have built it right... I'll try again.)

xenova commented 3 years ago

Oh you're right I forgot to specify the branch above. The above output does not user the latest branch (the continuation should print a dictionary)

It might be simpler to just downloader the zip (from https://github.com/xenova/chat-downloader/archive/docs.zip), extract it and run it from within there. Run using:

cd chat-downloader-docs
python -m chat_downloader https://youtu.be/wXvtyUJr9jM --cookies cookies.txt -v

ghost commented 3 years ago

[DEBUG] Created YouTubeChatDownloader session.
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://youtu.be/wXvtyUJr9jM', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'testing': False, 'verbose': True, 'quiet': False, 'message_groups': ['messages'], 'message_types': None, 'output': None, 'overwrite': False, 'sort_keys': True, 'indent': 4, 'format_file': None, 'chat_type': 'live', 'ignore': None, 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'format': 'youtube', 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=wXvtyUJr9jM HTTP/1.1" 200 None
[DEBUG] Match found: "<re.Match object; span=(0, 28), match='https://youtu.be/wXvtyUJr9jM'>". Running "_get_chat_by_video_id" function in "YouTubeChatDownloader".
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x7f3e6e1b9970>, 'callback': None, 'title': '【 メンバー限定 】作戦会議！ルーナイト集合なのら(・o・🍬)【姫森ルーナ/ホロライブ】', 'duration': 0, 'is_live': True, 'start_time': 1614085205393908.0, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x7f3e6e18b370>, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x7f3e6e18d280>}
[INFO] Retrieving chat for "【 メンバー限定 】作戦会議！ルーナイト集合なのら(・o・🍬)【姫森ルーナ/ホロライブ】".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTmhPVmsxTjJkbVpWa3dXbkp2WDI1dlNGSldjbTUzRWd0M1dIWjBlVlZLY2pscVRSb1Q2cWpkdVFFTkNndDNXSFowZVZWS2NqbHFUU0FCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
# 73 messages
[DEBUG] Total number of messages: 73
[DEBUG] Sleeping for 8000ms.
[DEBUG] Continuation params: {'context': {'client': {'clientName': 'WEB', 'clientVersion': '2.20210223.01.00'}}, 'continuation': '0ofMyAOkARpYQ2lrcUp3b1lWVU5oT1ZrMU4yZG1aVmt3V25KdlgyNXZTRkpXY201M0VndDNXSFowZVZWS2NqbHFUUm9UNnFqZHVRRU5DZ3QzV0haMGVWVktjamxxVFNBQij8yIfUnYDvAjAAOABAAkoVCAAQABgBIABQye3W1J2A7wJYA3gAUJWuttSdgO8CWOPiz9H3_e4CggECCAGIAQCgAauF3NSdgO8C'}
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] No continuation information found: {'responseContext': {'visitorData': 'Cgs4Q2piZ3FEZ1d0WSiiptSBBg%3D%3D', 'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210223.01.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0x04944a67b9f257e4'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '23934970,23994474,23891344,23857948,23983797,23968386,23987676,1714245,23839597,23944779,23891347,23986031,23890959,23974595,23744176,23882685,23976578,23994373,24000882,23969934,23992809,23966208,23970529,23991264,23804281,23996751,23748147,23918597,23974884,23884386,23946420'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210223'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'trackingParams': 'CAAQ0b4BIhMI5LH82Z2A7wIVDua0Ch2FrAAj', 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}, 'trackingParams': 'CAEQljsiEwjksfzZnYDvAhUO5rQKHYWsACM='}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.
# 2 messages

There we go. Interestingly, I got this error message earlier by watching the stream on my other (but linked) Google account. I could see the stream, but not the chat.

1614091269

xenova commented 3 years ago

Thanks again! This will definitely help debug.

xenova commented 3 years ago

Okay - I think I might have added something which could fix it. 81f5d25

Could you try update (the same way as before) and then try again?

ghost commented 3 years ago

I will do when I get the chance!

ghost commented 3 years ago

Here's the current log!

$ chat_downloader https://www.youtube.com/watch\?v\=poXJ9tiHiGc --cookies ~/cookies.txt -v
[DEBUG] Created YouTubeChatDownloader session.
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://www.youtube.com/watch?v=poXJ9tiHiGc', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'testing': False, 'verbose': True, 'quiet': False, 'message_groups': ['messages'], 'message_types': None, 'output': None, 'overwrite': True, 'sort_keys': True, 'indent': 4, 'format_file': None, 'chat_type': 'live', 'ignore': None, 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'format': 'youtube', 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=poXJ9tiHiGc HTTP/1.1" 200 None
[DEBUG] Match found: "<re.Match object; span=(0, 43), match='https://www.youtube.com/watch?v=poXJ9tiHiGc'>". Running "_get_chat_by_video_id" function in "YouTubeChatDownloader".
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x7f774d51d510>, 'callback': None, 'title': '【 映画鑑賞会＊メン限 】「アイアンマン IRON MAN」一緒に観ようなのら🎬✨【姫森ルーナ/ホロライブ】', 'duration': 0, 'is_live': True, 'start_time': 1615467607455174.0, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x7f774d4cc550>, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x7f774d4ed3d0>}
[INFO] Retrieving chat for "【 映画鑑賞会＊メン限 】「アイアンマン IRON MAN」一緒に観ようなのら🎬✨【姫森ルーナ/ホロライブ】".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTmhPVmsxTjJkbVpWa3dXbkp2WDI1dlNGSldjbTUzRWd0d2IxaEtPWFJwU0dsSFl4b1Q2cWpkdVFFTkNndHdiMWhLT1hScFNHbEhZeUFCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
// messages here
[DEBUG] Total number of messages: 75
[DEBUG] Sleeping for 8000ms.
[DEBUG] Continuation: 0ofMyAOiARpYQ2lrcUp3b1lWVU5oT1ZrMU4yZG1aVmt3V25KdlgyNXZTRkpXY201M0VndHdiMWhLT1hScFNHbEhZeG9UNnFqZHVRRU5DZ3R3YjFoS09YUnBTR2xIWXlBQii_6K_GqajvAjAAQAJKFQgAEAAYASAAUKmhucepqO8CWAN4AFCq2tvGqajvAliJx46H_abvAoIBAggBiAEAoAGDlL7HqajvAg%3D%3D
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210308.08.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0xccc13583cf438046'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '23997167,23858057,23891344,24007400,23969934,23996751,24004431,23804281,23986031,23891346,23884386,23970529,23966208,24001374,24003347,23885487,23988103,9407155,24005870,23815352,23944779,23983575,24011120,1714257,23990875,23991263,24008642,23987676,23940238,24006795,24009499,23968386,23882503,23918597,23744176,24000198,23946420,23994474,23983297,24007246,23890959,23857950,23940248,24000063,24008380,23934970,24000112,23974883,23974595'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210308'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.

Lytexx commented 3 years ago

still a problem here is my log


[DEBUG] Python version: 3.9.4 (tags/v3.9.4:1f2e308, Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)]
[DEBUG] Program version: 0.0.8
[DEBUG] Initialisation parameters: {'cookies': 'cookies.txt', 'proxy': None, 'headers': None}
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://www.youtube.com/watch?v=Lr27IVTQUm4', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'message_groups': ['messages'], 'message_types': None, 'format': 'youtube', 'format_file': None, 'chat_type': 'live', 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=Lr27IVTQUm4 HTTP/1.1" 200 None
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x000001F8D4281C10>, 'title': '【MEMBERS ONLY】Madoka Magica Watchalong!', 'duration': 0, 'is_live': True, 'start_time': 1618772464449827.0, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x000001F8D4288EE0>, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x000001F8D4289B80>}
[INFO] Retrieving chat for "【MEMBERS ONLY】Madoka Magica Watchalong!".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTjViREY2TTJwdk0xaElVakZ5YVV4R1MwYzFWVUZuRWd0TWNqSTNTVlpVVVZWdE5Cb1Q2cWpkdVFFTkNndE1jakkzU1ZaVVVWVnROQ0FCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
// messages here
[DEBUG] Total number of messages: 75
[DEBUG] Sleeping for 10000ms.
[DEBUG] Continuation: 0ofMyAOlARpYQ2lrcUp3b1lWVU41YkRGNk0ycHZNMWhJVWpGeWFVeEdTMGMxVlVGbkVndE1jakkzU1ZaVVVWVnROQm9UNnFqZHVRRU5DZ3RNY2pJM1NWWlVVVlZ0TkNBQii7y4vZxIjwAjAAQAJKGAgBEAAYASAAUMGg49nEiPACWAN4AKIBAFCkkLPZxIjwAljszbumvIjwAoIBAggBiAEAoAGI6-nZxIjwAg%3D%3D
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[INFO] Finished retrieving chat.
[DEBUG] Session closed.```

krichbanana commented 3 years ago

xenova commented 3 years ago

Likely related: yt-dlp/yt-dlp#433

Thanks! I'll check it out and implement this for the next update.

xenova commented 3 years ago

@krichbanana @Lytexx @tam-carre This branch (https://github.com/xenova/chat-downloader/tree/member-fix) should implement what yt-dlp did.

Could one of you confirm that this works? I am unfortunately unable to test this for myself as I am not a member to anyone on YouTube. Thanks!

Lytexx commented 3 years ago

Could one of you confirm that this works? I am unfortunately unable to test this for myself as I am not a member to anyone on YouTube. Thanks!

@xenova sadly doesn't seem to work

[DEBUG] Total number of messages: 75
[DEBUG] Sleeping for 8000ms.
[DEBUG] Continuation: 0ofMyAOjARpYQ2lrcUp3b1lWVU5OZDBkSVVqQkNWRnAxVEhOdGFsbGZUbFExVUhkbkVndExVazl4TWtKc04wUXpaeG9UNnFqZHVRRU5DZ3RMVWs5eE1rSnNOMFF6WnlBQijro_KfusXxAjAAQAJKFggBGAEgAFDG65qmusXxAlgDeACiAQBQ65ySoLrF8QJY4pSl6KHF8QKCAQIIAYgBAKAB74WlprrF8QI%3D
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210701.07.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0x29add7d49c35cbf9'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '24034796,24012512,24017848,23884386,24049577,24056839,23999405,24050957,24022729,23940238,24062573,23857949,23891346,23990875,24063702,23966208,24058293,24030040,24058128,24042870,24001373,24045214,24056265,23804281,23882685,24036948,23983296,24060199,23991737,24007246,24045411,24049820,24059521,23998056,24049567,24004644,24044124,24023960,24027649,24037794,23973490,23946420,24049569,23968386,23890959,24046872,24052245,23918597,24043961,24038425,23983813,24058812,24059898,23996830,24058380,23934970,1714251,23891344,24060795,24053866,24050630,9405963,24003105,23974595,23748146,23944779,24049573,24010337,24058240,24003586,24063016,24003103,23744176,24063146,24049575'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210701'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.

xenova commented 3 years ago

Hmmm... Let me try again.

xenova commented 3 years ago

Just to confirm, this only happens for member-only live streams, not replays?

Edit: Okay I think I have found the problem.

xenova commented 3 years ago

Okay, this should hopefully do the trick: https://github.com/xenova/chat-downloader/commit/b43b823b749d21edbb04e8ed3832b91b7ea8ffe9

Could you give it another go? 😃

Lytexx commented 3 years ago

@xenova still doesn't work ;w; maybe i'm doing this wrong? here is my full console log


C:\Users\Lytexx\Downloads>git clone https://github.com/xenova/chat-downloader
Cloning into 'chat-downloader'...
remote: Enumerating objects: 2088, done.
remote: Counting objects: 100% (263/263), done.
remote: Compressing objects: 100% (182/182), done. eceiving objects:   1% (21/2088)
remote: Total 2088 (delta 158), reused 160 (delta 76), pack-reused 1825
Receiving objects: 100% (2088/2088), 3.99 MiB | 7.31 MiB/s, done.
Resolving deltas: 100% (1391/1391), done.

C:\Users\Lytexx\Downloads>cd chat-downloader

C:\Users\Lytexx\Downloads\chat-downloader>git switch member-fix
Switched to a new branch 'member-fix'
Branch 'member-fix' set up to track remote branch 'member-fix' from 'origin'.

C:\Users\Lytexx\Downloads\chat-downloader>git pull
Already up to date.

C:\Users\Lytexx\Downloads\chat-downloader>py -m chat_downloader -v https://www.youtube.com/watch?v=KROq2Bl7D3g -c "C:\Users\Lytexx\Downloads\cookies.txt"
[DEBUG] Created YouTubeChatDownloader session.
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://www.youtube.com/watch?v=KROq2Bl7D3g', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'testing': False, 'verbose': True, 'quiet': False, 'message_groups': ['messages'], 'message_types': None, 'output': None, 'overwrite': True, 'sort_keys': True, 'indent': 4, 'format_file': None, 'chat_type': 'live', 'ignore': None, 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'format': 'youtube', 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=KROq2Bl7D3g HTTP/1.1" 200 None
[DEBUG] Match found: "<re.Match object; span=(0, 43), match='https://www.youtube.com/watch?v=KROq2Bl7D3g'>". Running "_get_chat_by_video_id" function in "YouTubeChatDownloader".
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x0000020695EDADD0>, 'callback': None, 'title': "【Member Only】 Char's Counterattack + Hathaway's Flash Watchalong", 'duration': 0, 'is_live': True, 'start_time': 1625266835490153.0, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x0000020695F381F0>, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x0000020695EFF0D0>}
[INFO] Retrieving chat for "【Member Only】 Char's Counterattack + Hathaway's Flash Watchalong".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTk5kMGRJVWpCQ1ZGcDFUSE50YWxsZlRsUTFVSGRuRWd0TFVrOXhNa0pzTjBRelp4b1Q2cWpkdVFFTkNndExVazl4TWtKc04wUXpaeUFCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
2021-07-03 01:09:10 | (Member (6 months)) tomacas: not friendly, but maybe respectful?
2021-07-03 01:09:11 | (Member (6 months)) Manny O: Respected enemies
2021-07-03 01:09:11 | (Member (2 months)) The MD Goatman: amuro should be dating chars sister
2021-07-03 01:09:11 | (Member (6 months)) NoctusTerrin: :_WWW::_AAA::_HHH: Just popped in to say hi Ina-sama! Got some work I need to tend to! Have a good stream!
2021-07-03 01:09:11 | (Member (2 months)) RCSI: They're not always trying to kill each other
2021-07-03 01:09:11 | (Member (6 months)) OK Kei: aahh
2021-07-03 01:09:12 | (Member (1 month)) Angus Ho: two words: quattro bajeena
2021-07-03 01:09:12 | (Member (6 months)) DaGreatRV0: suddenly, colonydrop
2021-07-03 01:09:12 | (Member (6 months)) jyeh0429: They are frienimies
2021-07-03 01:09:13 | (Member (1 month)) Moribundo: I feel like it'll all come back to me once I start watching, maybe
2021-07-03 01:09:13 | (Member (6 months)) Oğuzhan: I wouldnt say they are friendly
2021-07-03 01:09:13 | (Member (6 months)) shallistera th.: humu
2021-07-03 01:09:13 | (Member (6 months)) AkaRed10: "friendly"
2021-07-03 01:09:14 | (Member (2 months)) YAMI: those the lalaa thing is....
2021-07-03 01:09:14 | (Member (6 months)) Black Knight: PekoMiko dynamic
2021-07-03 01:09:14 | (Member (6 months)) KeimaKatsu: :_HUMU::_HUMU:
2021-07-03 01:09:15 | (Member (1 month)) Mandricardo Simp: Both of them aren't over Lalah >>
2021-07-03 01:09:15 | (New member) clone shark: they understand each other even though they dont agree
2021-07-03 01:09:18 | (New member) I'N Addicted: :_KDTD::_KDTD::_KDTD:
2021-07-03 01:09:19 | (Member (6 months)) Josh R: To quote tomino "Char and Amuro's rivalry is so intense you'll question if they are straight"
2021-07-03 01:09:20 | (New member) Kenny Nguyen: They both want the same outcome for earth but they go about it differently
2021-07-03 01:09:20 | (Member (1 month)) PerfectPlacenta: are they boyfriends? 😳
2021-07-03 01:09:20 | (Member (6 months)) processr: and remember, Quattro Bajeena is DEFINITELY not Char :_tehepero:
2021-07-03 01:09:20 | (Member (6 months)) zaini ikhwan: Just know that Quatro is not Char:_tehepero:
2021-07-03 01:09:22 | (Member (2 months)) Elmes: RX-93!
2021-07-03 01:09:22 | (Member (6 months)) Camtiy: gundams
2021-07-03 01:09:22 | (New member) Trains Hurt People: Char isn't a newtype which makes him so much cooler
2021-07-03 01:09:22 | (Member (6 months)) Disco Vader: They hate each other, but respect each other's skills
2021-07-03 01:09:23 | (New member) Allen Ryan: amuro did killed char gf
2021-07-03 01:09:23 | (Member (6 months)) James Connor: They understand each other as equals
2021-07-03 01:09:24 | (Member (6 months)) judgegrumble: lots of "tension"
2021-07-03 01:09:24 | (Member (2 months)) RoboChuuni: LOL PekoMiko
2021-07-03 01:09:24 | (Member (2 months)) Zephyer Alter: they are not that friendly
2021-07-03 01:09:24 | (New member) Cristhian Suematsu: They share the same pain.
2021-07-03 01:09:25 | (Member (2 months)) frojoe84: just like you and pikamee?
2021-07-03 01:09:25 | (Member (6 months)) giossan: Oh ok
2021-07-03 01:09:25 | (Member (2 months)) Khift: Can you start with what a gundam is? I'm not clear on that part yet...
2021-07-03 01:09:25 | (Member (2 months)) スカッドリッパー - Scud Ripper: I believe Char does, but I'm pretty sure Amuro straight-up hates the guy.
2021-07-03 01:09:25 | (Member (6 months)) Rapostrophe: Swan song if you may
2021-07-03 01:09:26 | (Member (6 months)) Alrifht, Lord of the Underdeath: The big showdown!
2021-07-03 01:09:26 | (Member (2 months)) Vicente: Oh okay
2021-07-03 01:09:26 | (New member) HoloSheepKnight: Chars midlife crisis the movie
2021-07-03 01:09:26 | (Member (2 months)) Hornet: Double Zeta is a comedy.
2021-07-03 01:09:27 | (Member (6 months)) GFlügel: ... Ina, did you ever draw "Amuro X Char" slashfic art? :_peek::_oii:
2021-07-03 01:09:27 | (Member (2 months)) Acadien: friennemies
2021-07-03 01:09:29 | (Member (2 months)) Myaku: They hate each other cause their views are so opposing but if it weren't for that they would be great friends:_HUMU::_HUMU:
2021-07-03 01:09:29 | (New member) TheFurtuka: char is singlehandedly responsible for the growth of doujin culture
2021-07-03 01:09:29 | (Member (6 months)) Coralld: Very true, they are the epidemy of rivals
2021-07-03 01:09:29 | (Member (6 months)) OK Kei: rivary
2021-07-03 01:09:29 | (Member (6 months)) Sarty: Oh:_OxO:
2021-07-03 01:09:29 | (Member (2 months)) Einsils Valier: Z Gundam is just a big BL to Char and Amuro
2021-07-03 01:09:31 | (New member) WesternTelecom: Do you like the Gundams or Zaku better?
2021-07-03 01:09:32 | (Member (2 months)) TheTrashCan: Theyre lovers
2021-07-03 01:09:32 | (Member (1 month)) Yureio Ch.: I'm assuming you've already seen it before?
2021-07-03 01:09:32 | (New member) Alpha_AJ: Oh boy, im not prepared
2021-07-03 01:09:34 | (Member (2 months)) Porway: :_HUMU::_HUMU:
2021-07-03 01:09:34 | (Member (1 month)) Slugien: OH
2021-07-03 01:09:35 | (Member (2 months)) Voidest: finavalry
2021-07-03 01:09:35 | (New member) spagredo: CCA is such a. weird movie....
2021-07-03 01:09:36 | (Member (2 months)) YAMI: :_WWW::_AAA::_HHH:valry?!
2021-07-03 01:09:36 | (Member (6 months)) jyeh0429: they are like PekoMiko w
2021-07-03 01:09:37 | (Member (6 months)) Gvrchomp: I saw a really good Amuro statue in a shop, I really wanted to get it
2021-07-03 01:09:38 | (Member (6 months)) dc: The "I came here to laugh at you" meme
2021-07-03 01:09:38 | (Member (6 months)) Kajico: Char vs Amuro = Miko vs Pekora
2021-07-03 01:09:39 | (Member (6 months)) KeimaKatsu: glad watching CCA with you for the first time
2021-07-03 01:09:41 | (Member (6 months)) Lyudmilia: it connects to HF
2021-07-03 01:09:42 | (Member (6 months)) Camtiy: i like spider mechs better like from ghost in the shell or 86
2021-07-03 01:09:43 | (Member (6 months)) Doug Rattman: So we're in for a cool robot fight? I'm in.
2021-07-03 01:09:43 | (Member (6 months)) racingmaniacgt1: also Char is really into genocide
2021-07-03 01:09:52 | (Member (2 months)) Langit Daun: this movie is villain side?:_MOGU:
2021-07-03 01:09:52 | (Member (1 month)) Duwan: UC?
2021-07-03 01:09:52 | (New member) Trains Hurt People: Hathaway is THE WORST in CCA
[DEBUG] Total number of messages: 72
[DEBUG] Sleeping for 8000ms.
[DEBUG] Continuation: 0ofMyAOjARpYQ2lrcUp3b1lWVU5OZDBkSVVqQkNWRnAxVEhOdGFsbGZUbFExVUhkbkVndExVazl4TWtKc04wUXpaeG9UNnFqZHVRRU5DZ3RMVWs5eE1rSnNOMFF6WnlBQijrsevswMXxAjAAQAJKFggBGAEgAFCB5bruwMXxAlgDeACiAQBQiMqL7cDF8QJY4pSl6KHF8QKCAQIIAYgBAKABhcDH7sDF8QI%3D
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210701.07.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0xaba7a8add61ca53d'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '24017848,24036948,24059521,24049567,24052245,24044124,24007246,24045411,23973490,24004644,23940238,24023960,24049820,24027649,23882685,24058380,23998056,23983813,24058812,24037794,24060199,23991737,24062573,24063702,23857949,23990875,24053866,24050630,24042870,23804281,23884386,24030040,23999405,24045214,23966208,24058240,23891346,24012512,24022729,23946420,24001373,24049577,24003105,23891344,24063146,24050957,24003586,24063016,24049575,24056839,23974595,23748146,24060795,24056265,23744176,24049573,24034796,24010337,24058293,24058128,24003103,9405963,23890959,24059898,24038425,23983296,23944779,23934970,23996830,23968386,24046872,24049569,1714244,23918597,24043961'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210701'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.

C:\Users\Lytexx\Downloads\chat-downloader>

xenova commented 3 years ago

Hmm, let me add some additional debug messages. I'm struggling to see what I am missing from the fix they implemented.

xenova commented 3 years ago

@Lytexx Okay, I added a commit which should hopefully print out some helpful info.

Lytexx commented 3 years ago

@Lytexx Okay, I added a commit which should hopefully print out some helpful info.

[DEBUG] Total number of messages: 75
[DEBUG] Continuation info: {'invalidationId': {'objectSource': 1056, 'objectId': 'Y2hhdH5LUk9xMkJsN0QzZ341NDE3NTY0', 'topic': 'chat~KROq2Bl7D3g~5417564', 'subscribeToGcmTopics': True, 'protoCreationTimestampMs': '1625269187481'}, 'timeoutMs': 10000, 'continuation': '0ofMyAOjARpYQ2lrcUp3b1lWVU5OZDBkSVVqQkNWRnAxVEhOdGFsbGZUbFExVUhkbkVndExVazl4TWtKc04wUXpaeG9UNnFqZHVRRU5DZ3RMVWs5eE1rSnNOMFF6WnlBQiis6sjDx8XxAjAAQAJKFggBGAEgAFDclNDEx8XxAlgDeACiAQBQg9Ltw8fF8QJY4pSl6KHF8QKCAQIIAYgBAKABrpvWxMfF8QI%3D', 'clickTrackingParams': 'CAEQl98BIhMI-ZHQxMfF8QIV1kR6BR1yWgCH'}
[DEBUG] Sleeping for 8000ms.
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] Continuation: {'click_tracking': {'clickTrackingParams': 'CAEQl98BIhMI-ZHQxMfF8QIV1kR6BR1yWgCH'}, 'continuation': '0ofMyAOjARpYQ2lrcUp3b1lWVU5OZDBkSVVqQkNWRnAxVEhOdGFsbGZUbFExVUhkbkVndExVazl4TWtKc04wUXpaeG9UNnFqZHVRRU5DZ3RMVWs5eE1rSnNOMFF6WnlBQiis6sjDx8XxAjAAQAJKFggBGAEgAFDclNDEx8XxAlgDeACiAQBQg9Ltw8fF8QJY4pSl6KHF8QKCAQIIAYgBAKABrpvWxMfF8QI%3D'}
[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210701.07.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0xb1f73414783d9a7f'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '23973490,24049567,24003105,24049820,23996830,24003586,24059898,24044124,24045411,24010337,24038425,23944779,23983296,23918597,24003103,23890959,9405963,24042870,24045214,24056265,23934970,24034796,24063016,24063146,23891346,24050957,23966208,24056839,24058128,24049577,24058293,24060795,24001373,23946420,23891344,23804281,24004644,24063702,24007246,24030040,24062573,24049575,23990875,24022729,24058240,23974595,23884386,23999405,24053866,23748146,24050630,24049573,23744176,24037794,24027649,24023960,24059521,23998056,23857949,23983813,24043961,23940238,24012512,24049569,24058812,24058380,24060199,23991737,23882685,23968386,24046872,24036948,24017848,24052245,1714243'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210701'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.

xenova commented 3 years ago

Alright, so the click tracking parameters are being passed to the next request (CAEQl98BIhMI-ZHQxMfF8QIV1kR6BR1yWgCH)... Could you maybe post the log from before the messages are displayed? (There should be some continuation information output there too, hopefully)

Lytexx commented 3 years ago

Alright, so the click tracking parameters are being passed to the next request (CAEQl98BIhMI-ZHQxMfF8QIV1kR6BR1yWgCH)... Could you maybe post the log from before the messages are displayed? (There should be some continuation information output there too, hopefully)

[DEBUG] Created YouTubeChatDownloader session.
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://www.youtube.com/watch?v=KROq2Bl7D3g', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'testing': False, 'verbose': True, 'quiet': False, 'message_groups': ['messages'], 'message_types': None, 'output': None, 'overwrite': True, 'sort_keys': True, 'indent': 4, 'format_file': None, 'chat_type': 'live', 'ignore': None, 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'format': 'youtube', 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=KROq2Bl7D3g HTTP/1.1" 200 None
[DEBUG] Match found: "<re.Match object; span=(0, 43), match='https://www.youtube.com/watch?v=KROq2Bl7D3g'>". Running "_get_chat_by_video_id" function in "YouTubeChatDownloader".
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x000002686ECE1F20>, 'callback': None, 'title': "【Member Only】 Char's Counterattack + Hathaway's Flash Watchalong", 'duration': 0, 'is_live': True, 'start_time': 1625266835490153.0, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x000002686ED39430>, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x000002686ED20820>}
[INFO] Retrieving chat for "【Member Only】 Char's Counterattack + Hathaway's Flash Watchalong".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTk5kMGRJVWpCQ1ZGcDFUSE50YWxsZlRsUTFVSGRuRWd0TFVrOXhNa0pzTjBRelp4b1Q2cWpkdVFFTkNndExVazl4TWtKc04wUXpaeUFCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
[DEBUG] Continuation: {'click_tracking': {'clickTrackingParams': 'IhMI3rLOxMfF8QIVR6lVCh1jlwHV'}, 'continuation': '0ofMyANhGlhDaWtxSndvWVZVTk5kMGRJVWpCQ1ZGcDFUSE50YWxsZlRsUTFVSGRuRWd0TFVrOXhNa0pzTjBRelp4b1Q2cWpkdVFFTkNndExVazl4TWtKc04wUXpaeUFCMAGCAQIIAQ%3D%3D'}

xenova commented 3 years ago

Okay thanks! The problem might be that I'm not doing the extra step that YT-DLP does for generating API headers. https://github.com/yt-dlp/yt-dlp/blob/169dbde946e290982fd0d15130217079324cf4f2/yt_dlp/downloader/youtube_live_chat.py#L182-L193

if frag_index > 1:
    request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
    if click_tracking_params:
        request_data['context']['clickTracking'] = {'clickTrackingParams': click_tracking_params}
    headers = ie._generate_api_headers(ytcfg, visitor_data=visitor_data)
    headers.update({'content-type': 'application/json'})
    fragment_request_data = json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n'
    success, continuation_id, offset, click_tracking_params = download_and_parse_fragment(
        url, frag_index, fragment_request_data, headers)
else:
    success, continuation_id, offset, click_tracking_params = download_and_parse_fragment(
        chat_page_url, frag_index)

As shown here, it appears as though these headers are required for all requests after the first. And this is inline with what seems to be happening (i.e., it breaks after the first request).

Let me see if adding this will fix it 🤞

xenova commented 3 years ago

Third time's a charm? 😬 https://github.com/xenova/chat-downloader/tree/member-fix

🤞

Lytexx commented 3 years ago

@xenova still the same i hope i'm doing this currently from my side is using git pull enough to update properly?

[DEBUG] Created YouTubeChatDownloader session.
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://www.youtube.com/watch?v=KROq2Bl7D3g', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'testing': False, 'verbose': True, 'quiet': False, 'message_groups': ['messages'], 'message_types': None, 'output': None, 'overwrite': True, 'sort_keys': True, 'indent': 4, 'format_file': None, 'chat_type': 'live', 'ignore': None, 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'format': 'youtube', 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=KROq2Bl7D3g HTTP/1.1" 200 None
[DEBUG] Match found: "<re.Match object; span=(0, 43), match='https://www.youtube.com/watch?v=KROq2Bl7D3g'>". Running "_get_chat_by_video_id" function in "YouTubeChatDownloader".
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x00000260C2D4CF90>, 'callback': None, 'title': "【Member Only】 Char's Counterattack + Hathaway's Flash Watchalong", 'duration': 0, 'is_live': True, 'start_time': 1625266835490153.0, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x00000260C2DA9550>, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x00000260C2D91850>}
[INFO] Retrieving chat for "【Member Only】 Char's Counterattack + Hathaway's Flash Watchalong".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTk5kMGRJVWpCQ1ZGcDFUSE50YWxsZlRsUTFVSGRuRWd0TFVrOXhNa0pzTjBRelp4b1Q2cWpkdVFFTkNndExVazl4TWtKc04wUXpaeUFCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
[DEBUG] Updated headers: ['User-Agent', 'Accept-Language', 'X-YouTube-Client-Name', 'X-YouTube-Client-Version', 'Origin', 'X-Goog-Visitor-Id', 'Authorization', 'X-Origin', 'content-type']
[DEBUG] Continuation: {'click_tracking': {'clickTrackingParams': 'IhMIhq3UydbF8QIVwpV7Ch1kXgJV'}, 'continuation': '0ofMyANhGlhDaWtxSndvWVZVTk5kMGRJVWpCQ1ZGcDFUSE50YWxsZlRsUTFVSGRuRWd0TFVrOXhNa0pzTjBRelp4b1Q2cWpkdVFFTkNndExVazl4TWtKc04wUXpaeUFCMAGCAQIIAQ%3D%3D'}
2021-07-03 02:46:39 | (Member (1 month)) Gemugami: :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:39 | (Member (6 months)) Anthony Salim: :_KDTD::_KDTD:
2021-07-03 02:46:40 | (Member (2 months)) Vicente: lol
2021-07-03 02:46:41 | (Member (1 month)) William Mei: :_oyaoya::_oyaoya::_oyaoya:
2021-07-03 02:46:41 | (Member (6 months)) Hiyu Vlei: :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:41 | (Member (2 months)) 온천판다: :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:41 | (Member (6 months)) Gravy: lmao
2021-07-03 02:46:41 | (Member (2 months)) RGB: LOL
2021-07-03 02:46:41 | (Member (6 months)) Sarty: Yeah Char an actual woman!
2021-07-03 02:46:42 | (Member (6 months)) OK Kei: playing hard to get lmaoo
2021-07-03 02:46:42 | (Member (6 months)) Def. not Kermit: :_KUSA::_KUSA::_KUSA:
2021-07-03 02:46:42 | (Member (6 months)) Coralld: :_BONK::_BONK::_BONK:
2021-07-03 02:46:43 | (Member (2 months)) Deranged Diction: :_oyaoya::_oyaoya::_oyaoya:
2021-07-03 02:46:43 | (Member (6 months)) TZKK: FLERT
2021-07-03 02:46:43 | (Member (6 months)) Graeyarea: Char is a piece of work
2021-07-03 02:46:43 | (Member (6 months)) Takonnoisseur: :_KUSA::_KUSA::_KUSA:
2021-07-03 02:46:43 | (Member (6 months)) GFlügel: Char a sensitive boi
2021-07-03 02:46:43 | (Member (6 months)) Twitching Bouse: so is that the truth or is he manipulating
2021-07-03 02:46:43 | (Member (6 months)) Jeffrey Ng: :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:44 | (Member (2 months)) Hoàng - KFP Madao Department: :_BONK::_BONK:
2021-07-03 02:46:44 | (Member (6 months)) Wydken: :_KUSA:
2021-07-03 02:46:44 | (Member (2 months)) Decretum: :_KDTD::_KDTD:
2021-07-03 02:46:44 | (Member (6 months)) Kelvin CCY: :_oii::_oii:
2021-07-03 02:46:45 | (Member (6 months)) slowchamp: Mommy issue
2021-07-03 02:46:45 | (Member (6 months)) Rum_Hard: the Alpha!
2021-07-03 02:46:45 | (Member (6 months)) Yuukiki: :_KDTD::_KDTD::_KDTD:handholding?! Heresy!
2021-07-03 02:46:45 | (New member) Alpha_AJ: What a scumbag
2021-07-03 02:46:46 | (Member (2 months)) Porway: :_oyaoya::_oyaoya::_oyaoya:
2021-07-03 02:46:46 | (Member (6 months)) Rayett Ray: char is a manipulator for sure
2021-07-03 02:46:46 | (Member (6 months)) KeimaKatsu: char and her feelings
2021-07-03 02:46:46 | (Member (6 months)) Peroni: :_KUSA::_KUSA:
2021-07-03 02:46:47 | (Member (6 months)) swoog: lmao
2021-07-03 02:46:47 | (Member (2 months)) TheWolfe570: booba aquired
2021-07-03 02:46:47 | (Member (6 months)) AŁ: thats pretty much Char in a nutshell
2021-07-03 02:46:47 | (New member) Mokey Jokey: hes face like :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:47 | (Member (6 months)) 美味しいたこ焼き: :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:48 | (Member (6 months)) jle: lmao
2021-07-03 02:46:49 | (Member (6 months)) Master Vandal: he's complicated
2021-07-03 02:46:49 | (Member (6 months)) Ittapon: BRUH
2021-07-03 02:46:49 | (Member (2 months)) Eli Smith: Hes literally me
2021-07-03 02:46:50 | (Member (1 month)) who even knows: :_KDTD::_KDTD::_KDTD:
2021-07-03 02:46:50 | (Member (6 months)) AlphaCentauri: Lmao
2021-07-03 02:46:50 | (Member (1 month)) Jerry: :_Jii:
2021-07-03 02:46:51 | (Member (2 months)) Takosuke Shrimpson: he knows women so well
2021-07-03 02:46:51 | (Member (1 month)) Yureio Ch.: Weren't they dating?
2021-07-03 02:46:51 | (Member (6 months)) KeimaKatsu: chan and his feelings
2021-07-03 02:46:52 | (Member (2 months)) PC_Principal _PC: not motorboating is missed opportunity char come on man
2021-07-03 02:46:52 | (Member (6 months)) RaMon: LOLICON
2021-07-03 02:46:53 | (New member) Kapodachi [Keeper of the Scriptures]: These relationships are all trainwrecks
2021-07-03 02:46:53 | (Member (2 months)) SBJTube: Char so pure pfft
2021-07-03 02:46:53 | (Member (2 months)) Not Aigis: LOL
2021-07-03 02:46:53 | (Member (2 months)) LegacyStar: Using anyone and everyone he can
2021-07-03 02:46:53 | (New member) zeiphie: babe..babe...BAAAABBBEEEE LUL
2021-07-03 02:46:54 | (Member (6 months)) warekosowa: he just wants mama
2021-07-03 02:46:55 | (Member (6 months)) AlphaCentauri: The Alpha!
2021-07-03 02:46:55 | (Member (6 months)) 荘garfield: :_KDTD::_KDTD:
2021-07-03 02:46:55 | (Member (6 months)) Lupine: dude wants it all
2021-07-03 02:46:55 | (Member (2 months)) Vicente: lol
2021-07-03 02:46:55 | (Member (2 months)) Langit Daun: :_OxO::_OxO::_OxO:
2021-07-03 02:46:56 | (Member (2 months)) RCSI: lol
2021-07-03 02:46:56 | (Member (2 months)) dreamyrobot: quess is so fucking spoiled its disgusting
2021-07-03 02:46:56 | (Member (2 months)) CHEENN: :_KUSA::_KUSA:
2021-07-03 02:46:56 | (Member (6 months)) swoog: meanies lmao
2021-07-03 02:46:56 | (Member (6 months)) Myaku: :_KUSA::_KUSA::_KUSA:
2021-07-03 02:46:57 | (Member (6 months)) Kioskhill: no u
2021-07-03 02:46:57 | (New member) Olan Yang: MA is always so cool
2021-07-03 02:46:57 | (Member (2 months)) ナナシさん: SHUT UP QUESS
2021-07-03 02:46:57 | (Member (1 month)) Slugien: 😆 😆 😆
2021-07-03 02:46:58 | (Member (6 months)) Hakase: Kid Removal Service
2021-07-03 02:46:58 | (Member (1 month)) mroziu 666: lol
2021-07-03 02:46:58 | (Member (6 months)) Gvrchomp: THIS GIRRLLLLLLLLL
2021-07-03 02:46:58 | (Member (6 months)) nox tech: yeet
2021-07-03 02:46:58 | (Member (2 months)) RGB: I wonder why QUESS
2021-07-03 02:46:59 | (Member (6 months)) Balthros: he wanted to get in on those red comets
2021-07-03 02:47:02 | (Member (6 months)) KeimaKatsu: lol
[DEBUG] Total number of messages: 75
[DEBUG] Continuation info: {'invalidationId': {'objectSource': 1056, 'objectId': 'Y2hhdH5LUk9xMkJsN0QzZ341NDE3NTc3', 'topic': 'chat~KROq2Bl7D3g~5417577', 'subscribeToGcmTopics': True, 'protoCreationTimestampMs': '1625273224623'}, 'timeoutMs': 10000, 'continuation': '0ofMyAOjARpYQ2lrcUp3b1lWVU5OZDBkSVVqQkNWRnAxVEhOdGFsbGZUbFExVUhkbkVndExVazl4TWtKc04wUXpaeG9UNnFqZHVRRU5DZ3RMVWs5eE1rSnNOMFF6WnlBQijDi_fI1sXxAjAAQAJKFggBGAEgAFD--dXJ1sXxAlgDeACiAQBQxZWdydbF8QJY4pSl6KHF8QKCAQIIAYgBAKABne3dydbF8QI%3D', 'clickTrackingParams': 'CAEQl98BIhMI2_bVydbF8QIVLtwRCB1x4AVE'}
[DEBUG] Sleeping for 8000ms.
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] Continuation: {'click_tracking': {'clickTrackingParams': 'CAEQl98BIhMI2_bVydbF8QIVLtwRCB1x4AVE'}, 'continuation': '0ofMyAOjARpYQ2lrcUp3b1lWVU5OZDBkSVVqQkNWRnAxVEhOdGFsbGZUbFExVUhkbkVndExVazl4TWtKc04wUXpaeG9UNnFqZHVRRU5DZ3RMVWs5eE1rSnNOMFF6WnlBQijDi_fI1sXxAjAAQAJKFggBGAEgAFD--dXJ1sXxAlgDeACiAQBQxZWdydbF8QJY4pSl6KHF8QKCAQIIAYgBAKABne3dydbF8QI%3D'}
[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210701.07.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0x51bb925f8149304c'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '23968386,23996830,24046872,24049820,24023960,23918597,24043961,24049567,23890959,9405963,24059898,24038425,23934970,23944779,23983296,23748146,23974595,24060795,24034796,23744176,24010337,24056265,24058128,24003103,24058293,1714254,24049577,24003105,23891344,24003586,24050957,24063146,24056839,24063016,23966208,23891346,24012512,24058240,24049575,23946420,24001373,24022729,24063702,24050630,23857949,24053866,23990875,24049573,24062573,23804281,24042870,24045214,23999405,23884386,24030040,24058380,23998056,23882685,23983813,24058812,24037794,24060199,23991737,24059521,24044124,24036948,24052245,24017848,24045411,24007246,24049569,23940238,23973490,24027649,24004644'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210701'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.

C:\Users\Lytexx\Downloads\chat-downloader>

xenova commented 3 years ago

i hope i'm doing this currently from my side is using git pull enough to update properly?

Yep that works - your latest log includes the debug messages from the latest commit I made.

xenova commented 3 years ago

Well this is quite perplexing... I can't seem to see what yt-dlp is doing that I am not :/

Although, what does seem off is that

[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210701.07.00'}, {'key': 'yt_li', 'value': '0'}, {'key': 'GetLiveChat_rid', 'value': '0x51bb925f8149304c'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '0'}, {'key': 'e', 'value': '23968386,23996830,24046872,24049820,24023960,23918597,24043961,24049567,23890959,9405963,24059898,24038425,23934970,23944779,23983296,23748146,23974595,24060795,24034796,23744176,24010337,24056265,24058128,24003103,24058293,1714254,24049577,24003105,23891344,24003586,24050957,24063146,24056839,24063016,23966208,23891346,24012512,24058240,24049575,23946420,24001373,24022729,24063702,24050630,23857949,24053866,23990875,24049573,24062573,23804281,24042870,24045214,23999405,23884386,24030040,24058380,23998056,23882685,23983813,24058812,24037794,24060199,23991737,24059521,24044124,24036948,24052245,24017848,24045411,24007246,24049569,23940238,23973490,24027649,24004644'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '0'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210701'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'loggedOut': True}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}

contains this: {'key': 'logged_in', 'value': '0'}, which suggests that the cookies aren't being applied properly.

Back to the drawing board! Thanks again for your help. 😄

Lytexx commented 3 years ago

Thanks again for your help. 😄

no problem thank you for doing this i'm happy to see that chat-downloader is getting updates again the stream is over now but there is another one in ~38 hours i can test there again if you find another possible solution

xenova commented 3 years ago

Awesome thanks! I kept trying things until ~6am last night but nothing yet. Although, I think I've found a way to test authentication on my computer, which should dramatically speed up the debugging process 🤣

I'll get something working by the next stream for testing 😃

xenova commented 3 years ago

Okay! I'm feeling a lot more confident about these latest changes (55c4fd1096bcdb3da89bb2b42d04bd4acbd48d15)

Hopefully this works 😄 https://github.com/xenova/chat-downloader/tree/member-fix

i'm happy to see that chat-downloader is getting updates again

Happy to be back 😃 I was super busy with university, and now I finally have some time to improve the tool.

Lytexx commented 3 years ago

@xenova it works! thank you very much 👍

xenova commented 3 years ago

Woohoo! 🥳 Thanks so much for testing!

xenova commented 3 years ago

This has been added to the latest version (v0.1.2). If you are using pip, you can install it using

pip install chat-downloader --upgrade

herowinb commented 3 years ago

Hello,

I tried newest version but still have problem in member-only stream (upcoming stream, I still not test live chat while stream live) Here my log. I need hide VideoID and title

/usr/src/app # chat_downloader --version
0.1.2
/usr/src/app # chat_downloader --cookies config/cookies_membership.txt https://www.youtube.com/watch?v=xxxx -v
[DEBUG] Created YouTubeChatDownloader session.
[INFO] Site: youtube.com
[DEBUG] Program parameters: {'url': 'https://www.youtube.com/watch?v=xxx', 'start_time': None, 'end_time': None, 'max_attempts': 15, 'retry_timeout': None, 'timeout': None, 'max_messages': None, 'logging': 'debug', 'pause_on_debug': False, 'exit_on_debug': False, 'testing': False, 'verbose': True, 'quiet': False, 'message_groups': ['messages'], 'message_types': None, 'output': None, 'overwrite': True, 'sort_keys': True, 'indent': 4, 'format_file': None, 'chat_type': 'live', 'ignore': None, 'message_receive_timeout': 0.1, 'buffer_size': 4096, 'format': 'youtube', 'inactivity_timeout': None}
[DEBUG] Starting new HTTPS connection (1): www.youtube.com:443
[DEBUG] https://www.youtube.com:443 "GET /watch?v=xxxxx HTTP/1.1" 200 None
[DEBUG] Match found: "<re.Match object; span=(0, 43), match='https://www.youtube.com/watch?v=xxxxx'>". Running "_get_chat_by_video_id" function in "YouTubeChatDownloader".
[DEBUG] Chat information: {'chat': <generator object YouTubeChatDownloader._get_chat_messages at 0x7f79a49c8c80>, 'callback': None, 'title': '【】', 'duration': 0, 'is_live': True, 'start_time': None, 'format': <function ChatDownloader.get_chat.<locals>.<lambda> at 0x7f79a49f3f70>, 'site': <chat_downloader.sites.youtube.YouTubeChatDownloader object at 0x7f79a4aabe20>}
[INFO] Retrieving chat for "【】".
[DEBUG] Getting Live chat (Live chat).
[DEBUG] https://www.youtube.com:443 "GET /live_chat?continuation=0ofMyANhGlhDaWtxSndvWVZVTlFNRUp6Y0U5ZlFVMUZaVE5oVVhGeGNHODRPVVJuRWdzM2RFWm5ZMll3V1dORlFSb1Q2cWpkdVFFTkNnczNkRVpuWTJZd1dXTkZRU0FCMAGCAQIIAQ%3D%3D HTTP/1.1" 200 None
[DEBUG] Continuation: {'click_tracking': {'clickTrackingParams': 'IhMI6rfJ0arQ8QIVEofmCh3BmQiI'}, 'continuation': '0ofMyANhGlhDaWtxSndvWVZVTlFNRUp6Y0U5ZlFVMUZaVE5oVVhGeGNHODRPVVJuRWdzM2RFWm5ZMll3V1dORlFSb1Q2cWpkdVFFTkNnczNkRVpuWTJZd1dXTkZRU0FCMAGCAQIIAQ%3D%3D'}
[DEBUG] Logged-in info: {'key': 'logged_in', 'value': '1'}
2021-07-07 03:18:55 | (Member (6 months)) 
2021-07-07 03:19:46 | (Member (6 months)) 
2021-07-07 03:20:03 | (Member (6 months)) 
2021-07-07 03:20:30 | (Member (2 months)) 
2021-07-07 03:20:56 | (Member (2 months)) 
2021-07-07 03:29:01 | (Member (6 months)) 
2021-07-07 03:32:19 | (Member (2 months)) 
2021-07-07 03:50:08 | (Member (2 months)) 
2021-07-07 03:50:33 | (Member (6 months)) 
2021-07-07 03:55:11 | (Member (2 months)) 
2021-07-07 04:03:16 | (Member (2 months)) 
2021-07-07 04:03:35 | (Member (2 months)) 
2021-07-07 04:44:11 | (Member (6 months)) 
2021-07-07 05:06:28 | (Member (6 months)) 
[DEBUG] Total number of messages: 14
[DEBUG] Continuation info: {'invalidationId': {'objectSource': 1056, 'objectId': 'Y2hhdH43dEZnY2YwWWNFQX41NDE4Nzk4', 'topic': 'chat~xxxx~5418798', 'subscribeToGcmTopics': True, 'protoCreationTimestampMs': '1625639387938'}, 'timeoutMs': 10000, 'continuation': '0ofMyAOjARpYQ2lrcUp3b1lWVU5RTUVKemNFOWZRVTFGWlROaFVYRnhjRzg0T1VSbkVnczNkRVpuWTJZd1dXTkZRUm9UNnFqZHVRRU5DZ3MzZEVablkyWXdXV05GUVNBQii0joCCmNDxAjAAQAJKFggAGAEgAFCll_zRqtDxAlgDeACiAQBQisCggpjQ8QJYm_LEiKTA8QKCAQIIAYgBAKAB2f2A0qrQ8QI%3D', 'clickTrackingParams': 'CAEQl98BIhMI-pT80arQ8QIVmzu3AB1jggv6'}
[DEBUG] Sleeping for 8000ms.
[DEBUG] https://www.youtube.com:443 "POST /youtubei/v1/live_chat/get_live_chat?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1" 200 None
[DEBUG] Continuation: {'click_tracking': {'clickTrackingParams': 'CAEQl98BIhMI-pT80arQ8QIVmzu3AB1jggv6'}, 'continuation': '0ofMyAOjARpYQ2lrcUp3b1lWVU5RTUVKemNFOWZRVTFGWlROaFVYRnhjRzg0T1VSbkVnczNkRVpuWTJZd1dXTkZRUm9UNnFqZHVRRU5DZ3MzZEVablkyWXdXV05GUVNBQii0joCCmNDxAjAAQAJKFggAGAEgAFCll_zRqtDxAlgDeACiAQBQisCggpjQ8QJYm_LEiKTA8QKCAQIIAYgBAKAB2f2A0qrQ8QI%3D'}
[DEBUG] Logged-in info: {'key': 'logged_in', 'value': '1'}
[DEBUG] No continuation information found: {'responseContext': {'serviceTrackingParams': [{'service': 'CSI', 'params': [{'key': 'c', 'value': 'WEB'}, {'key': 'cver', 'value': '2.20210701.07.00'}, {'key': 'yt_li', 'value': '1'}, {'key': 'GetLiveChat_rid', 'value': '0xa0ec8bf7af27655f'}]}, {'service': 'GFEEDBACK', 'params': [{'key': 'logged_in', 'value': '1'}, {'key': 'country-type', 'value': 'B'}, {'key': 'e', 'value': '23744176,23974595,24003103,24049575,24058240,23944779,24003105,23986029,24063702,24053866,24049573,23891344,24058812,24043961,24058380,23996830,23968386,24038425,24049569,24060241,23934970,23857950,23890959,24052245,23946420,24037794,23918597,23882502,23998056,24049567,24049820,23804281,23973490,24004644,23975058,24030040,23983296,24036948,24045411,24007246,24056265,24001373,24058128,24058293,23966208,1714259,23891346,24042870,24059522,24012512,24027649,23999405,24023960,23884386,24049577'}]}, {'service': 'GUIDED_HELP', 'params': [{'key': 'logged_in', 'value': '1'}]}, {'service': 'ECATCHER', 'params': [{'key': 'client.version', 'value': '2.20210701'}, {'key': 'client.name', 'value': 'WEB'}]}], 'mainAppWebResponseContext': {'datasyncId': '109470951785329220521||', 'loggedOut': False}, 'webResponseContextExtensionData': {'hasDecorated': True}}, 'contents': {'messageRenderer': {'text': {'runs': [{'text': 'Sorry, live chat is currently unavailable.'}]}}}}
[INFO] Finished retrieving chat.
[DEBUG] Session closed.
/usr/src/app #

xenova commented 3 years ago

Hmm, this is quite strange.

It says you are still logged in, so it shouldn't be an authentication issue.

[DEBUG] Logged-in info: {'key': 'logged_in', 'value': '1'}

I tried newest version but still have problem in member-only stream (upcoming stream, I still not test live chat while stream live)

Okay - could you let me know if it works with the actual live stream (i.e., when they are streaming)? Also, does this always happen? Can you rerun a few times?

In the meantime, I will try figure out what the problem might be. Thanks!

herowinb commented 3 years ago

Archive member-only stream: OK
Upcoming member-only stream ( it's freechat for members): No. This is log I send in here. I reruned many times but still not working
When streaming: Unfortunately I joined only 1 membership so I need wait till this weekend or next week to try it again.

Thank you for your hard work.

xenova commented 3 years ago

@herowinb Okay, I have been playing around with a few things, but it is quite difficult to debug as I am not a member to anyone on YouTube.

I have added a few headers to the requests in this branch, so you could try that. I am not super confident that those changes will fix the problem completely, but there is a chance.

If it doesn't work (which is most probable), could you see what it says on YouTube in dev tools when you are in the chat room? It may be the case that I am missing a header or two.

herowinb commented 3 years ago

I tried with your new branch and it worked with upcoming stream.

xenova commented 3 years ago

Okay great! 😊

These changes are now in the latest version 💪 (v0.1.3)