xenova / chat-downloader

A simple tool used to retrieve chat messages from livestreams, videos, clips and past broadcasts. No authentication needed!
https://chat-downloader.readthedocs.io/
MIT License
948 stars 132 forks source link

[QUESTION] Is the amount of membership_item correct ? #74

Closed Lyfhael closed 3 years ago

Lyfhael commented 3 years ago

Hi,

I'm trying to count the amount of new memberships subscribed during a Youtube livestream, and I'm a little surprised at the results I'm getting.

I know that some messages in the chat can only be seen during the live stream, and cannot be retrieved later with your library, and I'm wondering if it's the same for Memberships ?


Example, for bJjNnsWK_4U we can see there are 30 new memberships and 91 superchats On the same channel, for 1AIMAiWweOA there are 1172 new memberships and 48 superchats. Those two videos are only 1 month apart, same time length, approx. same amount of views

That's just one example, there are a lot of discrepancies between videos from a same channel. So I'm wondering if I can trust those numbers or not

xenova commented 3 years ago

Hi there - yes, the number of items you get should be correct (should be exactly what you see from the video's page). The streamer must have enabled memberships around that time.

If you go to 1AIMAiWweOA, you'll see: image

But if you go to bJjNnsWK_4U, you'll see: image


If you take more samples I am sure you will see clearer trends.

Lyfhael commented 3 years ago

If you take more samples I am sure you will see clearer trends.

Script is doing stats for all the videos on the channel, it may take a couple of hours to complete, once it's done I'll come back put the result for each day/month here if it seems odd.

edit: 4 Hours in, still running, at 61 / 110 videos.

xenova commented 3 years ago

If you take more samples I am sure you will see clearer trends.

Script is doing stats for all the videos on the channel, it may take a couple of hours to complete, once it's done I'll come back put the the result for each day/month here if it seems odd

Okay great 👍

Lyfhael commented 3 years ago

Alright so the issue is not your script, it is just Youtube not displaying all new Memberships.

I calculated the amount of memberships subscribed by checking the chat / superchat / membership_items and it's vastly under-estimated

DlfmLK8

If we just take in consideration membership_item it returns 8,416 for 6 months of streaming at 20,000/30,000 constant avg viewers during streams~, it's unbelievably low, especially considering how insane the superchats were.

But by checking the chat/superchat we can see there were in fact approx. 123,915 memberships subscribed during the whole period. EDIT: A more accurate version of the script gives me 112,000 EDIT2: An even more accurate one gives me 108,017, I've no further ideas to increase accuracy tho, it seems good enough

xenova commented 3 years ago

It is quite unfortunate that YouTube hides a lot of data when showing the chat replay. The only logical reason I can think of is to reduce client side lag (and perhaps alleviate server stress?).

The only accurate way - it seems - is to run the tool while the user is live. This should get every message that was sent (including memberships + superchats)

I am currently adding some functionality to the tool which allows users to get live chat as soon as a channel goes live (#62) and will be released as soon as possible. This will definitely help with these types of calculations.

Lyfhael commented 3 years ago

It is quite unfortunate that YouTube hides a lot of data when showing the chat replay. The only logical reason I can think of is to reduce client side lag (and perhaps alleviate server stress?).

The only accurate way - it seems - is to run the tool while the user is live. This should get every message that was sent (including memberships + superchats)

I am currently adding some functionality to the tool which allows users to get live chat as soon as a channel goes live (#62) and will be released as soon as possible. This will definitely help with these types of calculations.

oh, that could be indeed a nice feature

ghost commented 3 years ago

Linux-only, but I use this shell script to watch for livestreams and run chat-downloader every time a livestream is going on.

#!/bin/sh
channel="$1"

get_live_url() {
    _channel="$1"

    _video_code=$(youtube-dl --list-formats --playlist-end 1 "$_channel/live" | grep MPD)
    if [ -n "$_video_code" ]; then
        _url=${_video_code#*be] }
        _url="https://www.youtube.com/watch?v=${_url%:*}"
        echo "$_url"
    fi
}

while live_url=$(get_live_url "$channel"); do
    if [ -n "$live_url" ]; then
        chat_downloader "$live_url"
    else
        sleep 60
    fi
done