Comment logs limited to 400 entries

alexroston commented 4 years ago

[x] I am using the latest version of PyInstaLive: 3.2.0.
[x] I have installed either Python 2.7.x or 3.5+: 3.8
[x] I have read the README.
[x] I have read the FAQ.
[x] I have checked all existing issues, none of which could solve my issue.

I am using:

[x] Windows 10
[ ] Windows 8.1
[ ] Windows 8
[ ] Windows 7
[ ] Linux (distribution: )
[ ] macOS (version: )
[ ] Other (device & OS name: )

Required files

pyinstalive.default.log pyinstalive_raychell0113.log

PyInstaLive information

$ pyinstalive --info
---------------------------------------------------------------------------
[I] PYINSTALIVE (SCRIPT V3.2.0 - PYTHON V3.8.0) - 04-19-2020 09:21:08 PM
---------------------------------------------------------------------------
[I] To see all the available arguments, use the -h argument.

[I] PyInstaLive version:        3.2.0
[I] Python version:             3.8.0
[I] FFmpeg framework:           Available
[I] Cookie files:               1 (*****.json matches config user)
[I] CLI supports color:         No
[I] Command to run at start:    None
[I] Command to run at finish:   None
[I] Config file contents:

    [pyinstalive]
    username = *****
    password = *****
    download_path = *****
    download_lives = True
    download_replays = True
    download_comments = true
    show_cookie_expiry = True
    log_to_file = True
    ffmpeg_path =
    run_at_start =
    run_at_finish =
    use_locks = True
    clear_temp_files = False
    do_heartbeat = True
    proxy =
    verbose = False
    skip_merge = False

[I] End of PyInstaLive information screen.
---------------------------------------------------------------------------

Command used

pyinstalive -d "raychell0113"

Behavior

There could only be downloaded 400 comments during the Insta live. After the comment number exceeded 400 there was a warning [W] Comment collection ClientError: 400 {"message": "Sorry, unknown error", "status": "fail"} and no new comment log file was created with the remaining comments.

Steps to reproduce issue

I simply ran the script with the above mentioned command during the Insta live. No further steps taken.

Question

Can the comment limit please be lifted or a config option be created to decide the maximum number of comments to download during a live? Splitting the comment log files after 400-500 comments would be also ok.

dvingerh commented 4 years ago

There are no set limits for comment downloading. Past tests using replays have had over 1000 comments saved in total for me. Are you frequently experiencing this problem?

alexroston commented 4 years ago

Actually it seems not to occur with replays but with lives. And yes, after this one I got the same warning again earlier this day with another livestream (not replay). Didn't have this with replays either.

dvingerh commented 4 years ago

A recent download that I managed to run (not a replay) worked just fine with over 1700 saved comments. Older downloads from a few months back with a lot of comments have saved without issue as well for me.

Given the circumstances in which this issue occurs though (long livestreams with a lot of comments), the fact that it doesn't seem to happen all the time and the error message returned by Instagram is pretty unhelpful: Sorry, unknown error makes this hard to troubleshoot let alone try to find a solution.

I'll set up a few test instances and monitor the log files for this error if you could provide me with some users that frequently do livestreams with large amounts of comments.

alexroston commented 4 years ago

Got a livestream now where 607 comments were saved just fine, but the frequency of the comments seemed to be a bit slower.

---------------------------------------------------------------------------
[I] PYINSTALIVE (SCRIPT V3.2.0 - PYTHON V3.8.0) - 04-19-2020 10:42:59 PM
---------------------------------------------------------------------------
[I] Successfully logged into account: *****
[I] Cookie file expiry date: 2020-07-03 at 03:00:26 PM
---------------------------------------------------------------------------
[I] Getting info for 'nananananachume' successful.
---------------------------------------------------------------------------
[I] Livestream available, starting download.
---------------------------------------------------------------------------
[I] Username    : nananananachume
[I] Viewers     : 238 watching
[I] Airing time : 26 minutes and 7 seconds
[I] Status      : Active
---------------------------------------------------------------------------
[I] Downloading livestream, press [CTRL+C] to abort.
[W] dictionary changed size during iteration
---------------------------------------------------------------------------
[I] The livestream has been ended by the user.
---------------------------------------------------------------------------
[I] Airtime duration  : 60 minutes and 5 seconds
[I] Download duration : 33 minutes and 59 seconds
[I] Missing (approx.) : 26 minutes and 6 seconds
---------------------------------------------------------------------------
[I] Successfully saved 607 comments.
---------------------------------------------------------------------------
[I] Merging downloaded files into video.
[I] Successfully merged downloaded files into video.
---------------------------------------------------------------------------
[I] There are no available replays.
---------------------------------------------------------------------------

I don't load from widely know popstars who would maybe reach this comments frequency usually (maybe trying one of these might help). But you could check out these users who recently made longer livestreams: aiba_aiai, nananananachume, uesaka_sumire.

dvingerh commented 4 years ago

I've added the users, will keep it running for a week or two and see if there are any results

Just for the record I'd like to point out that the comment count is likely incorrect, there seem to be nearly twice as many comments in the comment log files that I get compared to what's reported by PyInstaLive. I already pushed out a commit with a possible fix https://github.com/dvingerh/PyInstaLive/commit/bb9e3243398ba174f1f25ece1f420754b069e8a4 but some more testing has to be done to make sure all is well

dvingerh commented 4 years ago

36 streams have been downloaded but none of them showed any issues regarding comment collection. I'll mark this issue as not reproducible but will keep an eye on the logs for the time being just in case.

alexroston commented 4 years ago

Actually after I switched to --download-following this didn't occur again. Seems it only happed when at first I was downloading directly by typing in the user name. Maybe because --download-following is threaded?

dvingerh commented 4 years ago

That shouldn't make a difference as far as I'm aware, all -df does is it just runs a separate instance of pyinstalive with -d (and reusing other arguments you might have passed initially) in the background for every user, behaving just the same as when typing in the username yourself.

If it somehow does make a difference I wouldn't know why.

https://github.com/dvingerh/PyInstaLive/blob/master/pyinstalive/dlfuncs.py#L410-L418

dvingerh / PyInstaLive