AlexAplin / nndownload

Download and process links from Niconico (nicovideo.jp)
MIT License
213 stars 28 forks source link

Download comment history #134

Open AlexAplin opened 1 year ago

AlexAplin commented 1 year ago

Nico uniquely has a rolling comment list with the most recent comments replacing past comments. However, users can filter the comment list by date to view comments from that point in time. That's reflected in requests made to https://nvcomment.nicovideo.jp/v1/threads containing:

"additionals":{"when":1210714200}

We should offer a flag to specify a timestamp (either epoch or %Y-%m-%d %H:%s) and/or to fetch all comments between two timestamps and aggregate them to one file.

AlexAplin commented 1 year ago

Requesting by epoch means there is tremendous granularity here, and you'll only receive 250 of the comments up to that timestamp, ordered ascending. Unfortunately the actual comments don't come with their own epoch time, but we do get a total comment count per thread, and each comment has a timestamp and is numbered overall:

commentCount: 30079

postedAt: '2007-09-22T08:42:17+09:00' no: 29826

So broadly I think what needs to happen is we request the newest comments as normal with no when set. Then, take the first comment postedAt, convert to epoch, and request at that point (may need to add some buffer here to ensure we don't miss anything). Collect based on no to prevent duplicates and to identify when we've reasonably gotten all comments.

We probably will need to refresh our thread key periodically, which is done with OPTIONS and GET to https://nvapi.nicovideo.jp/v1/comment/keys/thread?videoId=. So far we haven't done this.