Add Subreddit Comment Retrieval and Stream Concurrency

vartanbeno / go-reddit

Go library for accessing the Reddit API.

Other

306 stars 84 forks source link

Add Subreddit Comment Retrieval and Stream Concurrency #12

Open RGood opened 3 years ago

RGood commented 3 years ago

This PR currently sits on top of comment-support as it builds on top of some of it.

Changelog:

Move the set{} struct to its own helper file and refactor it to OrderedMaxSet
- The goal was to give it a maximum size while still checking for duplicates in the latest ids
- Helper since it's more abstract functionality and less go-reddit specific
Refactor streams to a request-callback approach, so requests don't block each other
- The purpose of this is to make requests on pace with the Ticker set for the stream and not have requests block each other
- The OrderedMaxSet should handle de-duping request responses

Considerations:

Perhaps make the streams configurable to either make requests sync or async

RGood commented 3 years ago

This PR should be ready for review / merge

darren commented 3 years ago

panic happens sometimes with this pr (may exist in original code)

panic: send on closed channel

goroutine 93559 [running]:
github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Posts.func2.1(0xc000081800, 0x64, 0x80, 0x0, 0x0)
       go-reddit/reddit/stream.go:91 +0xc5
github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).getPosts(0xc000286288, 0x9bc961, 0x3, 0xc00089c000)
       go-reddit/reddit/stream.go:193 +0x106
created by github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Posts.func2
       go-reddit/reddit/stream.go:60 +0x198

possible fix based on this PR: https://github.com/darren/go-reddit/commit/39267077688fc9cd37974d6420563191894a17b6

RGood commented 3 years ago

@darren Would you mind making a PR against this branch in my fork so I can easily add your code / give you credit for it?

darren commented 3 years ago

@RGood my fix above is still buggy, panic occasionally:

panic: send on closed channel

goroutine 347827 [running]: github.com/vartanbeno/go-reddit/v2/reddit.(StreamService).Comments.func2.1(0xc001434800, 0x64, 0x80, 0x0, 0x0) go-reddit/reddit/stream.go:196 +0x1a5 github.com/vartanbeno/go-reddit/v2/reddit.(StreamService).getComments(0xc0000a0700, 0xe26c10, 0x3, 0xc000138180) go-reddit/reddit/stream.go:218 +0xe3 created by github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Comments.func2 go-reddit/reddit/stream.go:161 +0x1a5

So using a quitChan is still tricky and not fixing the problem, I opened another PR https://github.com/RGood/go-reddit/pull/1 by using context