Clarify behavior of “after” parameter in reddit.Scanner.Listing

[J-Thompson12]

in reddit scanner.go you have the after var getting set to before. This is breaking pagination

func (s *scanner) Listing(path, after string) (Harvest, error) {
    return s.r.reap(
        path, map[string]string{
            "raw_json": "1",
            "limit":    "100",
            "before":    after,
        },
    )
}

[turnage]

This is intentional. Not necessarily defending the decision, but I believed at the time that Reddit was incorrect. “after” in the graw API means “chronologically after”.

At this point it would be a bad idea to change since users and the graw package depend on these semantics.

You can use ListingWithParams to set whatever parameters you need.

On Fri, Apr 3, 2020 at 8:10 AM Justin Thompson notifications@github.com wrote:

in reddit scanner.go you have the after var getting set to before. This is breaking pagination func (s *scanner) Listing(path, after string) (Harvest, error) { return s.r.reap( path, map[string]string{ "raw_json": "1", "limit": "100", "before": after, }, ) }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/turnage/graw/issues/71, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWR64GBEPX4REYADF6IOC3RKX37RANCNFSM4L4ESEXQ .

[turnage]

I’ll re-open this until the a comment about this is added to clear up confusion in the future. Unfortunately I think that’s the best that can be done without breaking users.

[J-Thompson12]

I just pulled in 200 mentions using pagination with after and it pulled in the second page with all comments createdUTC earlier than the first page

[turnage]

With which meaning of the parameter “after”? If you pass “after” to Reddit, you will get what is chronologically before. If you pass “after” to reddit.Scanner.Listing, I believe you will receive what comes chronologically after.

This is the way all of the streams in graw work.

[J-Thompson12]

i used the listingWithParams set to after and got the createdUTC times

end of page 1.             beginning of page 2
1585763973                1585762846
1585763157                 1585761938
1585763134                  1585761819
1585763072.                1585761806
1585763016.                 1585761502
1585763009.                 1585761462

[turnage]

If you use ListingWithParams, you are talking to directly to Reddit with your parameters, so that is expected.

On Fri, Apr 3, 2020 at 9:05 AM Justin Thompson notifications@github.com wrote:

i used the listingWithParams set to after and got

end of page 1. beginning of page 2 1585763973 1585762846 1585763157 1585761938 1585763134 1585761819

1585763072. 1585761806
1585763073. 1585761502
1585763074. 1585761462

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/turnage/graw/issues/71#issuecomment-608525323, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWR64B3XEFL77N6PT7ZBF3RKYCK3ANCNFSM4L4ESEXQ .