Many warnings for /r/wallstreetbets

[sanel]

When https://reddit.com/r/wallstreetbets is select as group, nntpit will report a lot of warnings, like:

[15] <- [GROUP wallstreetbets]

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json
...

but when you go to /r/programming, it will print a lot less warnings:

[15] <- [GROUP programming]

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json
[15] <- [HEAD 0]
[15] <- [HEAD 1]

I presume this has to do with number for posts in each thread? /r/wallstreetbets is overcrowded these days...

nntpit is started with: ./nntpit -D -t 3 -p 8119.

[maxcrees]

I adjusted the error messages slightly to debug this further since subreddit / comment parsing were using identical messages. Then I added the error message returned by json_tokener_error_desc, which turned all of these errors into the following:

** (process:13196): WARNING **: 21:52:15.978: failed to parse comment json: nesting too deep

This is because json-c by default only allows JSON objects to be nested with a depth of around 32 (JSON_TOKENER_DEFAULT_DEPTH).

Either json_object_from_fd_ex or json_tokener_new_ex needs to be used to override this depth limit to some higher (but still finite) value. Such a change will also have to be reflected when reading back from the spool file as well.

I'll try to write a patch this weekend.

[taviso]

Thanks for tracking it down! Let's try doubling it and see what happens.