[Bug]: comment recursion error

[thecoolreaver]

Version

falocalrepo 4.4.5

What happened?

When running falocalrepo download users --retry 3 --no-comments USER -f gallery -f scraps -f favorites -f journals -f userpage and it running though the users favorites it runs into a submission that has alot of comments and runs into a error

How to reproduce the bug?

falocalrepo download submissions 8601571

Relevant log output

No response

[thecoolreaver]

fa.log

[MatteoCampinoti94]

Wow, never thought there would be a submission that broke the recursion limit o.o

Could you please run these two lines in a Python prompt?

import sys
sys.getrecursionlimit()

I can solve the issue directly in the FAAPI library, but I cannot remove the recursion limit. There is a sys command to do it, but there is no way to predict whether it will cause issues :(

[thecoolreaver]

~# python3
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getrecursionlimit()
1000
>>>

[MatteoCampinoti94]

The fox will take longer than I thought. Because of how Python calculates its stack, the recursion fails after less than 500 comments in a chain, that specific function (_remove_recursion) cannot be avoided or the dict method will keep going in circles because comments contain an id link to the parent submission/journal.

Never encountered a comments chain that long before so it never popped up.

For the moment the solution is to edit the library manually to remove the problematic line.

Change line 220 in comment.py to:

        yield "comments", []  # [dict(_remove_recursion(c)) for c in self.comments]

It's only a temporary fix, but it will stop the recursion error for now.

[thecoolreaver]

I applied that patch but I still get an error.

# falocalrepo download submissions 8601571
1/1 0008601571 [#######][#]
Trace written to /mnt/gd-data/FALocalRepo/falocalrepo/FA.log

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8 <module> /usr/local/bin/falocalrepo
> sys.exit(main())

59 main /usr/local/lib/python3.10/dist-packages/falocalrepo/__main__.py
> exit(app.main(standalone_mode=False) or 0)

1055 main /usr/local/lib/python3.10/dist-packages/click/core.py
> rv = self.invoke(ctx)

1657 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return _process_result(sub_ctx.command.invoke(sub_ctx))

1657 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return _process_result(sub_ctx.command.invoke(sub_ctx))

1404 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return ctx.invoke(self.callback, **ctx.params)

760 invoke /usr/local/lib/python3.10/dist-packages/click/core.py
> return __callback(*args, **kwargs)

26 new_func /usr/local/lib/python3.10/dist-packages/click/decorators.py
> return f(get_current_context(), *args, **kwargs)

316 download_submissions /usr/local/lib/python3.10/dist-packages/falocalrepo/console/download.py
> downloader.download_submissions(list(submission_id))

783 download_submissions /usr/local/lib/python3.10/dist-packages/falocalrepo/downloader.py
> self.download_submission(submission_id,

406 download_submission /usr/local/lib/python3.10/dist-packages/falocalrepo/downloader.py
> save_comments(self.db, submissions_table, submission.id, submission.comments,

126 save_comments /usr/local/lib/python3.10/dist-packages/falocalrepo/downloader.py
> for comment in filter(lambda c: not c.hidden, flatten_comments(comments)):

161 flatten_comments /usr/local/lib/python3.10/dist-packages/faapi/comment.py
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})

161 <listcomp> /usr/local/lib/python3.10/dist-packages/faapi/comment.py
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})

.... repeats for awhile ....

161 flatten_comments /usr/local/lib/python3.10/dist-packages/faapi/comment.py
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})

161 <listcomp> /usr/local/lib/python3.10/dist-packages/faapi/comment.py
  Flattens a list of comments. Replies are not modified.

  :param comments: A list of Comment objects (flat or tree-structured)
  :return: A flat date-sorted (ascending) list of comments
  """
> return sorted({c for c in [r for c in comments for r in [c, *flatten_comments(c.replies)]]})

c: 62102234 THE-DALEK-SUPREME
OP

RecursionError:
maximum recursion depth exceeded

[MatteoCampinoti94]

You also have to use the --no-comments option. The FAAPI patch makes it so the comments tree is not traversed when casting the Submission object to a dict. The --no-comments options makes FALocalRepo ignore comments. this way neither _remove_recursion nor flatten_comments are called and the recursion limit isn't reached.

[MatteoCampinoti94]

I fixed the issue https://github.com/FurryCoders/FAAPI/commit/97944f188ec1bed80a041502925566fd8de69163!

Had to completely rewrite the tree-parser in a procedural style, which I really do not like, but alas that's the only solution :(

I will push a new FAAPI release soon and then a patch for FALocalRepo.

[MatteoCampinoti94]

Issue is completely fixed and I have also solved another problem caused by the new OP (Original Poster) tags that FA now adds to comments from the author of the submission/journal.

You can download release 4.4.6 to get the fixes :)