voussoir / timesearch

The subreddit archiver
BSD 3-Clause "New" or "Revised" License
172 stars 7 forks source link

AttributeError: 'DummySubmission' object has no attribute 'url' on some subreddits #13

Closed dunemafia closed 1 year ago

dunemafia commented 2 years ago

Hello Ethan,

Thank you so much for this great tool. I'd like to report an error I've been receiving while scraping some subreddits.

Traceback (most recent call last):
  File "/opt/reddit/timesearch.py", line 553, in <module>
    raise SystemExit(main(sys.argv[1:]))
  File "/home/arrakis/.local/lib/python3.10/site-packages/voussoirkit/vlogging.py", line 221, in wrapped
    return main(argv, *args, **kwargs)
  File "/opt/reddit/timesearch.py", line 545, in main
    return betterhelp.go(parser, argv)
  File "/home/arrakis/.local/lib/python3.10/site-packages/voussoirkit/betterhelp.py", line 621, in go
    return _go_multi(parser, argv, args_postprocessor=args_postprocessor)
  File "/home/arrakis/.local/lib/python3.10/site-packages/voussoirkit/betterhelp.py", line 617, in _go_multi
    return main(argv)
  File "/home/arrakis/.local/lib/python3.10/site-packages/voussoirkit/betterhelp.py", line 579, in main
    return args.func(args)
  File "/opt/reddit/timesearch.py", line 56, in get_submissions_gateway
    get_submissions.get_submissions_argparse(args)
  File "/opt/reddit/timesearch_modules/get_submissions.py", line 96, in get_submissions_argparse
    return get_submissions(
  File "/opt/reddit/timesearch_modules/get_submissions.py", line 75, in get_submissions
    step = database.insert(chunk)
  File "/opt/reddit/timesearch_modules/tsdb.py", line 347, in insert
    status = method(obj)
  File "/opt/reddit/timesearch_modules/tsdb.py", line 400, in insert_submission
    url = submission.url
AttributeError: 'DummySubmission' object has no attribute 'url'

This only happens on some subreddits (e.g. /r/unixporn), and only on submissions after September 06, 2022. I have upgraded voussoirkit to 0.0.74, and have the latest git updates. Thank you once again for fantastic tool.

voussoir commented 2 years ago

Hi, thanks for the well written bug report.

I wrote c8c160e00e600b5ee84c26044cbe6855eb96cacf to take the crosspost parent when available, and use a safe getattr fallback otherwise.

Could you give that a try and see if it fixes the issue for you?

dunemafia commented 2 years ago

Yes, it does fix the issue. Thank you for such a speedy resolution. Greatly appreciate your effort. Thanks again.