Closed azrafe7 closed 6 months ago
Hello, I've made another round of (hopefully) fixes. This time I based it on your main branch :P
One of the things it changes is switching to use permalinks if the comment is not found on the main-thread page (more on that later), and handle the case when the comment itself is collapsed, which led to screenshot like this one (with no comment text visible):
About the use_permalinks feature, here's how it works (as of now):
As loading each permalink is more robust, but also slower, I think this could be optimized by building a separate queue.
So, while processing accepted_comments
:
not_found_comments
, by using their permalinkAnyway... thanks for merging my previous PR, hopefully this one could be useful too.
Cheers, azrafe7
Another issue I've encountered (but can reproduce right now) is screenshots that seems to have an overlay, or are 'dimmed' (in other words, with a dark theme, the text is grey instead of the usual white).
If you can reproduce that please let me know π
Hey @azrafe7 !
Not sure about this change, it throws way more errors than the current version of the main repo. ie. if I try to generate a video from this subreddit ; https://www.reddit.com/r/AskReddit/comments/1avl30j/whats_your_favorite_this_much_of_this_will_kill/
With the actual version of screenshot.py, it works fine. With your updated version, it fails with this error :
Downloading screenshots... βββββββββββββββββββββββΈβββββββββββββββββ 57% -:--:--
Traceback (most recent call last):
File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 440, in <module>
process_submissions(submissions)
File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 64, in process_submissions
process_submission(submission)
File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 118, in process_submission
vid.create(
File "/home/sb/videos-creation/python-reddit-youtube-bot/video_generation/video.py", line 500, in create
download_screenshots_of_reddit_posts(
File "/home/sb/videos-creation/python-reddit-youtube-bot/comments/screenshot.py", line 253, in download_screenshots_of_reddit_posts
self_expand_button_loc.dispatch_event('click')
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 15591, in dispatch_event
self._sync(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
return task.result()
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 171, in dispatch_event
return await self._frame.dispatch_event(self._selector, strict=True, **params)
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 363, in dispatch_event
await self._channel.send(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 61, in send
return await self._connection.wrap_api_call(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 461, in wrap_api_call
return await cb()
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 96, in inner_send
result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 60000ms exceeded.
=========================== logs ===========================
waiting for locator("shreddit-comment[thingid=\"t1_krcsdsx\"]").first.locator("summary button").first
============================================================
I tried several times, I keep having this error.
Hey @sebbourgeois,
It seems I forgot to check for self_expand_button_loc.is_visible()...
Will fix it later today and update the PR.
I've updated the PR... I see the checks still fail, but it seems due to PRAW and not to my changes.
Anyway, can you please test this PR again and see if it works properly, please?
@sebbourgeois, probably the error you encountered was due to a self_expand_button not found as the comment itself has/was been deleted (the one mentioned in your error report https://github.com/alexlaverty/python-reddit-youtube-bot/pull/105#issuecomment-1955852732):
(not 100% sure, as I couldn't reproduce that)
@alexlaverty, any hints as why the checks are failing?
New PR, new test ! :D Thanks again for trying to fix this @azrafe7
I tried with the same subreddit, and I still face some issues with it :
Downloading screenshot 'assets/work_dir/1avl30j/comment_krb3yw4.png'...
Mmmmhhh... could not create screenshot!
2nd attempt, redirecting to comment permalink :
https://reddit.com/r/AskReddit/comments/1avl30j/whats_your_favorite_this_much_of_this_will_kill/krb3yw4/
Element 'shreddit-app' exists on the page.
Downloading screenshots... ββββββββββββββββββββββββββββββββββββββΊββ 94% -:--:--
Traceback (most recent call last):
File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 440, in <module>
process_submissions(submissions)
File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 64, in process_submissions
process_submission(submission)
File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 118, in process_submission
vid.create(
File "/home/sb/videos-creation/python-reddit-youtube-bot/video_generation/video.py", line 500, in create
download_screenshots_of_reddit_posts(
File "/home/sb/videos-creation/python-reddit-youtube-bot/comments/screenshot.py", line 333, in download_screenshots_of_reddit_posts
comment_loc.screenshot(path=comment_path)
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 17227, in screenshot
self._sync(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
return task.result()
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 491, in screenshot
return await self._with_element(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 103, in _with_element
handle = await self.element_handle(timeout=timeout)
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 297, in element_handle
handle = await self._frame.wait_for_selector(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 322, in wait_for_selector
await self._channel.send("waitForSelector", locals_to_params(locals()))
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 61, in send
return await self._connection.wrap_api_call(
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 461, in wrap_api_call
return await cb()
File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 96, in inner_send
result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("shreddit-comment[thingid=\"t1_krb3yw4\"]").first
============================================================
Overall, I tried to launch the script with no specific url, and I still got way more errors with this version than the one in the current main branch. I can provide further logs if you need !
I'm playing around tonight with some code and finding that reguarly the page is throwing the following error :
"Oops, something went wrong, please try again later."
this would probably explain why it randomly bombs out with timeouts.
I've made a function that'll loop and reload and retry if the above error shows up :
def reload_if_oops(page, max_retries=25, wait_time=5):
for _ in range(max_retries):
if check_text_in_page_content(page, "Oops, something went wrong, please try again later."):
print(f"Oops found, reloading page! {str(_)} tries...")
sleep(wait_time)
page.reload() # Reload the page if error message is found
else:
return # Exit the loop if the error message is not found
print("Max retries reached. Oops message still present.")
Can see when running the script the error is regularly showing up and takes a number of reloads and retries before it goes away...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krp4554/
Oops found, reloading page! 0 tries...
Oops found, reloading page! 1 tries...
Oops found, reloading page! 2 tries...
Oops found, reloading page! 3 tries...
Oops found, reloading page! 4 tries...
Oops found, reloading page! 5 tries...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krou8eq/
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krormu7/
Oops found, reloading page! 0 tries...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krpqqo5/
Oops found, reloading page! 0 tries...
Oops found, reloading page! 1 tries...
Oops found, reloading page! 2 tries...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krpt3lq/
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krqcalg/
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krqfreg/
Oops found, reloading page! 0 tries...
the new UI is ridiculously unstable, unless they're purposefully throttling people...
@alexlaverty, I've not encountered the "oops" thing yet. Have you pushed your changes to some branch?
@sebbourgeois, I can't reproduce your error. All screenshots get downloaded correctly:
Anyway if you can provide a full log I can look further into it.
Also, in settings.py
, try setting screenshot_debug
to True and headless_browser
to False, and relaunch the app.
Hopefully with that we'll figure out where and how it gets stuck.
Other thought I had was instead of trying to screenscrape and launch chrome browser and traverse comments was to either use Python Pillow or a custom comment html template to do a convincing mock up of a reddit comment, that way you could just generate the image locally and insert that into the video, would be 1000x faster to generate and would be pretty fool proof. wouldn't have to worry about pop ups and time outs and Oops errors etc.
@alexlaverty, I've had the exact same thought π, and even started fiddling with it a couple of days ago (https://github.com/azrafe7/python-reddit-youtube-bot/tree/screenshot_from_data)