[screenshot] more fixes

azrafe7 commented 6 months ago

login: try alternate method if no frame_loc_found
fix not_login_url_regex (hopefully)
use_permalinks if comment not found
print comment_excerpt
expand comment if collapsed

azrafe7 commented 6 months ago

Hello, I've made another round of (hopefully) fixes. This time I based it on your main branch :P

One of the things it changes is switching to use permalinks if the comment is not found on the main-thread page (more on that later), and handle the case when the comment itself is collapsed, which led to screenshot like this one (with no comment text visible):

About the use_permalinks feature, here's how it works (as of now):

it checks for the comment in the main-thread page, and screenshots that if it exists
if it doesn't exist, it toggles use_permalinks to True, and all current-and-future comments will trigger a goto(permalink)

As loading each permalink is more robust, but also slower, I think this could be optimized by building a separate queue. So, while processing accepted_comments:

update a list of comments not present on page
continue to process the ones that instead are in the main-thread
at the end process the ones in the not_found_comments, by using their permalink

Anyway... thanks for merging my previous PR, hopefully this one could be useful too.

Cheers, azrafe7

azrafe7 commented 6 months ago

Another issue I've encountered (but can reproduce right now) is screenshots that seems to have an overlay, or are 'dimmed' (in other words, with a dark theme, the text is grey instead of the usual white).

If you can reproduce that please let me know 😉

sebbourgeois commented 6 months ago

Hey @azrafe7 !

Not sure about this change, it throws way more errors than the current version of the main repo. ie. if I try to generate a video from this subreddit ; https://www.reddit.com/r/AskReddit/comments/1avl30j/whats_your_favorite_this_much_of_this_will_kill/

With the actual version of screenshot.py, it works fine. With your updated version, it fails with this error :

Downloading screenshots... ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━  57% -:--:--
Traceback (most recent call last):
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 440, in <module>
    process_submissions(submissions)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 64, in process_submissions
    process_submission(submission)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 118, in process_submission
    vid.create(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/video_generation/video.py", line 500, in create
    download_screenshots_of_reddit_posts(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/comments/screenshot.py", line 253, in download_screenshots_of_reddit_posts
    self_expand_button_loc.dispatch_event('click')
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 15591, in dispatch_event
    self._sync(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
    return task.result()
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 171, in dispatch_event
    return await self._frame.dispatch_event(self._selector, strict=True, **params)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 363, in dispatch_event
    await self._channel.send(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 461, in wrap_api_call
    return await cb()
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 96, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 60000ms exceeded.
=========================== logs ===========================
waiting for locator("shreddit-comment[thingid=\"t1_krcsdsx\"]").first.locator("summary button").first
============================================================

I tried several times, I keep having this error.

azrafe7 commented 6 months ago

Hey @sebbourgeois,

It seems I forgot to check for self_expand_button_loc.is_visible()...

Will fix it later today and update the PR.

azrafe7 commented 6 months ago

I've updated the PR... I see the checks still fail, but it seems due to PRAW and not to my changes.

Anyway, can you please test this PR again and see if it works properly, please?

@sebbourgeois, probably the error you encountered was due to a self_expand_button not found as the comment itself has/was been deleted (the one mentioned in your error report https://github.com/alexlaverty/python-reddit-youtube-bot/pull/105#issuecomment-1955852732):

(not 100% sure, as I couldn't reproduce that)

azrafe7 commented 6 months ago

@alexlaverty, any hints as why the checks are failing?

sebbourgeois commented 6 months ago

New PR, new test ! :D Thanks again for trying to fix this @azrafe7

I tried with the same subreddit, and I still face some issues with it :

Downloading screenshot 'assets/work_dir/1avl30j/comment_krb3yw4.png'...
Mmmmhhh... could not create screenshot!
2nd attempt, redirecting to comment permalink :
https://reddit.com/r/AskReddit/comments/1avl30j/whats_your_favorite_this_much_of_this_will_kill/krb3yw4/
Element 'shreddit-app' exists on the page.
Downloading screenshots... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━  94% -:--:--
Traceback (most recent call last):
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 440, in <module>
    process_submissions(submissions)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 64, in process_submissions
    process_submission(submission)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 118, in process_submission
    vid.create(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/video_generation/video.py", line 500, in create
    download_screenshots_of_reddit_posts(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/comments/screenshot.py", line 333, in download_screenshots_of_reddit_posts
    comment_loc.screenshot(path=comment_path)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 17227, in screenshot
    self._sync(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
    return task.result()
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 491, in screenshot
    return await self._with_element(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 103, in _with_element
    handle = await self.element_handle(timeout=timeout)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 297, in element_handle
    handle = await self._frame.wait_for_selector(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 322, in wait_for_selector
    await self._channel.send("waitForSelector", locals_to_params(locals()))
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 461, in wrap_api_call
    return await cb()
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 96, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("shreddit-comment[thingid=\"t1_krb3yw4\"]").first
============================================================

Overall, I tried to launch the script with no specific url, and I still got way more errors with this version than the one in the current main branch. I can provide further logs if you need !

alexlaverty commented 6 months ago

I'm playing around tonight with some code and finding that reguarly the page is throwing the following error :

"Oops, something went wrong, please try again later."

this would probably explain why it randomly bombs out with timeouts.

I've made a function that'll loop and reload and retry if the above error shows up :

def reload_if_oops(page, max_retries=25, wait_time=5):
    for _ in range(max_retries):
        if check_text_in_page_content(page, "Oops, something went wrong, please try again later."):
            print(f"Oops found, reloading page! {str(_)} tries...")
            sleep(wait_time)
            page.reload()  # Reload the page if error message is found
        else:
            return  # Exit the loop if the error message is not found

    print("Max retries reached. Oops message still present.")

Can see when running the script the error is regularly showing up and takes a number of reloads and retries before it goes away...

Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krp4554/
Oops found, reloading page! 0 tries...
Oops found, reloading page! 1 tries...
Oops found, reloading page! 2 tries...
Oops found, reloading page! 3 tries...
Oops found, reloading page! 4 tries...
Oops found, reloading page! 5 tries...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krou8eq/
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krormu7/
Oops found, reloading page! 0 tries...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krpqqo5/
Oops found, reloading page! 0 tries...
Oops found, reloading page! 1 tries...
Oops found, reloading page! 2 tries...
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krpt3lq/
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krqcalg/
Comment Permalink : https://reddit.com/r/AskReddit/comments/1axlxqo/what_is_so_unethical_it_should_be_illegal/krqfreg/
Oops found, reloading page! 0 tries...

the new UI is ridiculously unstable, unless they're purposefully throttling people...

azrafe7 commented 6 months ago

@alexlaverty, I've not encountered the "oops" thing yet. Have you pushed your changes to some branch?

@sebbourgeois, I can't reproduce your error. All screenshots get downloaded correctly:

Anyway if you can provide a full log I can look further into it.

Also, in settings.py, try setting screenshot_debug to True and headless_browser to False, and relaunch the app. Hopefully with that we'll figure out where and how it gets stuck.

alexlaverty commented 6 months ago

Other thought I had was instead of trying to screenscrape and launch chrome browser and traverse comments was to either use Python Pillow or a custom comment html template to do a convincing mock up of a reddit comment, that way you could just generate the image locally and insert that into the video, would be 1000x faster to generate and would be pretty fool proof. wouldn't have to worry about pop ups and time outs and Oops errors etc.

azrafe7 commented 6 months ago

@alexlaverty, I've had the exact same thought 😄, and even started fiddling with it a couple of days ago (https://github.com/azrafe7/python-reddit-youtube-bot/tree/screenshot_from_data)

alexlaverty / python-reddit-youtube-bot

[screenshot] more fixes #105