[Suggestion] Fix for click_here loop

[chdcl]

Hi,

Thanks for making this! I was having some problems with the script getting stuck due to mis-identifying mod names as "click here", and I was able to solve them with two small changes:

1. Match screenshot with slow_download before click_here
2. Change match criteria from m.distance < 0.5 * n.distance to m.distance < threshold, where threshold is a value which i manually set to 150 for slow_download and 80 for click_here

I have no familiarity with opencv or image recognition in general, these were just values i observed. Some incorrect matches for click_here had a smaller m.distance and n.distance than correct matches for slow_download, hence the difference.

So far this is working without error, point 2 is a bit dodgy but evaluating slow_download before click_here should be a solid fix with no downside if you feel like working on this project still :) Thanks again!

[parsiad]

Thanks! Could you send any one of a pull request / patch / your code?

(If not, I can follow your instructions to replicate)

[chdcl]

I'll just drop the code because I mangled it quite a bit, I hope that's alright!

The threshold works on my machine (Windows 10) in 1080p fullscreen and ~900x900 windowed. I'm not quite sure about the exact values, but correct matches have an m.distance as low as 50, and I have not gotten any false positives or false negatives yet.

I've also included a try / catch around the call to _find_and_click, because an OpenCV error occurs if there is nothing on the screen.

On a side note, pyautogui.screenshot() seems to only screenshot the main screen (at least on Windows 10), resulting in the script failing if the browser window is on a different monitor. This can be easily worked around by setting the other monitor to the main monitor in the display settings, but this would probably be worth pointing out - putting the browser on the second monitor seems like a rather common use case :)

# pylint: disable=missing-module-docstring

import os
import logging
import random
import sys
import time

import click
import cv2 as cv  # type: ignore
import numpy as np
import PIL
import PIL.ImageOps
import pyautogui  # type: ignore

@click.command()
@click.option('--sleep_max', default=5.)
@click.option('--sleep_min', default=0.)
def run(sleep_max: float, sleep_min: float) -> None:  # pylint: disable=missing-function-docstring
    logging.basicConfig(
        datefmt='%m/%d/%Y %I:%M:%S %p',
        format='%(asctime)s [%(levelname)s] %(message)s',
        level=logging.INFO,
    )
    templates = _get_templates()
    while True:
        sleep_seconds = random.uniform(sleep_min, sleep_max)
        logging.info('Sleeping for %f seconds', sleep_seconds)
        time.sleep(sleep_seconds)
        try:
            _find_and_click(templates)
        except cv.error:
            logging.info("ignoring error")
            pass

def _find_and_click(templates: dict[str, np.ndarray]) -> None:
    screenshot_image = pyautogui.screenshot()
    screenshot = _image_to_grayscale_array(screenshot_image)
    names = list(templates.keys())
    names.sort(key=lambda x: x == "click_here")
    for name in names:
        template = templates[name]
        sift = cv.SIFT_create()  # pylint: disable=no-member
        m_threshold = 150
        if name == "click_here":
            m_threshold = 80
        _, template_descriptors = sift.detectAndCompute(template, mask=None)
        screenshot_keypoints, screenshot_descriptors = sift.detectAndCompute(screenshot, mask=None)
        matcher = cv.BFMatcher()  # pylint: disable=no-member
        matches = matcher.knnMatch(template_descriptors, screenshot_descriptors, k=2)
        points = np.array([screenshot_keypoints[m.trainIdx].pt for m, n in matches if m.distance < m_threshold])
        for m, n in matches:
            logging.debug(f"{name=}, {m.distance=}, {n.distance=}, {screenshot_keypoints[m.trainIdx].pt=}")
        if points.shape[0] == 0:
            continue
        point = np.median(points, axis=0)
        pyautogui.click(*point)
        logging.info('Clicking on %s at coordinates x=%f y=%f', name, *point)
        return
    logging.info('No matches found')

def _get_templates() -> dict[str, np.ndarray]:
    arrays = {}
    try:
        # pylint: disable=no-member,protected-access
        root_dir = sys._MEIPASS  # type: ignore
    except AttributeError:
        root_dir = '.'
    templates_dir = os.path.join(root_dir, 'templates')
    basenames = os.listdir(templates_dir)
    for basename in basenames:
        name, _ = os.path.splitext(basename)
        path = os.path.join(templates_dir, basename)
        image = PIL.Image.open(path)  # pylint: disable=no-member
        array = _image_to_grayscale_array(image)
        arrays[name] = array
    return arrays

def _image_to_grayscale_array(image: PIL.Image.Image) -> np.ndarray:
    image = PIL.ImageOps.grayscale(image)
    array = np.array(image)
    return array

if __name__ == '__main__':
    run()  # pylint: disable=no-value-for-parameter

[jstawik]

Can confirm, replacing nexus_autodl.py fully with the code above fixed the issue.

[parsiad]

Sorry for the late response. I've cleaned up the changes and put them in a PR. Since I've never been able to replicate the click here loop, can someone verify that the corresponding branch bugfix/click-here-loop works as intended before I merge and release a new binary?

[parsiad]

I have merged and updated the binary.