Closed chdcl closed 1 year ago
Thanks! Could you send any one of a pull request / patch / your code?
(If not, I can follow your instructions to replicate)
I'll just drop the code because I mangled it quite a bit, I hope that's alright!
The threshold works on my machine (Windows 10) in 1080p fullscreen and ~900x900 windowed. I'm not quite sure about the exact values, but correct matches have an m.distance as low as 50, and I have not gotten any false positives or false negatives yet.
I've also included a try / catch around the call to _find_and_click, because an OpenCV error occurs if there is nothing on the screen.
On a side note, pyautogui.screenshot() seems to only screenshot the main screen (at least on Windows 10), resulting in the script failing if the browser window is on a different monitor. This can be easily worked around by setting the other monitor to the main monitor in the display settings, but this would probably be worth pointing out - putting the browser on the second monitor seems like a rather common use case :)
# pylint: disable=missing-module-docstring
import os
import logging
import random
import sys
import time
import click
import cv2 as cv # type: ignore
import numpy as np
import PIL
import PIL.ImageOps
import pyautogui # type: ignore
@click.command()
@click.option('--sleep_max', default=5.)
@click.option('--sleep_min', default=0.)
def run(sleep_max: float, sleep_min: float) -> None: # pylint: disable=missing-function-docstring
logging.basicConfig(
datefmt='%m/%d/%Y %I:%M:%S %p',
format='%(asctime)s [%(levelname)s] %(message)s',
level=logging.INFO,
)
templates = _get_templates()
while True:
sleep_seconds = random.uniform(sleep_min, sleep_max)
logging.info('Sleeping for %f seconds', sleep_seconds)
time.sleep(sleep_seconds)
try:
_find_and_click(templates)
except cv.error:
logging.info("ignoring error")
pass
def _find_and_click(templates: dict[str, np.ndarray]) -> None:
screenshot_image = pyautogui.screenshot()
screenshot = _image_to_grayscale_array(screenshot_image)
names = list(templates.keys())
names.sort(key=lambda x: x == "click_here")
for name in names:
template = templates[name]
sift = cv.SIFT_create() # pylint: disable=no-member
m_threshold = 150
if name == "click_here":
m_threshold = 80
_, template_descriptors = sift.detectAndCompute(template, mask=None)
screenshot_keypoints, screenshot_descriptors = sift.detectAndCompute(screenshot, mask=None)
matcher = cv.BFMatcher() # pylint: disable=no-member
matches = matcher.knnMatch(template_descriptors, screenshot_descriptors, k=2)
points = np.array([screenshot_keypoints[m.trainIdx].pt for m, n in matches if m.distance < m_threshold])
for m, n in matches:
logging.debug(f"{name=}, {m.distance=}, {n.distance=}, {screenshot_keypoints[m.trainIdx].pt=}")
if points.shape[0] == 0:
continue
point = np.median(points, axis=0)
pyautogui.click(*point)
logging.info('Clicking on %s at coordinates x=%f y=%f', name, *point)
return
logging.info('No matches found')
def _get_templates() -> dict[str, np.ndarray]:
arrays = {}
try:
# pylint: disable=no-member,protected-access
root_dir = sys._MEIPASS # type: ignore
except AttributeError:
root_dir = '.'
templates_dir = os.path.join(root_dir, 'templates')
basenames = os.listdir(templates_dir)
for basename in basenames:
name, _ = os.path.splitext(basename)
path = os.path.join(templates_dir, basename)
image = PIL.Image.open(path) # pylint: disable=no-member
array = _image_to_grayscale_array(image)
arrays[name] = array
return arrays
def _image_to_grayscale_array(image: PIL.Image.Image) -> np.ndarray:
image = PIL.ImageOps.grayscale(image)
array = np.array(image)
return array
if __name__ == '__main__':
run() # pylint: disable=no-value-for-parameter
Can confirm, replacing nexus_autodl.py fully with the code above fixed the issue.
Sorry for the late response. I've cleaned up the changes and put them in a PR. Since I've never been able to replicate the click here loop, can someone verify that the corresponding branch bugfix/click-here-loop works as intended before I merge and release a new binary?
I have merged and updated the binary.
Hi,
Thanks for making this! I was having some problems with the script getting stuck due to mis-identifying mod names as "click here", and I was able to solve them with two small changes:
m.distance < 0.5 * n.distance
tom.distance < threshold
, where threshold is a value which i manually set to 150 for slow_download and 80 for click_hereI have no familiarity with opencv or image recognition in general, these were just values i observed. Some incorrect matches for click_here had a smaller m.distance and n.distance than correct matches for slow_download, hence the difference.
So far this is working without error, point 2 is a bit dodgy but evaluating slow_download before click_here should be a solid fix with no downside if you feel like working on this project still :) Thanks again!