internetarchive / umbra

A queue-controlled browser automation tool for improving web crawl quality
Apache License 2.0
60 stars 25 forks source link

Add scrolling and clicking behavior #42

Closed vonrosen closed 9 years ago

nlevitt commented 9 years ago

I started out by writing the comments below, then I thought it might be simpler to implement, so I made this branch: https://github.com/nlevitt/umbra/tree/aitfive-451-alt It's not clear to me if it's working or not. :) That's why I'm passing this back over to you with my comments and my alternative implementation.

I would simplify the iframes handling. What you've got if I'm reading this right, is an extra setting "iframe_css_selector" which tells it to look inside the (first) matching iframe, and only there, for elements matching "click_css_selector". Instead, let's make it more generic and easier to configure, by eliminating the "iframe_css_selector" setting, and looking inside every iframe for things to click, as well as the top level page. That won't break anything we know of, will it?

Question: the case of the element that needs a mouseover, does it also need a click? I think it's fine to mouseover and the click everything, just curious.

Can you switch to camelcase for js variables, i.e. cssSelector instead of css_selector, for consistency. Can you also fix the formatting a little bit? Maybe just put these kinds of statements on a single line

+                               : document
+                                               .querySelectorAll(css_selector);
+                                       var mouseOverEvent = document
+                                                       .createEvent('Events');
+                                       mouseOverEvent.initEvent("mouseover",
+                                                       true, false);
+                                       clickTargets[i]
+                                                       .dispatchEvent(mouseOverEvent);
+                               console
+                                               .log("scrolling UP because everything on this screen has been clicked but we missed something above");