sneakypete81 / updatescanner

Firefox addon to monitor web pages for updates.
http://sneakypete81.github.io/updatescanner/
GNU General Public License v3.0
204 stars 31 forks source link

Option to ignore HTML-only (i.e. non-visible) changes? #298

Closed johnhawkinson closed 5 years ago

johnhawkinson commented 5 years ago

There are some webpages that I monitor that regularly change non-visible elements (e.g. javascript data associated with site-wide navigation or whatever), so Update Scanner alerts me that there is a change, but then displays no yellow highlighting (actual changes) on the page.

This is kind of frustrating/annoying. I would like to mark such pages for special filtering, and have US only tell me about visible changes on those pages.

(I don't think I would like to do that by default on all pages—a failure to highlight might reflect a hidden page element or a structural issue I would like to know about. So I'd rather verify from the source that such a change is of this nature before excluding such pages. On the other hand, most users won't be able to review HTML source for changes, so perhaps having such an option doesn't make as much sense as I initially thought. Perhaps the behavior should be on by default.)

I realize it may be hard to do this optimally — presumably US doesn't render the highlighting until it actually opens the page, which is after it appears on the list. I'm not sure the best way to handle that. Of course it could go through and remove tabs it just created if they turn out not to have visible changes.

That sounds like an ugly hack, but it'd be better than the current state. Probably a more elegant solution is available?

sneakypete81 commented 5 years ago

Could you please provide an example URL to demonstrate this behaviour?

The "scanning" operation ought to strip out changes within tags, but it's fairly limited - we've optimised a bit for speed to prevent performance problems when scanning. This means that it will sometimes trigger on non-visible changes, unfortunately.

I think the best solution to this is probably #50. Unfortunately noone's had time to implement this, and it would be a fair bit of work.

johnhawkinson commented 5 years ago

Sure. Sorry, I thought this was a sufficiently well-known issue that examples weren't needed, but it's good to hear that's not the case. http://www.uscourts.gov/about-federal-courts/cameras-courts is one of my examples, it seems to change every few days, sometimes multiple times a day.

Here's a diff:

--- "Cameras in Courts _ United States Courts-old(1).html"  2018-11-26 13:48:33.000000000 -0500
+++ "Cameras in Courts _ United States Courts-new(1).html"  2018-11-26 13:48:37.000000000 -0500
@@ -130,7 +130,7 @@
 <p>
    Filter by court, subject matter, and procedural posture to find the case you want to view. 
 </p>
-</div><div class="contextual-links-region"><div class="view view-cameras-in-court-landing view-id-cameras_in_court_landing view-display-id-entityreference_view_widget_1 view-dom-id-2c810087275decb6227488ef8c4f4621">
+</div><div class="contextual-links-region"><div class="view view-cameras-in-court-landing view-id-cameras_in_court_landing view-display-id-entityreference_view_widget_1 view-dom-id-0ef7deb6d6359824d5a913096cf414f1">

       <div class="view-filters">
       <form action="/about-federal-courts/cameras-courts" method="get" id="views-exposed-form-cameras-in-court-landing-entityreference-view-widget-1" accept-charset="UTF-8"><div><div class="views-exposed-form clearfix">
@@ -374,25 +374,25 @@
               </tr>
           <tr class="odd">
                           <td data-th="Title:" class="views-field views-field-title" >
-            <a href="/cameras-courts/waymo-llc-vs-uber-technologies">Waymo LLC vs. Uber Technologies</a>          </td>
+            <a href="/cameras-courts/regents-university-california-v-united-states-department-homeland-security">Regents of University of California v. United States Department of Homeland Security</a>          </td>
                   <td data-th="Name:" class="views-field views-field-name" >
             Northern District of California          </td>
                   <td data-th="Subject Matter:" >
-            Patent          </td>
+            Administrative Procedure Act          </td>
                   <td data-th="Procedural Posture:" >
-            Technology Tutorial Hearing          </td>
+            Case Management Conference          </td>
                   <td data-th="Last Updated:" class="views-field views-field-field-date-updated" >
             <span class="date-display-single">November 7, 2017</span>          </td>
               </tr>
           <tr class="even">
                           <td data-th="Title:" class="views-field views-field-title" >
-            <a href="/cameras-courts/regents-university-california-v-united-states-department-homeland-security">Regents of University of California v. United States Department of Homeland Security</a>          </td>
+            <a href="/cameras-courts/waymo-llc-vs-uber-technologies">Waymo LLC vs. Uber Technologies</a>          </td>
                   <td data-th="Name:" class="views-field views-field-name" >
             Northern District of California          </td>
                   <td data-th="Subject Matter:" >
-            Administrative Procedure Act          </td>
+            Patent          </td>
                   <td data-th="Procedural Posture:" >
-            Case Management Conference          </td>
+            Technology Tutorial Hearing          </td>
                   <td data-th="Last Updated:" class="views-field views-field-field-date-updated" >
             <span class="date-display-single">November 7, 2017</span>          </td>
               </tr>
@@ -767,7 +767,7 @@
   <script src="//w.sharethis.com/button/buttons.js"></script>

 <!--[if lte IE 9]>
-<script src="/sites/default/files/advagg_js/js__74vIRHeFUQy-tIqAAnJg3k7_huzoZxxmo76NsjT9I2E__yNaa1WyQNI_iUgrGwjmFDR4aHOXSAgxX9jSiRoqgMAQ__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js#ie9-" onload="if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(true);};
+<script src="/sites/default/files/advagg_js/js__7H4XIeYL4y8sxMES9eGREfTi2jQRLIRUdHsFtf2KZs4__K93o9O-gG4TCansPl6u1XLocskub8uDuFS-UdNKzigk__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js#ie9-" onload="if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(true);};
 function advagg_mod_1() {
   // Count how many times this function is called.
   advagg_mod_1.count = ++advagg_mod_1.count || 1;
@@ -802,7 +802,7 @@
 <![endif]-->

 <!--[if gt IE 9]>
-<script src="/sites/default/files/advagg_js/js__74vIRHeFUQy-tIqAAnJg3k7_huzoZxxmo76NsjT9I2E__yNaa1WyQNI_iUgrGwjmFDR4aHOXSAgxX9jSiRoqgMAQ__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js#ie10+" defer="defer" onload="if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(true);};
+<script src="/sites/default/files/advagg_js/js__7H4XIeYL4y8sxMES9eGREfTi2jQRLIRUdHsFtf2KZs4__K93o9O-gG4TCansPl6u1XLocskub8uDuFS-UdNKzigk__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js#ie10+" defer="defer" onload="if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(true);};
 function advagg_mod_1() {
   // Count how many times this function is called.
   advagg_mod_1.count = ++advagg_mod_1.count || 1;
@@ -837,7 +837,7 @@
 <![endif]-->

 <!--[if !IE]><!-->
-<script src="/sites/default/files/advagg_js/js__74vIRHeFUQy-tIqAAnJg3k7_huzoZxxmo76NsjT9I2E__yNaa1WyQNI_iUgrGwjmFDR4aHOXSAgxX9jSiRoqgMAQ__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer" onload="if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(true);};
+<script src="/sites/default/files/advagg_js/js__7H4XIeYL4y8sxMES9eGREfTi2jQRLIRUdHsFtf2KZs4__K93o9O-gG4TCansPl6u1XLocskub8uDuFS-UdNKzigk__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer" onload="if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(true);};
 function advagg_mod_1() {
   // Count how many times this function is called.
   advagg_mod_1.count = ++advagg_mod_1.count || 1;
@@ -871,8 +871,8 @@
 advagg_mod_1_check();"></script>
 <!--<![endif]-->
 <script src="/sites/default/files/advagg_js/js__OEqcZzhparL2ZlPDt8jAsxr-mWw5KWqvFbbAJlpwZmU__96uSKTq5kE7XaZFnG3Es6U5OAYpBlRj0cGm6H78HB00__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer"></script>
-<script src="/sites/default/files/advagg_js/js__NNX14JmkccjRSbLtnmNt_-tB7W5VDbeNZAj6HYDe_w8__UGJVExq6TpsgjshvuiHfDfMTziIBeakBz4eaBqE2q-0__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer"></script>
-<script src="/sites/default/files/advagg_js/js___y9XB41JXdkAQxdxKoq8OraRxaSTVKzCQF576h7W3Us__w1f8EgGYjjL3g0CntpENNW3in0LlW-rs7IaDZXJ8oIg__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer"></script>
+<script src="/sites/default/files/advagg_js/js__6qJpVeZD2daLUBVO9NTORILaP18Ro7Vc1_A_zx8zCPY__uhJGWa94x-VtOirS9mvz-qG0-d8pnr00kWH91kSwcJ4__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer"></script>
+<script src="/sites/default/files/advagg_js/js__EmaZA5mbDMjJpzGVY4DZd7O3zOxqBUwCo85w5eoo_kc__VxCW5mEe2ehSC6cKoi9J548qNcSQxnKFhPYVrLcaxnY__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer"></script>
 <script src="/sites/default/files/advagg_js/js__8e3R1DptW124gYoCM0Od2NehIqP0KX8gcx9a-EsMVSc__38gaAVIWh7MsRVy377eT3aI9X7oQsFP2S6HF6yRArC4__-JGaOhOTV30RttZs06rVKwjgQ4usFlckNbSZitG7nXs.js" defer="defer"></script>
 <script>
 function advagg_mod_2() {
@@ -938,7 +938,7 @@
   }
 }
 advagg_mod_3_check();</script>
-<script>function init_drupal_core_settings() {jQuery.extend(Drupal.settings, {"basePath":"\/","pathPrefix":"","ajaxPageState":{"theme":"usczen","theme_token":"4qiTxrVUx_5D2hLCoJNgA-inTvnWopGD0A-IzTDUn4g","css":{"modules\/system\/system.base.css":1,"modules\/system\/system.menus.css":1,"modules\/system\/system.messages.css":1,"modules\/system\/system.theme.css":1,"misc\/ui\/jquery.ui.core.css":1,"misc\/ui\/jquery.ui.theme.css":1,"misc\/ui\/jquery.ui.button.css":1,"misc\/ui\/jquery.ui.resizable.css":1,"misc\/ui\/jquery.ui.dialog.css":1,"sites\/all\/modules\/date\/date_api\/date.css":1,"sites\/all\/modules\/date\/date_popup\/themes\/datepicker.1.7.css":1,"sites\/all\/modules\/environment_indicator\/environment_indicator.css":1,"modules\/field\/theme\/field.css":1,"modules\/node\/node.css":1,"modules\/search\/search.css":1,"modules\/user\/user.css":1,"sites\/all\/modules\/extlink\/extlink.css":1,"sites\/all\/modules\/views\/css\/views.css":1,"profiles\/uscourts\/uscourts.css":1,"sites\/all\/modules\/lazyloader\/lazyloader.css":1,"sites\/all\/modules\/ckeditor\/css\/ckeditor.css":1,"sites\/all\/modules\/ctools\/css\/ctools.css":1,"profiles\/uscourts\/modules\/custom\/usc_alert\/css\/usc-alert.css":1,"profiles\/uscourts\/libraries\/sidr\/stylesheets\/jquery.sidr.dark.css":1,"sites\/all\/libraries\/superfish\/css\/superfish.css":1,"profiles\/uscourts\/themes\/usczen\/css\/navigation.css":1,"profiles\/uscourts\/themes\/usczen\/system.menus.css":1,"profiles\/uscourts\/themes\/usczen\/node.css":1,"profiles\/uscourts\/themes\/usczen\/system.messages.css":1,"profiles\/uscourts\/themes\/usczen\/system.theme.css":1,"profiles\/uscourts\/themes\/usczen\/search.css":1,"profiles\/uscourts\/themes\/usczen\/user.css":1,"profiles\/uscourts\/themes\/usczen\/ctools.css":1,"profiles\/uscourts\/themes\/usczen\/views.css":1,"profiles\/uscourts\/themes\/usczen\/date.css":1,"profiles\/uscourts\/themes\/usczen\/datepicker.1.7.css":1,"profiles\/uscourts\/themes\/usczen\/jquery.sidr.dark.css":1,"profiles\/uscourts\/themes\/usczen\/mediaelementplayer.min.css":1,"profiles\/uscourts\/themes\/usczen\/css\/normalize.css":1,"profiles\/uscourts\/themes\/usczen\/css\/font-awesome.min.css":1,"profiles\/uscourts\/themes\/usczen\/css\/components.css":1,"profiles\/uscourts\/themes\/usczen\/css\/jquery.sidr.light.css":1,"profiles\/uscourts\/themes\/usczen\/mediaelement\/mediaelementplayer.css":1,"profiles\/uscourts\/themes\/usczen\/css\/block.bean.css":1,"profiles\/uscourts\/themes\/usczen\/css\/styles.css":1,"profiles\/uscourts\/themes\/usczen\/css\/layout.css":1,"profiles\/uscourts\/themes\/usczen\/css\/custom.css":1,"profiles\/uscourts\/themes\/usczen\/css\/print.css":1,"profiles\/uscourts\/themes\/usczen\/css\/ie8.css":1},"js":{"http:\/\/w.sharethis.com\/button\/buttons.js":1,"misc\/jquery.js":1,"misc\/jquery.once.js":1,"misc\/drupal.js":1,"misc\/ui\/jquery.ui.core.min.js":1,"misc\/ui\/jquery.ui.widget.min.js":1,"misc\/ui\/jquery.ui.button.min.js":1,"misc\/ui\/jquery.ui.mouse.min.js":1,"misc\/ui\/jquery.ui.draggable.min.js":1,"misc\/ui\/jquery.ui.position.min.js":1,"misc\/ui\/jquery.ui.resizable.min.js":1,"misc\/ui\/jquery.ui.dialog.min.js":1,"sites\/all\/modules\/views\/js\/jquery.ui.dialog.patch.js":1,"misc\/jquery.cookie.js":1,"misc\/jquery.form.js":1,"misc\/ajax.js":1,"sites\/all\/modules\/environment_indicator\/tinycon.min.js":1,"sites\/all\/modules\/environment_indicator\/environment_indicator.js":1,"sites\/all\/modules\/environment_indicator\/color.js":1,"sites\/all\/modules\/extlink\/extlink.js":1,"sites\/all\/modules\/linkit\/js\/linkit.js":1,"misc\/progress.js":1,"sites\/all\/modules\/linkit\/editors\/ckeditor\/linkitDialog.js":1,"profiles\/uscourts\/modules\/custom\/usc_alert\/js\/usc-alert.js":1,"sites\/all\/modules\/views\/js\/base.js":1,"sites\/all\/modules\/views\/js\/ajax_view.js":1,"sites\/all\/modules\/google_analytics\/googleanalytics.js":1,"profiles\/uscourts\/libraries\/sidr\/jquery.sidr.min.js":1,"sites\/all\/libraries\/superfish\/superfish.js":1,"sites\/all\/modules\/superfish\/superfish.js":1,"profiles\/uscourts\/themes\/usczen\/js\/jquery.tablesorter.min.js":1,"profiles\/uscourts\/themes\/usczen\/js\/responsive_menus_sidr.js":1,"profiles\/uscourts\/themes\/usczen\/js\/script.js":1}},"linkit":{"autocompletePath":"http:\/\/www.uscourts.gov\/linkit\/autocomplete\/___profile___?s=","dashboardPath":"\/linkit\/dashboard\/","currentInstance":{}},"profile":"uscourts","profile_version":"7.x-1.5","urlIsAjaxTrusted":{"\/about-federal-courts\/cameras-courts":true,"\/views\/ajax":true},"views":{"ajax_path":"\/views\/ajax","ajaxViews":{"views_dom_id:2c810087275decb6227488ef8c4f4621":{"view_name":"cameras_in_court_landing","view_display_id":"entityreference_view_widget_1","view_args":"","view_path":"node\/235","view_base_path":"node\/235","view_dom_id":"2c810087275decb6227488ef8c4f4621","pager_element":0}}},"extlink":{"extTarget":0,"extClass":"ext","extLabel":"(link is external)","extImgClass":0,"extSubdomains":1,"extExclude":"\\.gov","extInclude":"","extCssExclude":"#footer-wrapper, #block-usc-settings-share-this, #content-usc-share-this","extCssExplicit":"","extAlert":"_blank","extAlertText":"You are now leaving uscourts.gov. \r\n\r\nThe Administrative Office of the U.S. Courts (AOUSC) provides this link for your convenience. The AOUSC takes no responsibility for and exercises no control over the organizations, views, accuracy, copyright or trademark compliance, or legality of the material contained on the site you are about to access. The content of the site you are about to access does not necessarily reflect the opinions, standards, or policies of the AOUSC. \r\n\r\nThank you for visiting our site.","mailtoClass":"mailto","mailtoLabel":"(link sends e-mail)"},"googleanalytics":{"trackOutbound":1,"trackMailto":1,"trackDownload":1,"trackDownloadExtensions":"7z|aac|arc|arj|asf|asx|avi|bin|csv|doc(x|m)?|dot(x|m)?|exe|flv|gif|gz|gzip|hqx|jar|jpe?g|js|mp(2|3|4|e?g)|mov(ie)?|msi|msp|pdf|phps|png|ppt(x|m)?|pot(x|m)?|pps(x|m)?|ppam|sld(x|m)?|thmx|qtm?|ra(m|r)?|sea|sit|tar|tgz|torrent|txt|wav|wma|wmv|wpd|xls(x|m|b)?|xlt(x|m)|xlam|xml|z|zip"},"superfish":{"1":{"id":"1","sf":{"delay":"0","animation":{"opacity":"show","height":"show"},"speed":"\u0027fast\u0027","autoArrows":false,"dropShadows":false,"disableHI":false},"plugins":{"supposition":false,"bgiframe":false}}}}); 
+<script>function init_drupal_core_settings() {jQuery.extend(Drupal.settings, {"basePath":"\/","pathPrefix":"","ajaxPageState":{"theme":"usczen","theme_token":"yO_3z8ZwPiz_3I0FoUfRTre4qNYrz3q7GkBBWdorvaQ","css":{"modules\/system\/system.base.css":1,"modules\/system\/system.menus.css":1,"modules\/system\/system.messages.css":1,"modules\/system\/system.theme.css":1,"misc\/ui\/jquery.ui.core.css":1,"misc\/ui\/jquery.ui.theme.css":1,"misc\/ui\/jquery.ui.button.css":1,"misc\/ui\/jquery.ui.resizable.css":1,"misc\/ui\/jquery.ui.dialog.css":1,"sites\/all\/modules\/date\/date_api\/date.css":1,"sites\/all\/modules\/date\/date_popup\/themes\/datepicker.1.7.css":1,"sites\/all\/modules\/environment_indicator\/environment_indicator.css":1,"modules\/field\/theme\/field.css":1,"modules\/node\/node.css":1,"modules\/search\/search.css":1,"modules\/user\/user.css":1,"sites\/all\/modules\/extlink\/extlink.css":1,"sites\/all\/modules\/views\/css\/views.css":1,"profiles\/uscourts\/uscourts.css":1,"sites\/all\/modules\/lazyloader\/lazyloader.css":1,"sites\/all\/modules\/ckeditor\/css\/ckeditor.css":1,"sites\/all\/modules\/ctools\/css\/ctools.css":1,"profiles\/uscourts\/modules\/custom\/usc_alert\/css\/usc-alert.css":1,"profiles\/uscourts\/libraries\/sidr\/stylesheets\/jquery.sidr.dark.css":1,"sites\/all\/libraries\/superfish\/css\/superfish.css":1,"profiles\/uscourts\/themes\/usczen\/css\/navigation.css":1,"profiles\/uscourts\/themes\/usczen\/system.menus.css":1,"profiles\/uscourts\/themes\/usczen\/node.css":1,"profiles\/uscourts\/themes\/usczen\/system.messages.css":1,"profiles\/uscourts\/themes\/usczen\/system.theme.css":1,"profiles\/uscourts\/themes\/usczen\/search.css":1,"profiles\/uscourts\/themes\/usczen\/user.css":1,"profiles\/uscourts\/themes\/usczen\/ctools.css":1,"profiles\/uscourts\/themes\/usczen\/views.css":1,"profiles\/uscourts\/themes\/usczen\/date.css":1,"profiles\/uscourts\/themes\/usczen\/datepicker.1.7.css":1,"profiles\/uscourts\/themes\/usczen\/jquery.sidr.dark.css":1,"profiles\/uscourts\/themes\/usczen\/mediaelementplayer.min.css":1,"profiles\/uscourts\/themes\/usczen\/css\/normalize.css":1,"profiles\/uscourts\/themes\/usczen\/css\/font-awesome.min.css":1,"profiles\/uscourts\/themes\/usczen\/css\/components.css":1,"profiles\/uscourts\/themes\/usczen\/css\/jquery.sidr.light.css":1,"profiles\/uscourts\/themes\/usczen\/mediaelement\/mediaelementplayer.css":1,"profiles\/uscourts\/themes\/usczen\/css\/block.bean.css":1,"profiles\/uscourts\/themes\/usczen\/css\/styles.css":1,"profiles\/uscourts\/themes\/usczen\/css\/layout.css":1,"profiles\/uscourts\/themes\/usczen\/css\/custom.css":1,"profiles\/uscourts\/themes\/usczen\/css\/print.css":1,"profiles\/uscourts\/themes\/usczen\/css\/ie8.css":1},"js":{"http:\/\/w.sharethis.com\/button\/buttons.js":1,"misc\/jquery.js":1,"misc\/jquery.once.js":1,"misc\/drupal.js":1,"misc\/ui\/jquery.ui.core.min.js":1,"misc\/ui\/jquery.ui.widget.min.js":1,"misc\/jquery.cookie.js":1,"misc\/jquery.form.js":1,"misc\/ui\/jquery.ui.button.min.js":1,"misc\/ui\/jquery.ui.mouse.min.js":1,"misc\/ui\/jquery.ui.draggable.min.js":1,"misc\/ui\/jquery.ui.position.min.js":1,"misc\/ui\/jquery.ui.resizable.min.js":1,"misc\/ui\/jquery.ui.dialog.min.js":1,"sites\/all\/modules\/views\/js\/jquery.ui.dialog.patch.js":1,"misc\/ajax.js":1,"sites\/all\/modules\/environment_indicator\/tinycon.min.js":1,"sites\/all\/modules\/environment_indicator\/environment_indicator.js":1,"sites\/all\/modules\/environment_indicator\/color.js":1,"sites\/all\/modules\/extlink\/extlink.js":1,"sites\/all\/modules\/linkit\/js\/linkit.js":1,"profiles\/uscourts\/modules\/custom\/usc_alert\/js\/usc-alert.js":1,"misc\/progress.js":1,"sites\/all\/modules\/linkit\/editors\/ckeditor\/linkitDialog.js":1,"sites\/all\/modules\/views\/js\/base.js":1,"sites\/all\/modules\/views\/js\/ajax_view.js":1,"sites\/all\/modules\/google_analytics\/googleanalytics.js":1,"profiles\/uscourts\/libraries\/sidr\/jquery.sidr.min.js":1,"sites\/all\/libraries\/superfish\/superfish.js":1,"sites\/all\/modules\/superfish\/superfish.js":1,"profiles\/uscourts\/themes\/usczen\/js\/jquery.tablesorter.min.js":1,"profiles\/uscourts\/themes\/usczen\/js\/responsive_menus_sidr.js":1,"profiles\/uscourts\/themes\/usczen\/js\/script.js":1}},"profile":"uscourts","profile_version":"7.x-1.5","linkit":{"autocompletePath":"http:\/\/www.uscourts.gov\/linkit\/autocomplete\/___profile___?s=","dashboardPath":"\/linkit\/dashboard\/","currentInstance":{}},"urlIsAjaxTrusted":{"\/about-federal-courts\/cameras-courts":true,"\/views\/ajax":true},"views":{"ajax_path":"\/views\/ajax","ajaxViews":{"views_dom_id:0ef7deb6d6359824d5a913096cf414f1":{"view_name":"cameras_in_court_landing","view_display_id":"entityreference_view_widget_1","view_args":"","view_path":"node\/235","view_base_path":"node\/235","view_dom_id":"0ef7deb6d6359824d5a913096cf414f1","pager_element":0}}},"extlink":{"extTarget":0,"extClass":"ext","extLabel":"(link is external)","extImgClass":0,"extSubdomains":1,"extExclude":"\\.gov","extInclude":"","extCssExclude":"#footer-wrapper, #block-usc-settings-share-this, #content-usc-share-this","extCssExplicit":"","extAlert":"_blank","extAlertText":"You are now leaving uscourts.gov. \r\n\r\nThe Administrative Office of the U.S. Courts (AOUSC) provides this link for your convenience. The AOUSC takes no responsibility for and exercises no control over the organizations, views, accuracy, copyright or trademark compliance, or legality of the material contained on the site you are about to access. The content of the site you are about to access does not necessarily reflect the opinions, standards, or policies of the AOUSC. \r\n\r\nThank you for visiting our site.","mailtoClass":"mailto","mailtoLabel":"(link sends e-mail)"},"googleanalytics":{"trackOutbound":1,"trackMailto":1,"trackDownload":1,"trackDownloadExtensions":"7z|aac|arc|arj|asf|asx|avi|bin|csv|doc(x|m)?|dot(x|m)?|exe|flv|gif|gz|gzip|hqx|jar|jpe?g|js|mp(2|3|4|e?g)|mov(ie)?|msi|msp|pdf|phps|png|ppt(x|m)?|pot(x|m)?|pps(x|m)?|ppam|sld(x|m)?|thmx|qtm?|ra(m|r)?|sea|sit|tar|tgz|torrent|txt|wav|wma|wmv|wpd|xls(x|m|b)?|xlt(x|m)|xlam|xml|z|zip"},"superfish":{"1":{"id":"1","sf":{"delay":"0","animation":{"opacity":"show","height":"show"},"speed":"\u0027fast\u0027","autoArrows":false,"dropShadows":false,"disableHI":false},"plugins":{"supposition":false,"bgiframe":false}}}}); 
 if(jQuery.isFunction(jQuery.holdReady)){jQuery.holdReady(false);}} if(window.jQuery && window.Drupal){init_drupal_core_settings();}</script>
   <script type="text/javascript">var _baTheme=0, _baMode='<thsentence rwthpgen="1">Listen to this page</thsentence><i class="fa fa-assistive-listening-systems fa-1x fa-fw"></i>', _baUseCookies=true;</script><noscript>Your browser does not support Javascript.</noscript>
   <script src="https://www.browsealoud.com/plus/scripts/2.5.2/ba.js" crossorigin="anonymous" integrity="sha256-pZUlaM0VaGsi0/tgIHnex2p/USKA0aujxOss+LCcUcU= sha384-SDqKNeiB6jmEcesjpzTEZzJG4Us+zZR2Oimur94XFciwMm7ixVAgd/6D7K4O8BEf sha512-u/Qw2M8T2H7AV8TIvPDTDAnMZ5ouIEF1PB2Prqe34FeaUSyJpd1HBEwE0xFQlIm/B3HuOnRQykVtP7IS+Mm0TQ=="></script><noscript>Your browser does not support Javascript.</noscript>

Ugh. There's clearly more going on there than just the first hunk, which changes the id of a <div> element. It looks like the ordering of 2 cases (Waymo… and Regents…) is flip-flopping. Curiously they are not marked with yellow highlighting. That sounds like...another bug. I guess it's like #34? I'm a little confused how the highlighter can manage to skip these...it would seem like extra effort is required to do so, and so that extra effort could just be turned off (either globally or with a knob).


I think the best solution to this is probably #50. Unfortunately noone's had time to implement this, and it would be a fair bit of work.

Well, I think you mean #9. It does seem like #9 should be pretty easy, just that the UI for it (#50) might be atrocious. Is there an appetite for a solution to 9 with only a bare-bones UI (enter XPath in a text field)?


Let's take another case where this happens...ugh, I gotta run off to a meeting, I'll try to dig up one later. A bunch of the cases I looked for just now seem to have a null "Old HTML" section, which seems wrong. Is this normal behavior?:

screen shot 2018-11-26 at 13 57 05

Also, is there a simple sqlite3 incantation I can run to pull out all the files at once? That would be handy when looking for this sort of thing without having to visually inspect each one by one.

Thanks.

johnhawkinson commented 5 years ago

Here's one where the CSS changed, as did an HTML comment: http://www.cambridgeday.com/2017/03/03/northpoint-affordable-housing-rate-locked-as-council-progresses-toward-20-citywide/

diff:

--- "NorthPoint affordable housing rate locked as council progresses toward 20% citywide _ Cambridge Day-old.html"  2018-11-26 23:44:06.000000000 -0500
+++ "NorthPoint affordable housing rate locked as council progresses toward 20% citywide _ Cambridge Day-new.html"  2018-11-26 23:44:09.000000000 -0500
@@ -827,6 +827,9 @@

            }
 @media (min-width: 450px){
+   .archive-default .entry-title{
+       width:107% !important;
+   }
 .single .col-md-6, .single .col-sm-8, .single .col-xs-12 {
     position: relative;
     min-height: 1px;
@@ -1337,7 +1340,7 @@

 </body>
 </html>
-<!-- Dynamic page generated in 0.468 seconds. -->
-<!-- Cached page generated by WP-Super-Cache on 2018-11-19 23:11:23 -->
+<!-- Dynamic page generated in 0.533 seconds. -->
+<!-- Cached page generated by WP-Super-Cache on 2018-11-26 23:15:46 -->

 <!-- Compression = gzip -->
\ No newline at end of file
johnhawkinson commented 5 years ago

Here's another one, where the value keys on a form element change: https://malegislature.gov/Events/List/10-27-2018/11-30-2018/

--- "Hearings & Events-old.html"    2018-11-27 02:51:22.000000000 -0500
+++ "Hearings & Events-new.html"    2018-11-27 02:51:25.000000000 -0500
@@ -97,7 +97,7 @@
                     </button>
                     <div id="loginFlyout" class="dropdown-menu dropdownLogin" aria-live="assertive">
                         <h2 class="text-center loginTitle">MyLegislature</h2>
-<form action="/Account/Login?isModal=False" class="form-horizontal clearfix" data-ajax="true" data-ajax-begin="inProgress" data-ajax-complete="loginFlyoutFormOnComplete" data-ajax-method="POST" id="loginFlyoutForm" method="post" role="form"><input name="__RequestVerificationToken" type="hidden" value="2G3CteAPnHB5YCKkQNrNxzVpAKDVEzzVrOdNxWPQVKj58gjZJlpKiINx1zJCGwNC2fYgEfzMsgX5W82DzwNXYP5sF581" />
+<form action="/Account/Login?isModal=False" class="form-horizontal clearfix" data-ajax="true" data-ajax-begin="inProgress" data-ajax-complete="loginFlyoutFormOnComplete" data-ajax-method="POST" id="loginFlyoutForm" method="post" role="form"><input name="__RequestVerificationToken" type="hidden" value="Lmc2G2NdOLysKG3l1i9yXc3Pj2-l37SYxW4QnL_wpX4OobRtDOpUzKy7faEbMJfh3Cwdg-Q0uDKdhF02zlGjASx9vg01" />
 <legend class="text-center">Sign in with MyLegislature Account</legend>
 <div class="form-group">
     <label class="col-sm-3 control-label" for="Email">Email</label>
@@ -1674,7 +1674,6 @@
                                 </tr>
                                 <tr>
                                         <td class="text-center" rowspan="2">
-                                                <span class="today"><span class="sr-only">Go to </span>TODAY</span>
                                                                                             <span class="month">NOV</span>
                                                 <span class="day">26</span>
                                                 <span class="month">2018</span>
@@ -1986,7 +1985,7 @@
                     </div>
                     <div class="col-xs-12 col-sm-7 vDivide">
                         <h5 class="text-center">Register With Your Email Address</h5>
-<form action="/Account/Register" data-ajax="true" data-ajax-begin="inProgress" data-ajax-method="POST" data-ajax-success="registrationModalFormOnSuccess" id="registrationModalForm" method="post" role="form"><input name="__RequestVerificationToken" type="hidden" value="T3yFCfhHbs0nMmheDX-m1R_uBUVGKqqHW748rrxJpOHggYu_FRiYVoqgQ0IYCaW_WDSeffBEVWkira-lZl4374oyT0I1" />
+<form action="/Account/Register" data-ajax="true" data-ajax-begin="inProgress" data-ajax-method="POST" data-ajax-success="registrationModalFormOnSuccess" id="registrationModalForm" method="post" role="form"><input name="__RequestVerificationToken" type="hidden" value="NvhKWzCiyQFOPekY3Wtnvehqo3Wsj5B7vLl-tS2Lu7LGad8dQfazNwFcrZHjzoW6irjzQ-sP-vD4Cj8IGbUokRDWNYI1" />

 <div class="form-group">
     <label class="sr-only" for="Email">Email Address</label>
@@ -2039,7 +2038,7 @@
                     </div>
                     <div class="col-xs-12 col-sm-7 vDivide">
                         <h5 class="text-center sr-only">Sign In With Your MyLegislature Account</h5>
-<form action="/Account/Login?isModal=True" data-ajax="true" data-ajax-begin="inProgress" data-ajax-complete="loginModalFormOnComplete" data-ajax-method="POST" id="loginModalForm" method="post" role="form"><input name="__RequestVerificationToken" type="hidden" value="opU5oRrRNf5pD5dvkcTTTQA2ReXsWTHKf_7MzhF2a2oE8KHKROCWFqxg0DwRZpXDgeQh44vFOUzKt4PeIM19CCkvgEY1" />
+<form action="/Account/Login?isModal=True" data-ajax="true" data-ajax-begin="inProgress" data-ajax-complete="loginModalFormOnComplete" data-ajax-method="POST" id="loginModalForm" method="post" role="form"><input name="__RequestVerificationToken" type="hidden" value="aPbYGaXbgDEvrI5bCbb_KGErtxsbzKOs_Yd-VLc6Xep75hbixQjyiHfwCV0ePg03P0QnSIeoqGGyu3qGNHZRTv9Qz6s1" />
 <div class="form-group">
     <label class="sr-only" for="Email">Email</label>
     <div class="input-group">
@@ -2082,7 +2081,7 @@
             </div>
             <div class="modal-body">
 <form action="/Account/ForgotPassword" data-ajax="true" data-ajax-complete="forgotPasswordModalFormOnComplete" data-ajax-method="POST" id="forgotPasswordModalForm" method="post" onsubmit="if($(this).valid()){ $(this).find(&#39;.fnEnd&#39;).show(); $(this).find(&#39;.fnStart&#39;).hide(); }" role="form">    <p><strong>Please enter your email address.</strong> If there is an account associated with that address, a password reset link will be mailed to you.</p>
-<input name="__RequestVerificationToken" type="hidden" value="7Wc7c7OZxlOTpPP1HIAnbYNJzZsB3uLKa1lU2Kpwq8jXJZi0UDLat5hU7nbugFxzHxYvj-Zh7Fx_d5ZivT2VlOGgEBc1" />    <div class="row">
+<input name="__RequestVerificationToken" type="hidden" value="7SOxWUvLpY_AHuU-j3nsaNgTMLu5WH0TREDhCRtmq-tXjg3tAQpkaPHod83jsFfn-lQE2dsgG-jSau5CMuR6u3ACROQ1" />    <div class="row">
         <div class="form-group clearfix">
             <div class="col-sm-12 col-md-9">
                 <label class="sr-only" for="EmailAddress">Email Address</label>

Although, also, <span class="today"><span class="sr-only">Go to </span>TODAY</span> disappears... not sure what's up with that, it doesn't -- oh, OK, I guess it is a deletion, and deletions are not signified in any way visually? Seems like there should be red strikethrough text or something. I guess this is #81?

Huh. It's funny how many different cases all appear under the "no visible changes" rubric....

johnhawkinson commented 5 years ago

I'm not entirely sure where we are with this, but @sneakypete81 do you have answer to this?:

Also, is there a simple sqlite3 incantation I can run to pull out all the files at once? That would be handy when looking for this sort of thing without having to visually inspect each one by one.

Thanks.

sneakypete81 commented 5 years ago

It's stored using the IndexedDB APi. There seem to be some options listed for accessing this at https://stackoverflow.com/questions/9846013/how-to-view-indexeddb-content-in-firefox.

You're right, deleted content will trigger the scanner, but won't appear highlighted.

sneakypete81 commented 5 years ago

@johnhawkinson, I"m struggling a bit to be sure if there's anything here that isn't covered by other issues. You're quite right though, the scanning and highlighting could probably be improved.

I think the following sums up the situation:

The "scanning" operation ought to strip out changes within tags, but it's fairly limited - we've optimised a bit for speed to prevent performance problems when scanning. This means that it will sometimes trigger on non-visible changes, unfortunately.

I'll close this for now, please let me know if I've missed something.