thp / urlwatch

Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
https://thp.io/2008/urlwatch/
Other
2.81k stars 352 forks source link

Implements command to change a job location. #739

Closed trevorshannon closed 1 year ago

trevorshannon commented 1 year ago

This PR adds a new command line option --change-location to change the location (i.e. url, command, or navigate) of a job while preserving history for that job. The tool ensures that the new location string is unique within the list of jobs prior to changing anything.

Note that in the (admittedly contrived) scenario of having abandoned entries in a redis db that use the user-provided new location (perhaps from old runs that were never garbage-collected) then those abandoned entries will be clobbered by the --change-location option. I think that should be ok, but I'm sure I could find a way to preserve those entries if needed.

I did some manual testing to verify functionality.

Save the starting version of a simple urls YAML:

➜  urlwatch git:(feat/update-url)  cp ~/urls.yaml ~/urls_init.yaml
➜  urlwatch git:(feat/update-url) cat ~/urls_init.yaml 
command: date
kind: shell
---
kind: url
url: https://www.example.org
---
kind: url
url: https://www.wikipedia.org
user_visible_url: https://en.wikipedia.org
---
kind: browser
navigate: https://www.google.com

Change location of all 4 jobs:

➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location 1 "date -v1m"                                
Moving location of <shell command='date'> to "date -v1m"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location https://www.example.org https://www.example.net
Moving location of <url url='https://www.example.org'> to "https://www.example.net"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location 3 https://es.wikipedia.org                     
Moving location of <url url='https://www.wikipedia.org' user_visible_url='https://en.wikipedia.org'> to "https://es.wikipedia.org"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location 4 https://www.google.co.uk
Moving location of <browser navigate='https://www.google.com'> to "https://www.google.co.uk"
Saving updated list to '/Users/trevorshannon/urls.yaml'

Compare the updated urls YAML to the initial reference version. Appropriate key values are updated according to job type:

➜  urlwatch git:(feat/update-url)  diff ~/urls_init.yaml ~/urls.yaml   
1c1
< command: date
---
> command: date -v1m
5c5
< url: https://www.example.org
---
> url: https://www.example.net
9c9
< user_visible_url: https://en.wikipedia.org
---
> user_visible_url: https://es.wikipedia.org
12c12
< navigate: https://www.google.com
---
> navigate: https://www.google.co.uk

Existing history is moved with --change-location:

➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml --dump-history 1
==============================
2022-12-20 13:08
------------------------------
Tue Dec 20 13:08:59 EST 2022

============================== 

==============================
2022-12-20 13:18
------------------------------
Tue Dec 20 13:18:54 EST 2022

============================== 

2 historic snapshot(s) available
➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml --change-location date "date -v1m"
Moving location of <shell command='date'> to "date -v1m"
Moved 2 snapshots of "date" to "date -v1m"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml                                   
===========================================================================
01. CHANGED: date -v1m
===========================================================================

---------------------------------------------------------------------------
CHANGED: date -v1m
---------------------------------------------------------------------------
--- @   Tue, 20 Dec 2022 13:18:54 -0500
+++ @   Tue, 20 Dec 2022 13:20:33 -0500
@@ -1 +1 @@
-Tue Dec 20 13:18:54 EST 2022
+Thu Jan 20 13:20:33 EST 2022
---------------------------------------------------------------------------

➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml --dump-history 1                  
==============================
2022-12-20 13:08
------------------------------
Tue Dec 20 13:08:59 EST 2022

============================== 

==============================
2022-12-20 13:18
------------------------------
Tue Dec 20 13:18:54 EST 2022

============================== 

==============================
2022-12-20 13:20
------------------------------
Thu Jan 20 13:20:33 EST 2022

============================== 

3 historic snapshot(s) available

I confirmed the same behavior with --cache redis://127.0.0.1:6379

Fixes #300

trevorshannon commented 1 year ago

Updated per our discussion. I ran the same manual tests documented in the initial PR description, plus verified that no history gets moved if a job has user_visible_url set when using --change-location (because the guid does not change in that case)

The thing to note below is the absence of a string like "Moved {n} snapshots of...":

➜  urlwatch git:(feat/update-url) ✗ urlwatch --urls ~/urls.yaml --change-location 2 https://www.wikipedia.org
Moving location of <url url='https://www.example.org' user_visible_url='https://www.google.com'> to "https://www.wikipedia.org"
Saving updated list to '/home/trevor/urls.yaml'
trevorshannon commented 1 year ago

Ok, all set. Thanks for the feedback!

thp commented 1 year ago

Merged, thanks!