search

nanos / FediFetcher

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.
https://blog.thms.uk/fedifetcher?utm_source=github
MIT License
293 stars 213 forks source link

Add `--log-level` option #114

Closed AndrewKvalheim closed 2 months ago

AndrewKvalheim commented 2 months ago

Currently FediFetcher’s logs contain a large number of messages about individual posts:

2024-04-21 10:45:04.043004 PDT: Starting FediFetcher
2024-04-21 10:45:04.169674 PDT: Found 0 reply toots
2024-04-21 10:45:04.169698 PDT: Found 0 known context toots
2024-04-21 10:45:04.169715 PDT: Added 0 new context toots (with 0 failures)
2024-04-21 10:45:08.349776 PDT: Found 218 toots in timeline
2024-04-21 10:45:08.838404 PDT: Got context for toot …
2024-04-21 10:45:09.302492 PDT: Got context for toot …
2024-04-21 10:45:09.816445 PDT: Got context for toot …
2024-04-21 10:45:10.026662 PDT: Got context for toot …
2024-04-21 10:45:10.216191 PDT: Got context for toot …
2024-04-21 10:45:10.540154 PDT: Got context for toot …
2024-04-21 10:45:10.873540 PDT: Got context for toot …
2024-04-21 10:45:11.542339 PDT: Got context for toot …
2024-04-21 10:45:11.974372 PDT: Got context for toot …
2024-04-21 10:45:12.128851 PDT: Got context for toot …
2024-04-21 10:45:12.331673 PDT: Got context for toot …
2024-04-21 10:45:12.463016 PDT: Got context for toot …
2024-04-21 10:45:12.675074 PDT: Got context for toot …
2024-04-21 10:45:13.366633 PDT: Got context for toot …
2024-04-21 10:45:13.901555 PDT: Got context for toot …
2024-04-21 10:45:14.147599 PDT: Got context for toot …
2024-04-21 10:45:14.777664 PDT: Got context for toot …
2024-04-21 10:45:15.415173 PDT: Got context for toot …
2024-04-21 10:45:16.103643 PDT: Got context for toot …
2024-04-21 10:45:16.766342 PDT: Got context for toot …
2024-04-21 10:45:16.949118 PDT: Got context for toot …
2024-04-21 10:45:17.153880 PDT: Got context for toot …
2024-04-21 10:45:17.759796 PDT: Got context for toot …
2024-04-21 10:45:18.336759 PDT: Got context for toot …
2024-04-21 10:45:18.549169 PDT: Got context for toot …
2024-04-21 10:45:18.706847 PDT: Got context for toot …
2024-04-21 10:45:19.274472 PDT: Got context for toot …
2024-04-21 10:45:20.087411 PDT: Got context for toot …
2024-04-21 10:45:20.293435 PDT: Got context for toot …
2024-04-21 10:45:20.552479 PDT: Got context for toot …
2024-04-21 10:45:20.791660 PDT: Got context for toot …
2024-04-21 10:45:22.051707 PDT: Got context for toot …
2024-04-21 10:45:22.890829 PDT: Got context for toot …
2024-04-21 10:45:23.270137 PDT: Got context for toot …
2024-04-21 10:45:23.591277 PDT: Got context for toot …
2024-04-21 10:45:24.366909 PDT: Got context for toot …
2024-04-21 10:45:24.930916 PDT: Got context for toot …
2024-04-21 10:45:26.002568 PDT: Got context for toot …
2024-04-21 10:45:26.558285 PDT: Got context for toot …
2024-04-21 10:45:27.120180 PDT: Got context for toot …
2024-04-21 10:45:27.322184 PDT: Got context for toot …
2024-04-21 10:45:27.890536 PDT: Got context for toot …
2024-04-21 10:45:28.147353 PDT: Got context for toot …
2024-04-21 10:45:28.320695 PDT: Got context for toot …
2024-04-21 10:45:29.817159 PDT: Got context for toot …
2024-04-21 10:45:30.184969 PDT: Got context for toot …
2024-04-21 10:45:30.663536 PDT: Got context for toot …
2024-04-21 10:45:31.677424 PDT: Got context for toot …
2024-04-21 10:45:31.974459 PDT: Got context for toot …
2024-04-21 10:45:33.005559 PDT: Got context for toot …
2024-04-21 10:45:33.411543 PDT: Got context for toot …
2024-04-21 10:45:33.413521 PDT: Error parsing toot URL …
2024-04-21 10:45:33.619264 PDT: Got context for toot …
2024-04-21 10:45:33.887026 PDT: Got context for toot …
2024-04-21 10:45:34.957243 PDT: Got context for toot …
2024-04-21 10:45:35.594570 PDT: Got context for toot …
2024-04-21 10:45:35.895323 PDT: Got context for toot …
2024-04-21 10:45:37.505427 PDT: Got context for toot …
2024-04-21 10:45:38.110339 PDT: Got context for toot …
2024-04-21 10:45:38.331957 PDT: Got context for toot …
2024-04-21 10:45:39.391917 PDT: Got context for toot …
2024-04-21 10:45:40.240024 PDT: Got context for toot …
2024-04-21 10:45:40.580148 PDT: Got context for toot …
2024-04-21 10:45:40.838929 PDT: Got context for toot …
2024-04-21 10:45:41.672060 PDT: Got context for toot …
2024-04-21 10:45:42.051274 PDT: Got context for toot …
2024-04-21 10:45:42.461997 PDT: Got context for toot …
2024-04-21 10:45:43.006871 PDT: Got context for toot …
2024-04-21 10:45:43.407793 PDT: Got context for toot …
2024-04-21 10:45:43.735261 PDT: Got context for toot …
2024-04-21 10:45:44.411488 PDT: Got context for toot …
2024-04-21 10:45:45.291763 PDT: Got context for toot …
2024-04-21 10:45:45.680694 PDT: Got context for toot …
2024-04-21 10:45:45.941765 PDT: Got context for toot …
2024-04-21 10:45:46.628374 PDT: Got context for toot …
2024-04-21 10:45:46.631161 PDT: Error parsing toot URL …
2024-04-21 10:45:47.701375 PDT: Got context for toot …
2024-04-21 10:45:48.520557 PDT: Got context for toot …
2024-04-21 10:45:49.312155 PDT: Got context for toot …
2024-04-21 10:45:49.776843 PDT: Got context for toot …
2024-04-21 10:45:50.098241 PDT: Got context for toot …
2024-04-21 10:45:50.423283 PDT: Got context for toot …
2024-04-21 10:45:51.469729 PDT: Got context for toot …
2024-04-21 10:45:51.780438 PDT: Got context for toot …
2024-04-21 10:45:52.155150 PDT: Got context for toot …
2024-04-21 10:45:52.559668 PDT: Got context for toot …
2024-04-21 10:45:53.375417 PDT: Got context for toot …
2024-04-21 10:45:54.461133 PDT: Got context for toot …
2024-04-21 10:45:55.392555 PDT: Got context for toot …
2024-04-21 10:45:56.162379 PDT: Got context for toot …
2024-04-21 10:45:56.443829 PDT: Got context for toot …
2024-04-21 10:45:56.688878 PDT: Got context for toot …
2024-04-21 10:45:56.907594 PDT: Got context for toot …
2024-04-21 10:45:57.129044 PDT: Got context for toot …
2024-04-21 10:45:57.319190 PDT: Got context for toot …
2024-04-21 10:45:57.506468 PDT: Got context for toot …
2024-04-21 10:45:57.710811 PDT: Got context for toot …
2024-04-21 10:45:57.895512 PDT: Got context for toot …
2024-04-21 10:45:58.082583 PDT: Got context for toot …
2024-04-21 10:45:58.273741 PDT: Got context for toot …
2024-04-21 10:45:58.971626 PDT: Got context for toot …
2024-04-21 10:45:59.277652 PDT: Got context for toot …
2024-04-21 10:45:59.861675 PDT: Got context for toot …
2024-04-21 10:46:00.425878 PDT: Got context for toot …
2024-04-21 10:46:00.489462 PDT: Error getting context for toot …
2024-04-21 10:46:00.705513 PDT: Got context for toot …
2024-04-21 10:46:01.339949 PDT: Got context for toot …
2024-04-21 10:46:01.586670 PDT: Got context for toot …
2024-04-21 10:46:01.807558 PDT: Got context for toot …
2024-04-21 10:46:02.006457 PDT: Got context for toot …
2024-04-21 10:46:02.198809 PDT: Got context for toot …
2024-04-21 10:46:02.396693 PDT: Got context for toot …
2024-04-21 10:46:02.585108 PDT: Got context for toot …
2024-04-21 10:46:02.864802 PDT: Got context for toot …
2024-04-21 10:46:03.052352 PDT: Got context for toot …
2024-04-21 10:46:03.237646 PDT: Got context for toot …
2024-04-21 10:46:03.985879 PDT: Got context for toot …
2024-04-21 10:46:04.176857 PDT: Got context for toot …
2024-04-21 10:46:04.366629 PDT: Got context for toot …
2024-04-21 10:46:04.555056 PDT: Got context for toot …
2024-04-21 10:46:04.744609 PDT: Got context for toot …
2024-04-21 10:46:04.966088 PDT: Got context for toot …
2024-04-21 10:46:05.147949 PDT: Got context for toot …
2024-04-21 10:46:05.516094 PDT: Got context for toot …
2024-04-21 10:46:05.705211 PDT: Got context for toot …
2024-04-21 10:46:05.888799 PDT: Got context for toot …
2024-04-21 10:46:06.095447 PDT: Got context for toot …
2024-04-21 10:46:06.280377 PDT: Got context for toot …
2024-04-21 10:46:06.667489 PDT: Got context for toot …
2024-04-21 10:46:06.851682 PDT: Got context for toot …
2024-04-21 10:46:07.041172 PDT: Got context for toot …
2024-04-21 10:46:07.416913 PDT: Got context for toot …
2024-04-21 10:46:08.013455 PDT: Got context for toot …
2024-04-21 10:46:08.321988 PDT: Got context for toot …
2024-04-21 10:46:10.364321 PDT: Got context for toot …
2024-04-21 10:46:10.697347 PDT: Got context for toot …
2024-04-21 10:46:11.013804 PDT: Got context for toot …
2024-04-21 10:46:11.391659 PDT: Got context for toot …
2024-04-21 10:46:11.777062 PDT: Got context for toot …
2024-04-21 10:46:12.098718 PDT: Got context for toot …
2024-04-21 10:46:12.792761 PDT: Got context for toot …
2024-04-21 10:46:13.126911 PDT: Got context for toot …
2024-04-21 10:46:13.501469 PDT: Got context for toot …
2024-04-21 10:46:14.221681 PDT: Got context for toot …
2024-04-21 10:46:14.858216 PDT: Got context for toot …
2024-04-21 10:46:15.283389 PDT: Got context for toot …
2024-04-21 10:46:17.010467 PDT: Got context for toot …
2024-04-21 10:46:17.198706 PDT: Got context for toot …
2024-04-21 10:46:18.571511 PDT: Got context for toot …
2024-04-21 10:46:18.773780 PDT: Got context for toot …
2024-04-21 10:46:19.350557 PDT: Got context for toot …
2024-04-21 10:46:19.923163 PDT: Got context for toot …
2024-04-21 10:46:20.582217 PDT: Got context for toot …
2024-04-21 10:46:20.847094 PDT: Got context for toot …
2024-04-21 10:46:21.969644 PDT: Got context for toot …
2024-04-21 10:46:22.740101 PDT: Got context for toot …
2024-04-21 10:46:23.631957 PDT: Got context for toot …
2024-04-21 10:46:24.356503 PDT: Got context for toot …
2024-04-21 10:46:24.653007 PDT: Got context for toot …
2024-04-21 10:46:24.926083 PDT: Got context for toot …
2024-04-21 10:46:25.538439 PDT: Discovered redirect for URL …
2024-04-21 10:46:26.823018 PDT: Got context for toot …
2024-04-21 10:46:27.270337 PDT: Got context for toot …
2024-04-21 10:46:28.398472 PDT: Got context for toot …
2024-04-21 10:46:30.493373 PDT: Got context for toot …
2024-04-21 10:46:31.280746 PDT: Got context for toot …
2024-04-21 10:46:32.517111 PDT: Got context for toot …
2024-04-21 10:46:32.728671 PDT: Got context for toot …
2024-04-21 10:46:33.448606 PDT: Got context for toot …
2024-04-21 10:46:34.123776 PDT: Got context for toot …
2024-04-21 10:46:34.382367 PDT: Got context for toot …
2024-04-21 10:46:34.968021 PDT: Got context for toot …
2024-04-21 10:46:35.520102 PDT: Got context for toot …
2024-04-21 10:46:35.585820 PDT: Error getting context for toot …
2024-04-21 10:46:36.194963 PDT: Got context for toot …
2024-04-21 10:46:36.254862 PDT: Error getting context for toot …
2024-04-21 10:46:36.588631 PDT: Got context for toot …
2024-04-21 10:46:36.772678 PDT: Got context for toot …
2024-04-21 10:46:37.797931 PDT: Got context for toot …
2024-04-21 10:46:37.999935 PDT: Got context for toot …
2024-04-21 10:46:38.568455 PDT: Got context for toot …
2024-04-21 10:46:38.892471 PDT: Got context for toot …
2024-04-21 10:46:39.212644 PDT: Got context for toot …
2024-04-21 10:46:39.554219 PDT: Got context for toot …
2024-04-21 10:46:39.614860 PDT: Error getting context for toot …
2024-04-21 10:46:39.866349 PDT: Got context for toot …
2024-04-21 10:46:40.664193 PDT: Got context for toot …
2024-04-21 10:46:41.386060 PDT: Got context for toot …
2024-04-21 10:46:41.996925 PDT: Got context for toot …
2024-04-21 10:46:42.283643 PDT: Got context for toot …
2024-04-21 10:46:43.006629 PDT: Got context for toot …
2024-04-21 10:46:43.557218 PDT: Got context for toot …
2024-04-21 10:46:44.177052 PDT: Got context for toot …
2024-04-21 10:46:44.673368 PDT: Got context for toot …
2024-04-21 10:46:44.900152 PDT: Got context for toot …
2024-04-21 10:46:45.648510 PDT: Got context for toot …
2024-04-21 10:46:46.820701 PDT: Got context for toot …
2024-04-21 10:46:47.277455 PDT: Got context for toot …
2024-04-21 10:46:47.671070 PDT: Got context for toot …
2024-04-21 10:46:48.436375 PDT: Got context for toot …
2024-04-21 10:46:48.981045 PDT: Got context for toot …
2024-04-21 10:46:49.468919 PDT: Got context for toot …
2024-04-21 10:46:49.837549 PDT: Got context for toot …
2024-04-21 10:46:50.188840 PDT: Got context for toot …
2024-04-21 10:46:50.656795 PDT: Got context for toot …
2024-04-21 10:46:51.227811 PDT: Got context for toot …
2024-04-21 10:46:51.916906 PDT: Got context for toot …
2024-04-21 10:46:52.468692 PDT: Got context for toot …
2024-04-21 10:46:53.105618 PDT: Got context for toot …
2024-04-21 10:46:53.433492 PDT: Got context for toot …
2024-04-21 10:46:54.123328 PDT: Got context for toot …
2024-04-21 10:46:54.880588 PDT: Got context for toot …
2024-04-21 10:46:55.510339 PDT: Got context for toot …
2024-04-21 10:46:56.730946 PDT: Got context for toot …
2024-04-21 10:46:56.886459 PDT: Got context for toot …
2024-04-21 10:46:57.280668 PDT: Got context for toot …
2024-04-21 10:46:58.638576 PDT: Got context for toot …
2024-04-21 10:46:58.825365 PDT: Got context for toot …
2024-04-21 10:46:59.574484 PDT: Got context for toot …
2024-04-21 10:46:59.577566 PDT: Found 1541 known context toots
2024-04-21 10:47:00.488434 PDT: Added context url …
2024-04-21 10:47:07.212501 PDT: Added context url …
2024-04-21 10:47:12.448783 PDT: Added context url …
2024-04-21 10:47:13.380911 PDT: Added context url …
2024-04-21 10:47:17.794750 PDT: Added context url …
2024-04-21 10:47:19.536675 PDT: Error adding url …
2024-04-21 10:47:21.282656 PDT: Added context url …
2024-04-21 10:47:22.457389 PDT: Added context url …
2024-04-21 10:47:23.596724 PDT: Added context url …
2024-04-21 10:47:25.051182 PDT: Added context url …
2024-04-21 10:47:29.453074 PDT: Added context url …
2024-04-21 10:47:33.632009 PDT: Added context url …
2024-04-21 10:47:35.520750 PDT: Added context url …
2024-04-21 10:47:36.402554 PDT: Added context url …
2024-04-21 10:47:37.686740 PDT: Added context url …
2024-04-21 10:47:38.147768 PDT: Added context url …
2024-04-21 10:47:39.774338 PDT: Added context url …
2024-04-21 10:47:40.294216 PDT: Added context url …
2024-04-21 10:47:40.801694 PDT: Added context url …
2024-04-21 10:47:42.520831 PDT: Added context url …
2024-04-21 10:47:43.750241 PDT: Added context url …
2024-04-21 10:47:45.471941 PDT: Added context url …
2024-04-21 10:47:46.824697 PDT: Added context url …
2024-04-21 10:47:51.121380 PDT: Added context url …
2024-04-21 10:47:52.863799 PDT: Added context url …
2024-04-21 10:47:54.204816 PDT: Added context url …
2024-04-21 10:47:54.676298 PDT: Added context url …
2024-04-21 10:47:54.677030 PDT: Added 26 new context toots (with 1 failures)
2024-04-21 10:47:54.688584 PDT: Processing finished in 0:02:50.645572.

I’d like to be able to filter it down to just high level reports and notices:

2024-04-21 10:45:04.043004 PDT: Starting FediFetcher
2024-04-21 10:45:04.169674 PDT: Found 0 reply toots
2024-04-21 10:45:04.169698 PDT: Found 0 known context toots
2024-04-21 10:45:04.169715 PDT: Added 0 new context toots (with 0 failures)
2024-04-21 10:45:08.349776 PDT: Found 218 toots in timeline
2024-04-21 10:45:33.413521 PDT: Error parsing toot URL …
2024-04-21 10:45:46.631161 PDT: Error parsing toot URL …
2024-04-21 10:46:00.489462 PDT: Error getting context for toot …
2024-04-21 10:46:35.585820 PDT: Error getting context for toot …
2024-04-21 10:46:36.254862 PDT: Error getting context for toot …
2024-04-21 10:46:39.614860 PDT: Error getting context for toot …
2024-04-21 10:46:59.577566 PDT: Found 1541 known context toots
2024-04-21 10:47:19.536675 PDT: Error adding url …
2024-04-21 10:47:54.677030 PDT: Added 26 new context toots (with 1 failures)
2024-04-21 10:47:54.688584 PDT: Processing finished in 0:02:50.645572.

Changes proposed:

nanos commented 2 months ago

That's fantastic. Thank you!

nanos commented 2 months ago

One thing I notice is that I get lots of lines like these in my logs now:

2024-04-22 18:05:02.084 UTC: Starting new HTTPS connection (1): hc-ping.com:443

One for each HTTPS connections. Do you have any thoughts on that @AndrewKvalheim ?

AndrewKvalheim commented 2 months ago

Sorry, somehow I never did a full run in DEBUG. This explains what’s happening:

If you are using the standard library logging module urllib3 will emit several logs. In some cases this can be undesirable. You can use the standard logger interface to change the log level for urllib3's logger:

logging.getLogger("urllib3").setLevel(logging.WARNING)

https://github.com/AndrewKvalheim/FediFetcher/commit/1fed215977fdb20aa9356fbd3e3f0861d16c1aa0 addresses it. I can send that as a PR if that’s helpful.

nanos commented 2 months ago

Great. Thanks for the very quick answer!

Would you mind sending that as PR, please?

Thanks again!