search

sissbruecker / linkding

Self-hosted bookmark manager that is designed be to be minimal, fast, and easy to set up using Docker.
MIT License
5.32k stars 261 forks source link

Internet archive huey task should not keep retrying forever #747

Open fingon opened 3 weeks ago

fingon commented 3 weeks ago

I dumped 11k links to Linkding (with internet archive bookmarking enabled). Almost two weeks later, I noticed it was still at it, and:

mstenber@fw ~>nfdump -R /flows -s srcip/flows dst ip 207.241.237.3
Top 10 Src IP Addr ordered by flows:
Date first seen                 Duration Proto       Src IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2024-05-25 11:11:32.740 12d 19:53:29.659 any        192.168.42.2   727360(100.0)    3.4 M(100.0)  344.7 M(100.0)        3     2487   100

So far 727k connection (attempts - I guess, given it is just few bytes each). The web.archive.org does not seem to be responding due to recent DoSes etc, but the task definitely should have just quit after N failures instead of keeping retrying it forever.

sissbruecker commented 2 weeks ago

Collecting IA snapshots after imports indeed doesn't work very well at the moment, it seems there is a very low rate limit on the API.

However the tasks themselves should stop scheduling after 5 tries with increasing back-off: https://github.com/sissbruecker/linkding/blob/fe40139838e75a162e5066c232f50a3f8e95402b/bookmarks/services/tasks.py#L25-L51

The background_tasks.log file in the container should give more info, it should print how many retries a task still has left.