nodiscc / hecat

Generic automation tool around data stored as plaintext YAML files
GNU General Public License v3.0
29 stars 5 forks source link

Improve handling of url_check processor by not failing by a single URL failure #133

Open Rabenherz112 opened 9 months ago

Rabenherz112 commented 9 months ago

Following our brief discussion on this GitHub thread, I've made some enhancements to the existing URL_check. Instead of triggering a failure after one unsuccessful attempt, the revised system now checks if the same URL has failed in the last X runs. Only after that does it flag an error; otherwise, it simply issues a warning.

I didn't want to modify the system so that it would only raise an error after a total of X URLs had failed. Though this would be beneficial for the badges, it wouldn't identify the exact URL that might need attention, particularly if it had been failing consistently for a week, for instance.

In light of these changes, failure records are now kept in a txt file, listing the number of times a URL has failed. This system will only escalate to an error if the failure count exceeds a certain threshold. However, this necessitated the creation of a new file, failed_urls.txt, which will be used to store this data between runs.

https://cdn.ravshort.com/rEGU2/NOnOcUjE47.png https://cdn.ravshort.com/rEGU2/wOWIFuyi58.png https://cdn.ravshort.com/rEGU2/HOWUQeco71.png https://cdn.ravshort.com/rEGU2/sUgoPenu05.png

Rabenherz112 commented 9 months ago

I also went ahead and changed the output slightly to actually show, how often the url failed to make it easier to see if a url is already dead since a longer time (before it would always show the esclation limit). Feel like this makes more sense now. https://cdn.ravshort.com/rEGU2/XiZOLIGI52.png

nodiscc commented 7 months ago

Sorry for the delay. Hopefully I will have time to review/test this in the coming weeks.

Rabenherz112 commented 5 months ago

Todo: [ ] Use certifi package to get rid of currently SSL validation errors (databag) @Rabenherz112