search

dead-hosts / dev-center

This is the place where non-organisation members can ask questions about the organisation workflow or simply, inclusion requests.
MIT License
5 stars 0 forks source link

An option for live update of the input list ? #8

Closed funilrys closed 4 years ago

funilrys commented 4 years ago

As @ryanbr mentioned in https://github.com/easylist/easylist/issues/2374#issuecomment-574587338, we should be more reactive by allowing a live update of the input source.

funilrys commented 4 years ago

For follow-up: cc @jawz101 too :smile_cat:

jawz101 commented 4 years ago

Yeah, I feel like I agree with how that other person assumes PyFunceble would work.

If I'm actively adding and removing entries on a daily, hourly, or by minute basis, if I'd want to redownload the file and scan some new domains one moment while others are no longer needed. And if they're marked inactive, it may be a temporary issue with my ISP, DNS, some intrusion detection server detecting weird traffic or the whois server.

I don't know what live update means because that's how it should work if the intention is to scan lists in repositories under constant change. if the inactive list will always stick then it implies domains never get active again nor can it recover from temporary resolution failures

Similar to @ryanbr I habitually do a PyFunceble clean all before every check. It's not ideal because then I have to reprocess thousands of actives again. If it removed any I removed, added any new entries, scanned that list. And then a second pass of the hosts marked inactive, it might reduce the false positives.

ryanbr commented 4 years ago

I presumed that PyFunceble would parse the list(s), then spit out the results. The question is why would it spit out old domains? Not sure why domains removed many days before hand would show up still, unless I'm missing something

funilrys commented 4 years ago

@jawz101 I agree, I should write a retry option. Somewhere in the future.

@ryanbr It's probably because of the Inactive DB which stores all previously inactive subjects. On retest, if one becomes ACTIVE, it is reintroduced in the output. If it's not needed simply update the info.json file as follow:

[...]
"custom_pyfunceble_config": {
    "inactive_database": false
},
[...]

Closing as it's now deployed as per https://github.com/dead-hosts/infrastructure-launcher/commit/c0b7191c24b3820016de0dfe8226e9151b51b378.