search

Ultimate-Hosts-Blacklist / dev-center

The place to talk about our infrastructure or everything related to the Ultimate Hosts Blacklist project.
MIT License
11 stars 2 forks source link

lists with outdated URL's... #7

Closed dnmTX closed 5 years ago

dnmTX commented 6 years ago

@funilrys there are three lists so far that i've encounter which moved their repos to GitLab, very much active with at least one weekly update and here and in dead-hosts you still using the old URL's from GitHub which are outdated to say the least.If you don't mind look in to it.I'd rather download those lists from here but knowing that they're outdated is a big NO NO for me:

https://github.com/Ultimate-Hosts-Blacklist/quidsup_notrack_trackers https://github.com/Ultimate-Hosts-Blacklist/quidsup_malicious-sites New URL: https://gitlab.com/quidsup/notrack-blocklists

https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists New URL: https://gitlab.com/ZeroDot1/CoinBlockerLists

Thank you.

P.S. Another thought...on active lists like the one mentioned above or any other the domain.list needs to be updated more often,like at least once weekly(don't want to push it but twice weekly sounds even better 🙂)

funilrys commented 6 years ago

Hi @dnmTX thanks for reporting will update the links asap.

About the domain.list, if the upstream link is update it is also updated...

Have a nice day/night.

dnmTX commented 6 years ago

About the domain.list, if the upstream link is update it is also updated...

@funilrys that is a nice feature,let's hope it's working as intended.I'll be checking on it to make sure it does once you update the URL's.Thank you.

P.S. For CoinBlockerLists please add HOSTS and HOSTS_BROWSER Thank you !!!!

funilrys commented 6 years ago

Hi @dnmTX, As referenced, this issue is partially fixed.

dnmTX commented 6 years ago

Thank you @funilrys .You know....on a second thought(as previously discussed with @ZeroDot1) many of the domains on his lists are going offline for a short time and then coming back to life(i guess some tactic of a sort).So i'd say you leave the dead hosts filtering to him,He does check and removes really dead ones but he checks them for a longer time before taking any decisions. On the other hand(based so far on my observations),whoever maintains the quidsup lists he hardly ever removes any,just adding more and more.And considering how big the lists is i can't believe that they're all active,so those two lists need filtering for sure.

funilrys commented 6 years ago

Hi @dnmTX,

If we match them as INACTIVE we move them into inactive_db.json and after x days (defined by info.json but by default 1 day), we retest them and if they became ACTIVE, we include them into the list of ACTIVE which actually means clean.list.

mitchellkrogza commented 6 years ago

I think we need some logic in PyFunceble that if a domain was inactive once and became active again we mark it as suspicious and keep it on active?

dnmTX commented 6 years ago

@funilrys i think the clean.list in quidsup_malicious-sites is not updating. Any info on when this commit will be merged or it's still in test? I still see the GitHub's URL's on both of them.

funilrys commented 6 years ago

Hi @dnmTX, if there is no changes in the tests it's normal that it is not updating.

funilrys commented 5 years ago

https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_optional and https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_browser created.

Closing.

dnmTX commented 5 years ago

@funilrys consider changing CoinBlockerLists_optional with hosts. I think the optional one is the one that is more aggressive and have more false positives.The hosts one covers wider area and it gives you better protection then the browser one in general(i personally using it on my end).

P.S. You told @quidsup that you are informed of the changes but here the URL's are still from GitHub.What's up with that?

funilrys commented 5 years ago

In Detail @dnmTX.

Please report to

The system will do its work automatically and correctly once the repository will have an execution time from Travis CI.

Repository Next session in
https://github.com/Ultimate-Hosts-Blacklist/quidsup_notrack_trackers 7h
https://github.com/Ultimate-Hosts-Blacklist/quidsup_malicious-sites 7h

I do not understand what you say.

Repo Tested
https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists https://zerodot1.gitlab.io/CoinBlockerLists/list.txt
https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_browser https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt
https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists_optional https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_optional.txt
funilrys commented 5 years ago

Also, our whole organization (Ultimate-Hosts-Blacklist) - except for special repositories - has for purpose to test input source and provide the clean.list. The whole process is done automatically.

Understand that this organization is only the backend of the main repository mitchellkrogza/Ultimate.Hosts.Blacklist.

Everyday at T14:06:39Z the generation of the unified hosts file is started. What it does it getting the list of all repository of this organization from the GitHub API then get clean.list or domains.list if clean.list is not found.

As we need many IP to have consistent test for long list, we choose by design to use Travis CI which is great as it's offer a free usage for open repository. To understand why the repository are not updated right in the way, you have to understand that Travis CI only allow us to run 5 instances at the same time.

We are aware that our system provide some stress to the Travis CI team and infrastructure as our system is maybe one of those rare who run almost 24/24/7 that's why we accept those 5 instances limitation.

In order to allow bigger list to run stright away when they are under test, we implemented a code side "switch". Indeed, as some upstream may not be update their list every day we also have the days_until_next_test in order to stop the system right before it test if the number of day < days_until_next_test.

Therefor if you find that days_until_next_test is not realistic for a repository let me know, I will check and update if needed.

I hope that this clarify what was misunderstood (if something was actually misunderstood).

Cheers, Nissar

dnmTX commented 5 years ago

@funilrys this is what misled me: test

Makes the impression that still the old URL's are in use.Obviously my mistake,sorry for the confusion,basically disregard my last post.Thank you for that in detail explanation. Cheers!

funilrys commented 5 years ago

Fixed.