search

Ultimate-Hosts-Blacklist / dev-center

The place to talk about our infrastructure or everything related to the Ultimate Hosts Blacklist project.
MIT License
11 stars 2 forks source link

[Feedback] New infrastructure #32

Closed funilrys closed 4 years ago

funilrys commented 5 years ago

Hello, World!

Please report here every infrastructure/build/test issues (suspected or confirmed) from now.

Note: This is issue is only about something which is not normal with our new deployement logic. Everything else should be reported as a new issue.

Cheers, Nissar

funilrys commented 5 years ago

@dnmTX:

funilrys commented 5 years ago

You're right those are not filtering but infrastructure issue. Will look into a way to speed up the test overall in our infrastructure along with a way to restart if a new version is present ... From now it's doesn't have anything to do with PyFunceble ... Just the way our infrastructure live and work...

dnmTX commented 5 years ago

You're right those are not filtering but infrastructure issue.

Well,in my defense if the infrastructure is broken anyone would assume that there is something wrong with the filtering but thank's for acknowledging that there is a issue and please,keep me posted this time on what did you fixed.This way i'll monitor more closely and give you confirmation. Now...let's get the ball rolling and fix that damn infrastructure once and for all 😄

funilrys commented 5 years ago

Time increased from 5 to 10 minutes.

funilrys commented 5 years ago

We now test 25 subjects at the same time instead of the overloading "as much as possible".

dnmTX commented 5 years ago

Also ⏰ reminder ⏰ to ADD the new,improved and unified LIST from our friend @smed79 😉 👍 and remove all the rest which contain smed79 in them as they're not needed anymore.

dnmTX commented 5 years ago

....and after...approximately...28 days i was finally able to update all of my lists. I call this progress 😅 Thank you @funilrys ❤️ ❤️ ❤️ 👍 💯

funilrys commented 5 years ago

@dnmTX I lost my automation tool for repository creation (Yeah I made the stupid error to not create a private repo for it :sob:) so everything around repo creation/deletion will have to wait a bit!

dnmTX commented 5 years ago

@funilrys no rush,at least we know that it's in your TO DO LIST and thanks for the update 😉

dnmTX commented 5 years ago

@funilrys looks like the INVALID folder doesn't refresh after each filtering and there are some duplicates. Also when comparing it to the latest downloaded domains.list those (invalid) domains are not even present there anymore:

lightswitch05

Please take a look when you have time. Thanks 👍

funilrys commented 5 years ago

Added to the workflow :+1:

funilrys commented 5 years ago

Hello there,

just a short update, I'm almost done with the repository creation logic/tool 🙂 I'll start with @smed79's list later today (Berlin Time). All other issues which are in my workflow are in my internal queue.

Cheers, Nissar

funilrys commented 5 years ago

Just created @smed79 :+1: https://github.com/Ultimate-Hosts-Blacklist/smed79_blacklist

funilrys commented 5 years ago

Waiting for push confirmation from the test with PyFunceble so I can delete the others :)

funilrys commented 5 years ago

@funilrys looks like the INVALID folder doesn't refresh after each filtering and there are some duplicates. Also when comparing it to the latest downloaded domains.list those (invalid) domains are not even present there anymore:

lightswitch05

Please take a look when you have time. Thanks

@dnmTX About that issue I'll start working on that next. It should not have influences in the outputs as those are re-test outputs but I do understand the confusion it creates.

funilrys commented 5 years ago

@smed79 @dnmTX Do you have a list of smed79_* repository to delete?

dnmTX commented 5 years ago

@smed79 @dnmTX Do you have a list of smed79_* repository to delete?

  1. https://github.com/Ultimate-Hosts-Blacklist/smed79_hilltopads_adservers
  2. https://github.com/Ultimate-Hosts-Blacklist/smed79_admeasures_adservers
  3. https://github.com/Ultimate-Hosts-Blacklist/smed79_getadmiral.com
  4. https://github.com/Ultimate-Hosts-Blacklist/smed79_popads_adservers
  5. https://github.com/Ultimate-Hosts-Blacklist/smed79_propellerads_adservers
  6. https://github.com/Ultimate-Hosts-Blacklist/smed79_parkingcrew.com
  7. https://github.com/Ultimate-Hosts-Blacklist/smed79_uponit.com_adservers
  8. https://github.com/Ultimate-Hosts-Blacklist/smed79_assorted

@smed79 can you confirm the list above please.Thanks....EDIT...Confirmed ☝️

@funilrys thank you for adding the new @smed79's unified lists.Very much appreciated and thanks for fixing the issue with the invalid domains 👍

dnmTX commented 5 years ago

❗️ ❗️ ❗️ @funilrys there is something wrong with the infrastructure again. Some builds showing error,on others the lists are not updating,even though it shows that it passed. Can you please take a look.Thanks. ❗️ ❗️ ❗️

smed79 commented 5 years ago

@funilrys

Do you have a list of smed79_* repository to delete?

keep only :

fyi, in the day that follows i am planning to upload a nsfw list.

funilrys commented 5 years ago

@smed79 ==> :heavy_check_mark: + :checkered_flag:(it's a link :wink:)

@dnmTX Was a Travis CI. I'm planning a full restart this night (Berlin Time).

dnmTX commented 5 years ago

@funilrys still not fixed. Please,when you have time do a deeper assessment and hopefully get it fixed. Thanks a bunch 👍

dnmTX commented 5 years ago

@funilrys where are all those INVALID DOMAINS coming from if they don't even exist in the same form in the original lists? This is definitely some parsing bug that needs to be looked at.

tracking-extended

P.S. Also reminder about the duplicate entries bug,still present.

funilrys commented 5 years ago

@dnmTX They come from the inactive database aka inactive-db.json. I fixed that on the mainstream.

Indeed, from now, if a subject is listed into the inactive database and the status is still the same (also INACTIVE or INVALID) it's not included in the results.

For our use case (our infrastructure), https://github.com/funilrys/PyFunceble/commit/670921776524ccf74a4d53d6ffa3d6a4a74f4447 make the mainstream patch available because we use the API here and not the CLI.

funilrys commented 5 years ago

Duplicated entries ==> in progress

dnmTX commented 5 years ago

@funilrys some builds are failing: https://github.com/Ultimate-Hosts-Blacklist/ZeroDot1_CoinBlockerLists https://github.com/Ultimate-Hosts-Blacklist/anudeepND_blacklist_ad_server And this one is not updating the lists: https://github.com/Ultimate-Hosts-Blacklist/quidsup_malicious-sites

Please take a look when you have time. Thank you 👍

dnmTX commented 5 years ago

@funilrys as of late builds are passing but none of the lists got updated,last update on most of them is showing 8 days ago.Hopefully you'll have time to fix it.Thank you 👍

funilrys commented 5 years ago

Hi @dnmTX, I hope that everything is OK on your side!

Since last weekend we released our monitoring tool which I wrote for the past months at hosts . Ubuntu 101 . Co . za / app / infra (remove spaces)

I still have to adjust it but it's correct. Indeed, as we launch test more often, sometime the results won't change. Which come to what you're discribing ☺️

Have a nice day!

Cheers Nissar

dnmTX commented 5 years ago

Indeed, as we launch test more often, sometime the results won't change. Which come to what you're discribing

@funilrys you know,i've been monitoring certain lists for a long time now and never before i have seen none of the lists to not be updated for a period of 12-14 days. So far,every time when test is launched the results are different(no exception).This is not normal,and needs to be looked at. Also it's not normal eight out of eight lists ,on all of them,the volatile.list etc. not to get updated even once for a period of two weeks.

P.S. Results of the test are always different because all those lists that i'm monitoring are very much active and on some,changes are made every day(not even weekly). Just to prove my point here and push you to find the problem 😉

funilrys commented 4 years ago

@dnmTX, you were (one again) right. After testing, and reading the last 10 days of logs of our infrastructure launcher running in my raspberry pi, I finally found it.

The patch (PyFunceble) is on its way. I created a git-hook on my machine to check that it still works before committing.

Thanks again for everything!

funilrys commented 4 years ago

Infrastructure restart (from the beginning of each list) ordered. Should start in the next minutes/hour.

Time to go to bed. Catch up later!

funilrys commented 4 years ago

Closing, Please report to the weekly issues. Thanks again for everything!