State of the testing infrastructure of the Ultimate-Hosts-Blacklist project

[funilrys]

Hello, World!

As you may have noticed, for a few weeks, our testing infrastructure is not delivering any result.

I faired that day, but what happened is that Travis CI is limiting their services - even to OSS projects and repositories. From our calculation, with the actual situation, Travis CI allows @ultimate-hosts-blacklist (the organization) to run for around 16 hours (cumulated) in a month. It is disturbing as they are not saying that they don't want any OSS project on their platform.

In the last weeks, I took contact with them and asked them what does their credit system means and why do public repositories consume "credits" and not "OSS credits".

7 days ago, I finally got the following response from them:

Hello,

Thanks for writing in and sorry for the delayed response.

I'm afraid we don't have new directives yet regarding the issuance of OSS credits to customers, both users and organizations.

The credits consumed for your public repos are meant for OSS builds only. For private builds, credits need to be purchased separately. I hope this clarifies.

Thank you

Therefore, we (@mitchellkrogza and I) chose to abandon them. We are still looking for a suitable and probably better solution for the future, but right we are really limited. In the meantime, it means that we have to start switching to GitHub Actions. In the hope that GitHub won't cut us out of their service that quickly, the switch to GitHub Actions should be effective in the coming weeks.
From a personal point of view, the switch of the @dead-hosts project to GitHub Actions is almost done and I will start the switch of @ultimate-hosts-blacklist as soon as possible. If you can help, please let me know!

I would hereby like to renew my thank you to @smed79, @dnmTX, @Somebodyisnobody and others I forget for everything in the last weeks. I still have a lot of feeds to read, but I'm grateful to have you around. ❤️ Please remember that everything thing I built or build for the @ultimate-hosts-blacklist project is done in my free time. So please be patient.

Happy New Year 2021! Stay safe and healthy.

P.S.: Feel free to contact me personally if you can help with any of our issues regarding the testing infrastructure.

[funilrys]

Consider this done, the tests are running again under GitHub Actions. I'm still going to update the launcher in the next hours but consider this as done, stable enough, and running.

As of this morning (Berlin Time), I uninstalled Travis CI from the @ultimate-hosts-blacklist GitHub organization. Meaning that we are officially unplugged from them.

From now on, each input source will automatically try to restart every 12 hours. That means that at those times

• 00:00 UTC
• 12:00 UTC

Input sources will try to start a test. They get the authorization to start the test if the last time of the test was X days ago. Where X is the number defined as days_until_next_test in all info.json file. Meaning that if days_until_next_test is set to 1 and the last time we tested was not 24 * 1 hours ago, the launcher will just stop until it is restarted 12 hours later.

As usual, please let me know if something is not working as it should be or if something is not clear.

cc @dnmTX @mitchellkrogza @Ultimate-Hosts-Blacklist/blacklister @Ultimate-Hosts-Blacklist/whitelister,

[funilrys]

Note: I just requested Travis CI to delete all repositories and data behind:

• https://travis-ci.org/github/Ultimate-Hosts-Blacklist
• https://travis-ci.com/github/Ultimate-Hosts-Blacklist

[dnmTX]

@funilrys where can i see the filtering progress? Could you post a link or something as i'm yet not very familiar with GitHub Actions. Thanks 👍

[funilrys]

@dnmTX, unfortunately, there is no central place with all the running builds. But it's under the Actions tabs of all repositories. Or if you prefer: https://github.com/Ultimate-Hosts-Blacklist/{repo_name}/actions

[Somebodyisnobody]

From now on, each input source will automatically try to restart every 12 hours. That means that at those times

* 00:00 UTC

* 12:00 UTC

Maybe you should shift the time a few minutes before 12 and 0 to get more server capacity. I think a lot of people are processing something at the current set times

[Somebodyisnobody]

So if I want to look, if new entries were processed successfully and now in the ultimate hosts list, which job i need to look into?

[funilrys]

Uhh thanks for the screenshot. That dead-hosts thing went through with copy paste.

Actually, you can look at anyone running (orange). When done (like in your screenshot) the last [results] shall be looked at. That's the one that provides the latest update of all *.list files.

[Somebodyisnobody]

Okay, is it possible to show additions and deletions in the run-launcher stage by the script?

[spirillen]

Maybe you should shift the time a few minutes before 12 and 0 to get more server capacity. I think a lot of people are processing something at the current set times

That does not currently seems possible. I've experimented with this on my own runners, by setting various starting minutes, but they are not picked up. They all starts at about 0 x x x x x

The only part that is respected is hour/dayOfWeek etc, not minutes. But if you can break that part, I'm all ears.

[funilrys]

Note: As @Somebodyisnobody mentioned previously, I changed the scheduler. It will be every 12:30.

Meaning at those times:

• 00:30 UTC
• 12:30 UTC

This change will by applied as soon as a single launcher will be executed.

[funilrys]

@spirillen ==> https://crontab.guru/ may help.

[spirillen]

hmm... I'm exited to see the results of that

Taken from another private runners of my at 17:27 CEST

[spirillen]

@spirillen ==> https://crontab.guru/ may help.

I know my crons, but thanks :smile:

[funilrys]

Keep in mind that the time under GHA in UTC too.

[spirillen]

Keep in mind that the time under GHA in UTC too.

Yeps, and to help others, a link to the rightful docs https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onschedule

You can schedule a workflow to run at specific UTC times using POSIX cron syntax. Scheduled workflows run on the latest commit on the default or base branch. The shortest interval you can run scheduled workflows is once every 5 minutes.

PS: UTC != CEST :yum: UTC is at winter time CEST -1 hour :smirk:

[Somebodyisnobody]

And nobody want's to switch between summer and winter time every half year but also this nobody doesn't want to change something...

[funilrys]

Hello @dnmTX, @smed79 @Somebodyisnobody @mitchellkrogza @Ultimate-Hosts-Blacklist/whitelister @Ultimate-Hosts-Blacklist/blacklister

Just for information, GitHub disabled GitHub action in this organization. I guess we were putting a lot of "burden" in there even if I configured PyFunceble to be as slow as possible.

It's now too much for me. Give me some time to ingest it.

I'm going to work on other projects until I can find something suitable. We need to find an alternative or completely abandon the testing behind GitHub Actions.

Stay safe and healthy.

P.S.: If someone wants to help, the test launcher source code is available there: https://github.com/Ultimate-Hosts-Blacklist/test-launcher. The deployment launcher will be available this weekend out there: https://github.com/Ultimate-Hosts-Blacklist/deployment-launcher.

[spirillen]

Get these one up and running... That is the entire idea of the infrastructure of what you are trying to do within this project

• https://www.mypdns.org/project/view/34/
• https://www.mypdns.org/project/view/15/

But there will need a rewrite of the db structure to make it functional

Tree main tables

Primary: Table containing records with there test status https://www.mypdns.org/T1250 (all kinds) sub-table: source linked to from Primary Whois: linked to from primary

Then we "just" needs some thin clients https://www.mypdns.org/project/view/26/ which will relay on voluntary users installing the test client.

[dnmTX]

@funilrys tha is not goood news. Any thoughts if we can implement this in GitLab?

[Somebodyisnobody]

Just for information, GitHub disabled GitHub action in this organization. I guess we were putting a lot of "burden" in there even if I configured PyFunceble to be as slow as possible.

It's now too much for me. Give me some time to ingest it.

Oh no! I suspected that the thing with the Github Actions has a pitfall. Every bait must have one hook. Nothing is given for free. Now you've put so much time into it and it's all for the birds.

Tree main tables

Primary: Table containing records with there test status https://www.mypdns.org/T1250 (all kinds) sub-table: source linked to from Primary Whois: linked to from primary

Many more :) I don't have the time at the moment but I dream about a revision-based system where users can edit and comment entries.

[Somebodyisnobody]

[spirillen]

...Many more :) I don't have the time at the moment but I dream about a revision-based system where users can edit and comment entries.

That was as a followup on many brainstorms, talks etc @funilrys and I have had over the time, but yes. the final DB should contain a hole lot more tables with crossrefs for speed, cache, data pulling etc... :smile:

[Somebodyisnobody]

That was as a followup on many brainstorms, talks etc @funilrys and I have had over the time, but yes. the final DB should contain a hole lot more tables with crossrefs for speed, cache, data pulling etc... 😄

Do you know a good software for creating ER-modells? (Or which one you are using?)

[spirillen]

Well 20 years of experience in working with DB's :smirk: (Have them running, not programming) So I have a little idea about where you can find the smal changes to optimize them :) (I hate coding my self, that's why I'm crossing my fingers on you guys, for that part)

PS: The answer is paper, pencil and a fresh soft rubber :wink: .... yes I'm that old :older_man: :bald_man:

[spirillen]

I might actually be able to offer some decent power to the lists, but they comes with a trade.

I'll need a program that can manage the RPZ zone not only the adult related https://www.mypdns.org/T2475

In return we can setup Conda on https://www.mypdns.org/ and run your tests and get https://www.mypdns.org/project/view/15/ started.

Your UHB script should be running in Conda, cron, and for each run do a git fetch, run test, delete git (To keep disk usage low).

Or if you prefer https://github.com/Ultimate-Hosts-Blacklist/test-launcher & https://github.com/Ultimate-Hosts-Blacklist/deployment-launcher to step up to this project I'm very fine with that too. But from the perspective of time, I'll suggest to start it as cronjob.

This is my offer.

[funilrys]

The (new) testing infrastructure is up and running.

I created a Jenkins instance at safedns2.allover.co.za. You can follow the status at https://safedns2.allover.co.za/jenkins/job/ultimate-hosts-blacklist/. As of now, it is still my objective to reduce the overall load. Therefore, all re-test will be started every 4h (will change in the future).

People who can or want to help by providing a few resources to help us with our infrastructure (behind Jenkins) can contact me directly by email, KeyBase (preferred) or Telegram. My SSH keys can be found at https://github.com/funilrys.keys.

@dnmTX What's next on your list? (Sorry, I lost track)

[funilrys]

Hey all,

an update regarding this issue. We plugged the Jenkins instance off a few weeks ago because of a huge instability. After discussion with the Support team of GH, we are back on GHA. It's not perfect yet as there is still a lot to individually configure. But we are going to get there - somehow.

Don't be afraid to @\ping me, if I miss something of if there is any questions.