Github lists no longer being updated

[philwhineray]

I'm making an educated guess this was being done automatically by the london.nedata.rocks server which seems to serve some of the files directly?

Whether is was that server or some other, the push to github seems to be broken, so the files are going stale.

[scriptzteam]

https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

[alyandon]

Is there another independent/blessed way to acquire these lists while the mirror process is fixed? I don't want to assume that having automation download using the direct download links like https://iplists.firehol.org/files/firehol_level3.netset would be acceptable.

[alyandon]

Just leaving this here in case anyone else comes along - I forked this repo and use a script to download the lists I'm interested in from the firehol.org website and add that as a commit to the repo. I run the script every 24 hours to keep things in sync and use that to drive my other firewall automation.

[alyandon]

Leaving this here as an example:

#!/bin/sh

curl -s -o firehol_level2.netset 'https://iplists.firehol.org/files/firehol_level2.netset'
curl -s -o firehol_level3.netset 'https://iplists.firehol.org/files/firehol_level3.netset'

git add -A *.netset
git commit -m 'updating lists'
git push

[mlodic]

@ktsaou do we consider this project definitely abandoned or not?

[v3DJG6GL]

@philwhineray or @ktsaou could you please make a statement what's going on with this project? Do we have to find other solutions in the near future?

[philwhineray]

I’ve never really been involved in the iplists side of FireHOL, but in the absence of any response from Costa, if a few people step up, I'd be quite happy to try and help them stop things falling into further disrepair.

I'm not really in a position to review code changes, but if people collaborate to make merge requests and review and confirm things continue to work OK, I'll be happy to pull those requests.

The site https://iplists.firehol.org/ which distributes the IP lists was set up by Costa - I think it runs on netdata infrastructure, and that it is still updating, but the recent github ssh key change ( https://github.com/firehol/blocklist-ipsets/issues/263 ) has broken its push to the firehol/blocklist-ipsets repo.

I don't have any access to the site, but if someone were to set up an alternative, I have the necessary credentials to repoint the cloudflare proxy to it.

On Thu, 22 Jun 2023 at 16:13, v3DJG6GL @.***> wrote:

@philwhineray https://github.com/philwhineray or @ktsaou https://github.com/ktsaou could you please make a statement what's going on with this project? Do we have to find other solutions in the near future?

— Reply to this email directly, view it on GitHub https://github.com/firehol/blocklist-ipsets/issues/263#issuecomment-1602818626, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJWKCZJ6FI6GSZKIJDOS53XMROJVANCNFSM6AAAAAAWOYTJ7Q . You are receiving this because you were mentioned.Message ID: @.***>

[alyandon]

@philwhineray My opinion is that if you are not able to fully assume control then the project is still doomed and people should be actively looking elsewhere for a similar solution. If there is a way to rehost all the essential core components (including the bits responsible for fetching and aggregating the raw upstream source lists) then maybe setting up on an alternate domain is a viable option.

[philwhineray]

To be clear, I have permissions to the necessary infra such that if someone were to set up an iplists site mirror with updated lists, I could update DNS etc., all under the existing domain.

But I don't have the time (or desire) to run a blocklist project, so yeah, unless someone steps up to take over that bit, looking elsewhere seems reasonable.

On Fri 28 Jul 2023, 17:53 Shelby Cain, @.***> wrote:

@philwhineray https://github.com/philwhineray My opinion is that if you are not able to fully assume control then the project is still doomed and people should be actively looking elsewhere for a similar solution. If there is a way to rehost all the essential core components (including the bits responsible for fetching and aggregating the raw upstream source lists) then maybe setting up on an alternate domain is a viable option.

— Reply to this email directly, view it on GitHub https://github.com/firehol/blocklist-ipsets/issues/263#issuecomment-1656001979, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJWKCYUWAFB7A54YJJVLKDXSPVB7ANCNFSM6AAAAAAWOYTJ7Q . You are receiving this because you were mentioned.Message ID: @.***>

[alyandon]

Right - and that is a totally fair statement to make. I could donate some of my personal time and resources to help get something up and running but there is no way that I could help out long term with such an endeavor either.

[Enrico204]

But I don't have the time (or desire) to run a blocklist project, so yeah, unless someone steps up to take over that bit, looking elsewhere seems reasonable.

@philwhineray Just to be clear, does this include also the update-ipsets.sh script in the FireHOL repo? Or that project is still maintained?

[philwhineray]

The update-iplists.sh script is what is powering the iplists site as I understand it. Pull requests do happen but I'm not in a position to review them.

I'm happy to merge updates which get some review (and grant comitter privileges in due course) but I don't have time to do the work or the reviews.

More generally the firehol repo is not being actively worked on, but I personally would still use the main firehol script if I had a use-case, because all it really does is generate iptables rules. At some point I guess maybe that will become non-viable if backwards compatibility is dropped from nftables, but until then I would just regard it as very stable.

On Fri 28 Jul 2023, 19:13 Enrico, @.***> wrote:

But I don't have the time (or desire) to run a blocklist project, so yeah, unless someone steps up to take over that bit, looking elsewhere seems reasonable.

@philwhineray https://github.com/philwhineray Just to be clear, does this include also the update-ipsets.sh script in the FireHOL repo? Or that project is still maintained?

— Reply to this email directly, view it on GitHub https://github.com/firehol/blocklist-ipsets/issues/263#issuecomment-1656134146, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJWKCYDFF5UORXZ7BLBIBTXSP6MPANCNFSM6AAAAAAWOYTJ7Q . You are receiving this because you were mentioned.Message ID: @.***>

[Enrico204]

I use some FireHOL generated lists on my personal server and router, but I don't think I have enough resources to set up a mirror. Also, maintaining a bash script of nearly 8k lines (with an embedded XML parser!) it's not feasible, at least for me.

If no one steps up, I think I will create and share my own index of curated lists. I think that nowadays FireHOL is used mostly for IP blocklists (as firewalls like OPNSense can automatically download and configure a set of IPs in pf), as I do.

I am grateful to you and @ktsaou for all the work you've done :-)

[Enrico204]

In the past few days I extracted and checked all IP lists from the update-ipsets script. Some of them are dead, some changed URL.

Shameless plug: I created a new repository to track them in separated files. It should be easy to keep updated: https://github.com/Enrico204/blocklists

I also developed a tool in Go to download, clean and merge these lists. In some way, it is similar to update-ipsets, although it is focused on download and merge lists, nothing more. I plan to add the history and the web page generation in the future, just like the FireHOL script does :-)

I also compiled a list of "changes" between my index and the current update-ipsets (mostly dead blocklists, etc.), so if someone wants to update FireHOL index can start from here: https://github.com/Enrico204/blocklists/blob/master/FIREHOL.md

I plan to add other lists in my index. Feel free to use it and contribute :-)

[mlodic]

great job, thanks for sharing!

[alyandon]

@Enrico204 Very nice - I was actually toying with re-writing the core stuff in Go and you already did it. :-D

[borestad]

I've added a firehol mirror here: https://github.com/borestad/firehol-mirror

[ploink]

Firehol is updating again! I sent an email to the address on @ktsaou's github profile page and he promptly fixed it!

Thank you for this notification. I think I fixed it. I didn't notice because the site https://iplists.firehol.org was working fine.

See also #270 and #283