rsync support for sanesecurity updates

[rfc1036]

I have recently started trying fangfrisch as a replacement for the old clamav-unofficial-sigs package, but now I have serious doubts about its viability. That script only uses rsync and, while rsync.sanesecurity.net has an healthy list of IPs, I cannot find a list of official HTTP mirrors for sanesecurity. The default configuration uses an australian HTTP mirror which is seriously overloaded and rarely responds in less that 30 seconds. Also, this is not a sensible default because the bandwidth costs for Australia are very high.

Also, I see that the default configuration still downloads files like malware.expert.* and phishtank.ndb, which have been empty for years.

[rseichter]

Sanesecurity timeouts and signatures have been discussed in #27 and #29 over the past two weeks. I also mentioned an alternative HTTP mirror. Please check these (closed) issues and let me know if you have further questions which weren't answered there.

[rfc1036]

I am aware that it is possible to change the timeout and/or use a different mirror, which indeed I am using, but I still believe that it is not sustainable to rely on just two unofficial mirrors, one of which is hardly usable because downloading a file takes one minute (and which will be probably shut down once they notice the international traffic).

[rseichter]

So what exactly are you asking for by opening this issue? That rsync support is added to Fangfrisch? I want to ensure that there are no misunderstandings, and that different subjects each have their separate GitHub issues.

[rfc1036]

I cannot ask for anything, but I suggest that rsync support is implemented for the sanesecurity rules since this is what that project uses officially and recommends, and the HTTP mirrors are unofficial and unreliable.

[rseichter]

I found this page while checking up on Sanesecurity mirrors lately. mirror.rollernet.us appears to be a reasonable option for now. Still, rsync support would be nice. However, Fangfrisch delegates all file retrieval to the Requests library, which does not include rsync support. This requires more research on my end. I'll leave this issue open as an enhancement request for now.

[rseichter]

I have contacted Sanesecurity regarding information about their mirror network. Currently waiting for their response.

[kitterma]

I checked the download script that sanesecurity recommends on their web site, and it uses rsync. While I recognize the added complexity, I think it's the way to go.

[rseichter]

As I wrote, I am currently waiting for information from the Sanesecurity team. I need to know more about their mirrors and procedures before I can work on design changes.

[rseichter]

I received some feedback from the Sanesecurity folks, based on which I have started testing ideas.

[rseichter]

In cooperation with Steve Basford (Sanesecurity), I have set up two new mirrors on my production servers. One uses rsync to pull data off Sanesecurity. The other one, backed by the rsync'd data, serves Fangfrisch using HTTP. Release 1.8.0 reflects the new mirror situation in its default configuration. Custom Fangfrisch configuration files need only drop the prefix = http://mumble.example.com for Sanesecurity—if present—to use the new mirror.

[rseichter]

The number of Fangfrisch users connecting to the new HTTP mirror is slowly ramping up. Things are looking as expected so far, and I cannot spot any trouble in my log files.

[rseichter]

The number of unique IP addresses connecting to my new mirror has since climbed to n>400. The logs still show no problems.

[kitterma]

I looked at the Debian popcon data (which is opt-in, so it's not a representative sample, but it's the one we have). Roughly 5% of systems reporting what they have installed have clamav installed. I looked back at clamav-unofficial-sigs when it was maintained and about 5% of clamav users also had it installed.

There's no way to know how many users there are of Debian and its derivatives, but it appears to be near half of Linux users. If you end up with only 0.25% of them running Fangfrisch, it's still a lot of users. Are you prepared for an order of magnitude more than that?

[rseichter]

Thank you for the heads-up, Scott. The production machine I chose is routinely handling around 100 GB 95 TB of monthly network traffic and can handle more. Edit: I looked at the wrong server report. Sorry for that. 🫢

[rseichter]

With a new, reliable HTTP mirror in place, rsync support in Fangfrisch is no longer a pressing matter. Therefore, I am closing this issue for the time being. That does not mean I want to cut the conversation short, and the issue could be revisited in the future.