IP sources and bot type

[kevallakhani]

Hi,

Thank you for creating this repo so that other developers can use it and take advantage of an open source blocklist. I have a few questions -

• I know it mentions there are 30+ feeds that you connect to. Is there any spec on what those feeds cover? and how many updates or new ips do you plug in from those every day or in the frequency you update it at?
• What malware types are covered? My use case specifically needs to cover bot mitigation cases like web scraping bad bots and content abuse bots over the net. Are those cases covered?

Thanks in advance.

[stamparm]

1) List are literary these: https://github.com/stamparm/maltrail/tree/master/trails/feeds. Based on those python script scraping script names you can easily deduce what each of those have as a source 2) These lists are more for "inbound" traffic blocking than for "outbound" traffic blocking. Thus, if you are watching for "malware" blacklist of IPs I believe that you are at the wrong place. Even though there are IPs also for known C&Cs included here, majority of those are pretty much static, without using some fresh list of C&C IPs 3) "web scraping bad bots" and "content abuse bots" are mostly covered with these lists. I believe that you've mixed the "malware" with "bot" terminology

[kevallakhani]

Hi,

One more question - the list when updated periodically, is it generated as a totally separate list from the previous list or is generated as an override list that overrides the previous list values?

Thanks.

On Tue, Nov 10, 2020 at 1:50 PM Miroslav Stampar notifications@github.com wrote:

1. List are literary these: https://github.com/stamparm/maltrail/tree/master/trails/feeds. Based on those python script scraping script names you can easily deduce what each of those have as a source
2. These lists are more for "inbound" traffic blocking than for "outbound" traffic blocking. Thus, if you are watching for "malware" blacklist of IPs I believe that you are at the wrong place. Even though there are IPs also for known C&Cs included here, majority of those are pretty much static, without using some fresh list of C&C IPs
3. "web scraping bad bots" and "content abuse bots" are mostly covered with these lists. I believe that you've mixed the "malware" with "bot" terminology

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stamparm/ipsum/issues/10#issuecomment-724987988, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIV3SIC23S7LLTTQHDUJBETSPGYTXANCNFSM4TRA3KHA .

-- Keval Jayesh Lakhani | Engineering | 201.989.7548 | keval@houzz.com

[stamparm]

@kevallakhani it is done from scratch every time. This means that if IP disappears from all retrieved blacklists, it will disappear from ipsum too

[kevallakhani]

Got it.

Thanks for answering my questions. Do you also sometimes include any good search engine bots in the list like google, yahoo, bing, etc? just curious about that.

On Wed, Nov 18, 2020 at 12:50 AM Miroslav Stampar notifications@github.com wrote:

@kevallakhani https://github.com/kevallakhani it is done from scratch every time. This means that if IP disappears from all retrieved blacklists, it will disappear from ipsum too

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stamparm/ipsum/issues/10#issuecomment-729531650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIV3SIBFEQSQHPN6J4TATFDSQODHBANCNFSM4TRA3KHA .

-- Keval Jayesh Lakhani | Engineering | 201.989.7548 | keval@houzz.com