Too much load on bgp.tools

[benjojo]

For the past few months bgp.tools has been gaining more and more traffic table.jsonl from a mysterious chrome user agent, from a large range of what are suspected to be VPN IP ranges.

This user agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36

Is against the API/Scripting recommendations of the site:

You must a HTTP User-Agent to identify yourself on these requests. Default user agents are not supported as we have no contact method to reach you if something goes wrong.

Systems who use a default or generic user agents to fetch bgp.tools data constantly may be blocked.

The preferred user agent format is:

acmeco bgp.tools - contact@acme.co

The current request rate from this user-agent (that has been blocked for at least the last 60 days) has reached over 5 RPS.

That sounds small, however that makes a huge bandwidth impact when the file it is fetching is 50MiB, meaning to serve that the website would be pumping out 2gbit/s of pure jsonl file to serve this workload. At the cheapest possible carrier rates that would cost at least $100/m in bandwidth to serve.

Had I been serving this out from AWS Cloudfront this would cost me $29,450 a month.

Anyway, I investigated this by looking up bgp.tools on virustotal and found that amass is used in what look like a large number of payloads. And upon investigation, there is code that behaves as I am witnessing.

It seems reasonable to assume there is a large malware campaign using amass right now due to the sheer % of these requests coming out of VPN ranges

https://github.com/OWASP/Amass/blob/master/resources/scripts/api/bgptools.ads added on https://github.com/OWASP/Amass/commit/5abed3b1ca54a4a88cab96e3e69ae06e865d3da8 by @caffix appears to be driving this load. And that Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 appears to be the default user agent of amass.

My requests are:

A) Please do not use the default user agent here. This has taken me significant time to discover. Please consider using owasp amass <version> for easy identification. B) Please do not fetch table.jsonl if you think you are only going to be making 1 or 2 lookups. Please use the whois server instead.

---

bgp.tools plans to display a banner warning of compromise for the IPs that are hitting this. Since it seems that these requests look highly likely to be of malware aligned amass installs

[caffix]

Hello Ben,

First of all, thank you for providing such a helpful service! I'm sorry to hear that our tool could be creating any unnecessary burden on your organization. Here's what I've concluded and/or done since reading your issue post:

1) I can definitely change the user-agent. I have modified it to your liking, causing future versions of the integration to have a user-agent that look like the following example: OWASP Amass v3.21.2 - admin@owasp.com

2) We do need the table.jsonl file to obtain all the required data, but we were previously caching that file for 5 hours. I have changed the caching to now hold the file for 24 hours.

3) Some of the traffic coming from VPN ranges could just be bug bounty hunters using the OWASP Amass tool.

I hope this satisfies your requests. Thank you for the helpful feedback and please let me know how else we can help.

[timlaw71]

@benjojo I want to make sure the script I created for our pre sales recon isn't causing this. Is there any way you can share the ips with me so I can make sure this isn't us?

[benjojo]

For data privacy reasons I don't keep detailed logs of user IP addresses for general web requests, however if you have ip's you think might be doing this, please email @.*** and I can try and do some digging

On Sun, Mar 12, 2023 at 12:48 AM timlaw71 @.***> wrote:

@benjojo I want to make sure the script I created for our pre sales recon isn't causing this. Is there any way you can share the ips with me so I can make sure this isn't us?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[prisoner881]

Hello Ben,

First of all, thank you for providing such a helpful service! I'm sorry to hear that our tool could be creating any unnecessary burden on your organization. Here's what I've concluded and/or done since reading your issue post:

1. I can definitely change the user-agent. I have modified it to your liking, causing future versions of the integration to have a user-agent that look like the following example: OWASP Amass v3.21.2 - admin@owasp.com
2. We do need the table.jsonl file to obtain all the required data, but we were previously caching that file for 5 hours. I have changed the caching to now hold the file for 24 hours.
3. Some of the traffic coming from VPN ranges could just be bug bounty hunters using the OWASP Amass tool.

I hope this satisfies your requests. Thank you for the helpful feedback and please let me know how else we can help.

caffix, something to consider. In my use case, I'm running amass against many different domains at the same time. I do this by launching amass with domain1 and putting the results in a specific directory using the -dir flag, then doing the same with domain2 (but in a different directory). It looks like the bgp file is being downloaded with each amass instance, as I see the bgptools.jsonl file in every directory.

I know your intended use is I should run all these domains using the -df flag, but in my use case that's not practical as the list of domains is constantly changing. WIth this being the case, is there a way you could pull the bgptools.jsonl file once and have all flag that tells amass where to find it?