rc-cts-urlscanio Not Returning Hits

[lmahoney1]

Description

Our analysts mentioned that they had not seen a hit from the urlscan.io cts in awhile. We deployed rc-cts-urlscanio from the community.

It appears that the check the CTS does to determine if the results are malicious or not no longer works (around line 136). Maybe URL Scan.io changed their API response?

We used the following url https://urlscan.io/result/38135c02-509d-471f-91a7-0466060c3474/ for testing. We found that the urlscan.io GUI marked the URL as 'potentially malicious', however when submitting it to a running CTS we got an empty hit.

The following code sample may explain better:

import requests
headers = { 'Content-Type': 'application/json' }
# looking up URL in urlscanio
url_results = requests.get('https://urlscan.io/api/v1/search/?q=page.url:"router-ed1c9cd2-b691-4867-b4d4-fb8843fa91b1.eastus.cloudapp.azure.com"', headers)

# the first search result found - one of the times the URL was analyzed on the site
result = url_results.json().get("results")[0]
# ID of the result / analysis
result_id = result.get("_id")

# getting a detailed report of the analysis
result_response = requests.get('https://urlscan.io/api/v1/result/38135c02-509d-471f-91a7-0466060c3474', headers).json()

# the current 'is it malicious' check
result_response.get("stats").get("malicious") # outputs 0 -> not malicious, no hit is returned

# potential new 'is it malicious' check
result_response.get('verdicts').get('urlscan').get('malicious') # outputs True, a hit should be generated

In the example above I'm only checking one of the two (at the time of writing this) analyses available for the URL. This is just for the example, I verified that both of the analysis have the same issue.

Describe How to Reproduce

Submit a 'potentially malicious' URL to the rc-cts-urlscanio CTS.

[lmahoney1]

Also, I tried spinning this up in apphost after adding all of the required apphost files, but I wasn't able to communicate with it. Maybe I was doing it wrong?

I uncommented the EXPOSE 9000 line in the Dockerfile I used to generate my docker image. The problem was communicating with the pod/container the app runs on from outside the apphost server. I tried sending requests to both the app host server's IP as well as the container ID (retrieved from sudo kubectl describe pod <rc-cts-urlscanio pod ID>.

[mscherfling]

Custom Threat Feeds cannot be deployed inside a container. This is because containers don't have fixed IP Addresses which the webhook requires when registering it with Resilient.

[lmahoney1]

That's too bad but it makes sense. The following line in the Dockerfile gave me hope

# uncomment to expose port only if a custom threat feed
#EXPOSE 9000