XForceExchange - URL wrong cast

[BHanq]

What happened

XForceExchange URL analyzer cast special character into %xx characters. Then the api, is wrongly called. And an error is raised.

Environment

1. OS: Ubuntu 20.0.4
2. IntelOwl version: 3.3.2

What did you expect to happen

Having an analysis for this : https://exchange.xforce.ibmcloud.com/url/http:~2F~2Fmoviesfoundonline.com

How to reproduce your issue

Start an analysis on XForceExchangeAnalyser with an URL (like http://moviesfoundonline.com), the analysis will fail because the url is cast into special characters in string using the %xx escape

Error messages and logs

404 Client Error: Not Found for url: https://api.xforce.ibmcloud.com/url/malware/http%3A%2F%2Fmoviesfoundonline.com

Why it happens

I think that the issue comes from the XForceExchangeAnalyzer at line 29, url is parsed using quote_plus(), thus it replace characters...

[mlodic]

thanks for reporting this. I checked and the error was the main URL that was missing the "/api" but still worked anyway sometimes....the right url is "https://api.xforce.ibmcloud.com/api" The fix will be available next release.

[BHanq]

Are you sure it is not : https://exchange.xforce.ibmcloud.com/api/ Because in api.xforce.ibmcloud.com I have found nowhere in the documentation that missing /api. Sample stays : https://api.xforce.ibmcloud.com/url/www.ibm.com%2Fsmarterplanet in the docs

[mlodic]

yeah, I also tested that manually. In their API docs they also show raw calls with curl. There I found out that bug

[BenjaminH-Sogeti]

@mlodic have you already done it or would you like help ?

[mlodic]

done this but not released yet :P I'll plan a new release soon, please follow the project so you get the update!

[BenjaminH-Sogeti]

Has it changed ?

Still got : 404 Client Error: Not Found for url: https://api.xforce.ibmcloud.com/api/url/malware/www.moviesfoundonline.com

Images

Following the actual xforce website API call should be like :

WWW

curl -X GET --header 'Accept: application/json' -u {API_KEY:API_PASSWORD} 'https://exchange.xforce.ibmcloud.com/api/url/www.moviesfoundonline.com'

HTTP

curl -X GET --header 'Accept: application/json' -u {API_KEY:API_PASSWORD} 'https://exchange.xforce.ibmcloud.com/api/url/http://moviesfoundonline.com'

[mlodic]

thanks for reporting this again!

I am sad cause I tested that manually. I retried that now and you are right, it does not work. 🤔

I followed their docs:

Meanwhile, if you want to fix it locally (to avoid waiting for the next release), you can just apply the modification by calling git checkout xforce_fix, shut down your instance and rerun it with a local build python3 start.py test down && python3 start.py test up --build -d. I have also added a "link" to the web GUI in the resutls

[mlodic]

on a second thought, I think that it is normal behavior. It just means that that URL was not found in their DB.

We should change the analyzer to do not fail in that case.

[mlodic]

If the result is 404, thanks to the new change in the branch xforce_fix, you will found a new key "found" that is set to False.

[BenjaminH-Sogeti]

Thanks a lot !

[mlodic]

consider this solved and shipped in the next release