Closed ssnkhan closed 3 years ago
ninoseki
commented
3 years ago I cannot reproduce the error. The following two commands work in my env.
mihari shodan 'ssl:"MetasploitSelfSignedCA" http.favicon.hash:"-127886975"'
mihari shodan 'hash:-1564315793'
ssnkhan
commented
3 years ago How odd.
The queries work providing there is not a result which contains something that breaks the parser. When that happens, none of the results (for that query) get pushed into the db. I had the same issue today:
/usr/lib/ruby/2.7.0/json/common.rb:156:in `parse'
/usr/lib/ruby/2.7.0/json/common.rb:156:in `parse'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:45:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-1.5.0/lib/mihari/analyzers/shodan.rb:44:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-1.5.0/lib/mihari/analyzers/shodan.rb:54:in `block in search'This is the IP causing the issue today: https://beta.shodan.io/host/180.215.106.226
Is there an equivalent of on error resume next or something similar? So if the Metasploit query say has 100 results, if one triggers an error, none of the remaining 99 are added to the DB.
Really appreciate your help.
ninoseki
commented
3 years ago I cannot fix the issue which cannot reproduce
ssnkhan
commented
3 years ago I have done a fresh install of Ruby 3.0 on a fresh Ubuntu machine, and seem to keep running into this issue. I tried updating cli.rb with :quirks_mode => true (i.e. json.parse (input, :quirks_mode => true) but that falls over completely. According to Google, the error is caused by invalid JSON, in this instance being returned by the Shodan API.
Is there anything I can provide with regard to debugging that might help identify the root cause? I am wondering whether this issue happens if there are many results (some searches return 50+ results).
Thanks again for all your help!
ssnkhan
commented
3 years ago Just had another look into the log file that I have been using to capture the errors. Looks like the same parse error is generated when the API returns a api.shodan.io | 502: Bad gateway error. What follows is the Cloudflare interstitial (HTML).
I wonder whether the size of the response could also be at play here; the other JSON errors appear in the log file and abruptly cut off part way through the http.favicon_hash key/value, which does contain numerous characters.
ninoseki
commented
3 years ago @ssnkhan my understandings are:
Are they right? If so I think it's good to claim the issue to Shodan. 🤔
ninoseki
commented
3 years ago Also, please provide the actual command you ran. It will make me easier to find a clue from your complaints.
ssnkhan
commented
3 years ago Thanks @ninoseki. The query is:
mihari shodan 'ssl:"MetasploitSelfSignedCA" http.favicon.hash:-127886975 -http.title:"Bam! Application Error"' --title "Metasploit C2" --tags "C2"
This query, and maybe 20 others, are ran via a cronjob. Looking at the result on Shodan (for the specific error in my log file) for the host https://beta.shodan.io/host/180.215.106.226/raw, the JSON is complete, but you will note that the response is very lengthy.
Having done some more Googling, I suspect the issue may also be related to throttling by Shodan. Let me add some sleep 5 into my script and see if the issue persists.
ssnkhan
commented
3 years ago Update: sprinkling sleep 5 has had no effect, and the same query continues to generate the above errors (not always, but reasonably consistently).
ninoseki
commented
3 years ago Could you report the issue to Shodan, please? The issue is caused by the error on the Shoda API side. There is nothing I can do it with.
ssnkhan
commented
3 years ago Thanks @ninoseki — I’ll raise the issue with Shodan.
However, could error handling be improved so that invalid JSON doesn’t interrupt the importing of other valid IOCs as part of the same query?
ninoseki
commented
3 years ago However, could error handling be improved so that invalid JSON doesn’t interrupt the importing of other valid IOCs as part of the same query?
Okey, it makese sense to me. I will work on that.
Hello there,
I have noticed that a rogue Shodan Metasploit search is causing Mihari to throw an error, resulting in all search results for that query not being added to the database. The error is:
The corresponding JSON in the error log is:
This specific IP/result is creating the error (and always this specific "Bam! Application Error"): https://www.shodan.io/search?query=hash%3A-1564315793
The original query is:
ssl:"MetasploitSelfSignedCA" http.favicon.hash:"-127886975"I have added
-http.title:"Bam! Application Error"to the original query to suppress this result, but not sure if the issue can be addressed in the core Mihari code, either with more graceful error handling, or addressing the parse issue please?Really appreciate all your hard in this excellent tool!