Koodous API doesn't work anymore

[vicpala]

     INFO     [mvt.android.download_apks] Found non-system package with name
              "com.microsoft.teams" installed by "com.android.vending" on   
              2022-05-11 14:24:27                                           
     ERROR    [mvt.android.lookups.virustotal] Unfortunately VirusTotal     
              lookup is disabled until further notice, due to unresolved    
              issues with the API service.                                  
     INFO     [mvt.android.lookups.koodous] Looking up all extracted files  
              on Koodous (www.koodous.com)                                  
     INFO     [mvt.android.lookups.koodous] This might take a while...      

Looking up 151 packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- Traceback (most recent call last): File "/home/vicpala/.local/lib/python3.10/site-packages/requests/models.py", line 910, in json return complexjson.loads(self.text, **kwargs) File "/home/vicpala/.local/lib/python3.10/site-packages/simplejson/init.py", line 525, in loads return _default_decoder.decode(s) File "/home/vicpala/.local/lib/python3.10/site-packages/simplejson/decoder.py", line 373, in decode raise JSONDecodeError("Extra data", s, end, len(s)) simplejson.errors.JSONDecodeError: Extra data: line 1 column 5 - line 2 column 1 (char 4 - 19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/vicpala/.local/bin/mvt-android", line 8, in sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke return __callback(args, *kwargs) File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), args, **kwargs) File "/home/vicpala/.local/lib/python3.10/site-packages/mvt/android/cli.py", line 93, in download_apks download.run() File "/home/vicpala/.local/lib/python3.10/site-packages/mvt/android/download_apks.py", line 181, in run self.get_packages() File "/home/vicpala/.local/lib/python3.10/site-packages/mvt/android/download_apks.py", line 117, in get_packages m.run() File "/home/vicpala/.local/lib/python3.10/site-packages/mvt/android/modules/adb/packages.py", line 266, in run koodous_lookup(packages_to_lookup) File "/home/vicpala/.local/lib/python3.10/site-packages/mvt/android/lookups/koodous.py", line 34, in koodous_lookup report = res.json() File "/home/vicpala/.local/lib/python3.10/site-packages/requests/models.py", line 917, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: [Errno Extra data] 404 page not found : 4 vicpala@vicpala-H97N-WIFI:~/mvt$

[mzalazar]

Same here:

Looking up 154 packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- 14:24:12 ERROR [mvt.android.modules.adb.packages] Error in running extraction from module Packages: [Errno Extra data] 404 page not found
: 4
Traceback (most recent call last):
File "/home/mzalazar/.local/lib/python3.8/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/home/mzalazar/.local/lib/python3.8/site-packages/simplejson/init.py", line 525, in loads
return _default_decoder.decode(s)
File "/home/mzalazar/.local/lib/python3.8/site-packages/simplejson/decoder.py", line 373, in decode
raise JSONDecodeError("Extra data", s, end, len(s))
simplejson.errors.JSONDecodeError: Extra data: line 1 column 5 - line 2 column 1 (char 4 - 19)

              During handling of the above exception, another exception occurred:                                                                                                                                                          

              Traceback (most recent call last):                                                                                                                                                                                           
                File "/home/mzalazar/.local/lib/python3.8/site-packages/mvt/common/module.py", line 152, in run_module                                                                                                                     
                  module.run()                                                                                                                                                                                                             
                File "/home/mzalazar/.local/lib/python3.8/site-packages/mvt/android/modules/adb/packages.py", line 243, in run                                                                                                             
                  koodous_lookup(packages_to_lookup)                                                                                                                                                                                       
                File "/home/mzalazar/.local/lib/python3.8/site-packages/mvt/android/lookups/koodous.py", line 34, in koodous_lookup                                                                                                        
                  report = res.json()                                                                                                                                                                                                      
                File "/home/mzalazar/.local/lib/python3.8/site-packages/requests/models.py", line 917, in json                                                                                                                             
                  raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)                                                                                                                                                                       
              requests.exceptions.JSONDecodeError: [Errno Extra data] 404 page not found                                                                                                                                                   
              : 4                                                                                                                                                                                                                          

[90n20]

Seems that Koodus api has changed as per documentation and no longer points to https://api.koodous.com/apks/-> https://docs.koodous.com/api/index.html

Moreover, it has now usage limits and paid options => https://docs.koodous.com/quotas.html

[Te-k]

Thanks for raising that issue, the free or fan API is unusable for MVT, we need to think if we want to adds support for Koodous or just remove it.

[botherder]

I am going to take care of this issue. Since it's the same problem as with VirusTotal, at this point we might as well just add support for providing account keys.

[VMARTTARIN]

yes, unfortunately, if you want use API KOODOUS y VIRUSTOTAL, paid. The prices are excesive. A colleague tells me that he has modified the code and it works for him, I doubt that he paid, but I am willing to pay something to be able to use this tool. if i can help with anything here this.

Looking up 99 packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/models.py",line 910, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.10/dist-packages/simplejson/init.py",line 525, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.10/dist-packages/simplejson/decoder.py",line 373, in decode raise JSONDecodeError("Extra data", s, end, len(s))simplejson.errors.JSONDecodeError: Extra data: line 1 column 5 - line 2 column 1 (char 4 - 19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/mvt-android", line 8, in sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke return __callback(args, *kwargs) File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/mvt/android/cli.py",line 93, in download_apks download.run() File"/usr/local/lib/python3.10/dist-packages/mvt/android/download_apks.py",line 181, in run self.get_packages() File"/usr/local/lib/python3.10/dist-packages/mvt/android/download_apks.py",line 117, in get_packages m.run() File"/usr/local/lib/python3.10/dist-packages/mvt/android/modules/adb/packages.py",line 243, in run koodous_lookup(packages_to_lookup) File"/usr/local/lib/python3.10/dist-packages/mvt/android/lookups/koodous.py",line 34, in koodous_lookup report = res.json() File "/usr/local/lib/python3.10/dist-packages/requests/models.py",line 917, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)requests.exceptions.JSONDecodeError: [Errno Extra data] 404 page not found: 4

[OliverCG]

I'm experiencing the same issue with koodous and virustotal. However, what's the point of having these APIs working? I mean, does it affect to the functionality to identify pegasus, for example?

[VMARTTARIN]

Hello, MVT is a tool developed by Amnesty International's IT department for the investigation of Saudi journalist Khashoggi. MVT is a forensic tool, not a disinfection tool. AI developed MVT for research, using repositories where cybersecurity professionals added their knowledge. MVT is powerful, because it obtains reports on the use of the APKs of the camera, micro, etc. of your mobile. MVT downloads the APK's of your phone to compare with those repositories (Virustotal, Koodous). With the news of Pegassus (I remind you that it has always existed but in other versions, with names like Spy...) the repositories of colleagues that published for free, now ask for money for consultation and comparison of that APK of your mobile in their repository. Without a collaborative complaint database it is not possible to know if your terminal is infected, although I still say that MVT is a very powerful application. I hope the collaboration of all of us who are dedicated to cybersecurity, because the intervention of a third party in our communications constitutes a crime, and more serious when used by governments.

[OliverCG]

Hello, MVT is a tool developed by Amnesty International's IT department for the investigation of Saudi journalist Khashoggi. MVT is a forensic tool, not a disinfection tool. AI developed MVT for research, using repositories where cybersecurity professionals added their knowledge. MVT is powerful, because it obtains reports on the use of the APKs of the camera, micro, etc. of your mobile. MVT downloads the APK's of your phone to compare with those repositories (Virustotal, Koodous). With the news of Pegassus (I remind you that it has always existed but in other versions, with names like Spy...) the repositories of colleagues that published for free, now ask for money for consultation and comparison of that APK of your mobile in their repository. Without a collaborative complaint database it is not possible to know if your terminal is infected, although I still say that MVT is a very powerful application. I hope the collaboration of all of us who are dedicated to cybersecurity, because the intervention of a third party in our communications constitutes a crime, and more serious when used by governments.

Thank you for your information. That's exactly what I though but I wanted to be sure. Of course, I understand that this is not a disinfection tool and that it could depend on several factors like: device type (iOS or Android); SO version; rooted/jailbreaked; etc. So the disinfection should be treated separately. Also, I fully agree with you that MVT is a very good tool and application that could be used for a lot of scenarios and audits.

However, after reading your message, I think I could have a false negative if the functionality is not fully working. Is that correct? For example, if we focus on Pegasus (regardless the version and names during the time), I understand that MVT could try to detect something strange on SMS, WhatsApp backups, etc. but now, MVT is not able to compare the APKs (talking about Android) to see if any app is infected. Please don't hesitate to correct me if I'm wrong.

I'm just trying to know the consequences of not having these APIs working. As you could imagine, if we need to create an inform with a security audit and we're using this tool, if we could have a false negative, it could be totally useless. I understand that there are many ways to infect a device and, for that, there is no tool that could detect it with a 100% case ratio, but if these APIs increased the effectiveness by 20%, It would be a great loss for our privacy and cybersecurity.

[botherder]

Any detection service such as VirusTotal or Koodous should only be treated as additional data points. MVT is intended as a tool for forensic collection and analysis, it's not an antivirus. It tries to implement as many additional detection capabilities as possible through support of indicators as well as originally with those lookup services in order to aide the analysis, but ultimately a forensic analyst using it should not rely exclusively on those to make a conclusive assessment. It's important to learn the process of interpreting system diagnostic data, and become accustomed with spotting anomalies or suspicious artifacts. A tool won't be able always able to do so for you.

That said, I will work on implementing their new APIs support. It will only mean that users will need to register accounts on those services and provide the API keys to MVT in some way.

[VMARTTARIN]

Reviewing the procedure and reviewing the documentation with the command mvt-android download-apks --output /home/tmp should download the APK's for later inspection with KOODOUS or VIRUSTOTAL. This command does not download, but inspects directly, why? Then it gives the error that I already put in the post.

[Oguzbey06]

SAME ERROR IS THIS PROBLEM SOLVED?

[VMARTTARIN]

Hi, i see it's a new version 1.5.5. this versión fix the problem? Thanks

[botherder]

Not yet. When fixed I will update this ticket.

[rondinellepqd]

I would also like to be informed as I am having the same problem when running MVT. I get the error below:

Looking up 137 packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/models.py", line 910, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3/dist-packages/simplejson/init.py", line 525, in loads return _default_decoder.decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 373, in decode raise JSONDecodeError("Extra data", s, end, len(s)) simplejson.errors.JSONDecodeError: Extra data: line 1 column 5 - line 2 column 1 (char 4 - 19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/kali/.local/bin/mvt-android", line 8, in sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke return __callback(args, *kwargs) File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), args, **kwargs) File "/home/kali/.local/lib/python3.10/site-packages/mvt/android/cli.py", line 93, in download_apks download.run() File "/home/kali/.local/lib/python3.10/site-packages/mvt/android/download_apks.py", line 181, in run self.get_packages() File "/home/kali/.local/lib/python3.10/site-packages/mvt/android/download_apks.py", line 117, in get_packages m.run() File "/home/kali/.local/lib/python3.10/site-packages/mvt/android/modules/adb/packages.py", line 266, in run koodous_lookup(packages_to_lookup) File "/home/kali/.local/lib/python3.10/site-packages/mvt/android/lookups/koodous.py", line 34, in koodous_lookup report = res.json() File "/usr/lib/python3/dist-packages/requests/models.py", line 917, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: [Errno Extra data] 404 page not found : 4

[botherder]

Unfortunately I'm afraid I'm forced to remove Koodous support entirely. Besides the fact that now all API endpoints request authentication, the free account introduces a rate limit of 4 lookups per minute, which would just not be functional for our use case.

[rondinellepqd]

It would be interesting for you to put the option of putting the API key by those who are able to use the tool, that is, when paying for the APIs. Your project is excellent, but without this signature verification option in Koodous and VirusTotal it is disheartening.

Em ter., 14 de jun. de 2022 às 04:57, Nex @.***> escreveu:

Unfortunately I'm afraid I'm forced to remove Koodous support entirely. Besides the fact that now all API endpoints request authentication, the free account introduces a rate limit of 4 lookups per minute https://docs.koodous.com/quotas.html, which would just not be functional for our use case.

— Reply to this email directly, view it on GitHub https://github.com/mvt-project/mvt/issues/273#issuecomment-1154906812, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE366NNYNGMH246FXRHJPMTVPBCQ7ANCNFSM5WPI6OAQ . You are receiving this because you commented.Message ID: @.***>

[botherder]

I am going to add VT API support, but in the case of Koodous even with supporting adding a key, the rate limiting just won't cut the kind of amount of lookups mvt would need to do.

[rondinellepqd]

Excellent my friend. It will help a lot. Thanks!

[botherder]

Turns out, the same problem will apply to VirusTotal. In the new v3 API there doesn't seem to be a bulk detection rate lookup, and by default they also seem to enforce a 4req/min rate. So, in most cases you will almost certainly immediately run out of quota.

This is all rather unfortunate, but it's out of our control.

[VMARTTARIN]

Hi, my friends, i understand, it's updated MVT with VT function? i download the package but i d0n't see changes...

[VMARTTARIN]

In IOS it is not possible to analyze the apps, mvt-ios only analyzes the sms of the ios mobile terminal? STIX files are updated regularly? Can you tell me the latest updated version of this file? Thank you

[azeemh]

finally got mvt to run got the same error.

 ERROR    [mvt.android.lookups.virustotal] Unfortunately VirusTotal lookup is disabled until further notice, due to unresolved issues with the API service.                                                        
         INFO     [mvt.android.lookups.koodous] Looking up all extracted files on Koodous (www.koodous.com)                                                                                                                
         INFO     [mvt.android.lookups.koodous] This might take a while...                                                                                                                                                 
Looking up 140 packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
         ERROR    [mvt.android.modules.adb.packages] Error in running extraction from module Packages: Extra data: line 1 column 5 (char 4)                                                                                
                  Traceback (most recent call last):                                                                                                                                                                       
                    File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 971, in json                                                                                                                    
                      return complexjson.loads(self.text, **kwargs)                                                                                                                                                        
                    File "/usr/local/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads                                                                                                               
                      return _default_decoder.decode(s)                                                                                                                                                                    
                    File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 373, in decode                                                                                                               
                      raise JSONDecodeError("Extra data", s, end, len(s))                                                                                                                                                  
                  simplejson.errors.JSONDecodeError: Extra data: line 1 column 5 - line 2 column 1 (char 4 - 19)                                                                                                           

                  During handling of the above exception, another exception occurred:                                                                                                                                      

                  Traceback (most recent call last):                                                                                                                                                                       
                    File "/usr/local/lib/python3.8/site-packages/mvt/common/module.py", line 152, in run_module                                                                                                            
                      module.run()                                                                                                                                                                                         
                    File "/usr/local/lib/python3.8/site-packages/mvt/android/modules/adb/packages.py", line 266, in run                                                                                                    
                      koodous_lookup(packages_to_lookup)                                                                                                                                                                   
                    File "/usr/local/lib/python3.8/site-packages/mvt/android/lookups/koodous.py", line 34, in koodous_lookup                                                                                               
                      report = res.json()                                                                                                                                                                                  
                    File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 975, in json                                                                                                                    
                      raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)                                                                                                                                                   
                  requests.exceptions.JSONDecodeError: Extra data: line 1 column 5 (char 4)                                                                                                                                
         INFO     [mvt.android.modules.adb.logcat] Running module Logcat...                                                                          

[azeemh]

I just want to know if my phone has been compromised or not and MVT has failed to answer the question or even provide a solution. I tried running for Congress in NY-04 and literally this is quite aggravating as even the tools don't work.

None of the api lookups work so no STIX files are made and thus you can't figure out if your phone was compromised.

all in all this was a huge waste of time.

just buy a new phone and don't use any backup restoration or install whatsapp.

Please let me know how we can fix or if we need funding or something because this is necessary as a public service and i'm sure we could organize public funding to make sure our phones are not compromised by the NSO group.

Normally I'm chill about open source projects but considering the supreme court as of late and our loss of privacy rights stemming from overturning Roe V Wade I really, seriously, cannot stress how important this is -- we need forensic proof that someone is hacking.

Pegasus enables attackers to commit mass FELONY EAVESDROPPING, Intellectual Property and Copyright Violations, and much more...

Please update when the application actually runs completely. I'll help to package and make it mass accessible. We need this. Thanks

[azeemh]

botherder the issue was never solved.

closing it as complete won't help anyone. just leave it open and can we have it support a paid service?

now mvt is a useless tool and Amnesty International can't even use it.