API shows a different set of references than what is on the website

mmulich commented 1 year ago

Describe the bug I'm seeing some actor data differences when using the website vs the API.

For example, the beijing_group has 4 references in the website.

However, when using the API, I see 3 url references. For posterity, here is the api response using the malpediaclient library. See the difference in .meta.refs.

{
  "families": {},
  "meta": {
    "attribution-confidence": "50",
    "cfr-suspected-state-sponsor": "China",
    "cfr-suspected-victims": [
      "United States",
      "Canada",
      "United Kingdom",
      "Switzerland",
      "Hong Kong",
      "Australia",
      "India",
      "Taiwan",
      "China",
      "Denmark"
    ],
    "cfr-target-category": [
      "Private sector",
      "Civil society"
    ],
    "cfr-type-of-incident": "Espionage",
    "country": "CN",
    "refs": [
      "https://www.cfr.org/interactive/cyber-operations/sneaky-panda",
      "https://www-west.symantec.com/content/dam/symantec/docs/security-center/white-papers/elderwood-project-12-en.pdf",
      "https://attack.mitre.org/groups/G0066/"
    ],
    "synonyms": [
      "SNEAKY PANDA",
      "Elderwood",
      "Elderwood Gang",
      "SIG22",
      "G0066"
    ]
  },
  "related": [
    {
      "dest-uuid": "03506554-5f37-4f8f-9ce4-0e9f01a1b484",
      "tags": [
        "estimative-language:likelihood-probability=\"likely\""
      ],
      "type": "similar"
    }
  ],
  "uuid": "da754aeb-a86d-4874-b388-d1d2028a56be",
  "value": "Beijing Group"
}

Furthermore, only 2 of the 4 reference links are the same. So not just missing one, but also different.

To Reproduce Steps to reproduce the behavior:

Find an actor in the site and view their details
Lookup that same actor via the API
Compare the differences between the "References" shown in the site and the "refs" in response data of the API
See differences

Expected behavior I'd have expected to see the same list of links in both the website and API.

I'm also aware that using /api/get/bib/actor/<actor_id> returns the 4 references in bibtex format.

mmulich commented 1 year ago

Also, somewhat related but tangential to the main issue here... The https://www-west.symantec.com/content/dam/symantec/docs/security-center/white-papers/elderwood-project-12-en.pdf url in the API response does not match with any known entries in the malpedia bibtex data.

mmulich commented 1 year ago

Another example is the apk.daam family. There are three BibTeX CiteKey references in the data under library_entries. However, in the bibliographic data there are only two entries.

I'm using the apk.daam example here, but there are many others (e.g. win.appleseed, osx.bella, jar.strrat, elf.wellmess, and the list goes on). Only trying to illustrate that it is not a problem on just a few of the items.

family API response

GET /api/get/family/apk.daam responds with:

{
    "alt_names": [
        "BouldSpy"
    ],
    "attribution": [],
    "common_name": "DAAM",
    "description": "According to PCrisk, DAAM is an Android malware utilized to gain unauthorized access to targeted devices since 2021. With the DAAM Android botnet, threat actors can bind harmful code with a genuine application using its APK binding service.\r\n\r\nLookout refers to this malware as BouldSpy and assesses with medium confidence that this Android surveillance tool is used by the Law Enforcement Command of the Islamic Republic of Iran (FARAJA).",
    "library_entries": [
        "cyble:20230420:daam:8b46773",
        "labs:20230420:daam:0cdd5ef",
        "schmittle:20230427:lookout:3956976"
    ],
    "notes": [],
    "sources": [],
    "updated": "2023-05-30",
    "urls": [
        "https://blog.cyble.com/2023/04/20/daam-android-botnet-being-distributed-through-trojanized-applications/",
        "https://www.lookout.com/blog/iranian-spyware-bouldspy"
    ],
    "uuid": "37a3b62e-99da-47d7-81fb-78f745427b16"
}

family bib API response

GET /api/get/bib/family/apk.daam responds with:

@online{schmittle:20230427:lookout:3956976,
   author = {Kyle Schmittle and Alemdar Islamoglu and Paul Shunk and Justin Albrecht},
   title = {{Lookout Discovers Android Spyware Tied to Iranian Police Targeting Minorities: BouldSpy}},
   date = {2023-04-27},
   organization = {Lookout},
   url = {https://www.lookout.com/blog/iranian-spyware-bouldspy},
   language = {English},
   urldate = {2023-05-30}
}

@online{cyble:20230420:daam:8b46773,
   author = {Cyble},
   title = {{DAAM Android Botnet being distributed through Trojanized Applications}},
   date = {2023-04-20},
   organization = {Cybleinc},
   url = {https://blog.cyble.com/2023/04/20/daam-android-botnet-being-distributed-through-trojanized-applications/},
   language = {English},
   urldate = {2023-05-10}
}

danielplohmann commented 11 months ago

Hi!

Thanks for the comments!

We had a look to figure out what's going on here.

Also, somewhat related but tangential to the main issue here... The https://www-west.symantec.com/content/dam/symantec/docs/security-center/white-papers/elderwood-project-12-en.pdf url in the API response does not match with any known entries in the malpedia bibtex data.

This specific entry is "problematic" as it uses URL that is invalid in the opinion of several URL verifiers (one of which we use internally) so it was added manually way back then and is not compatible with the automations we did afterwards. It's also contained in the MISP threat actor galaxy cluster, where it should possibly be removed since it's a dead link anyway...

Another example is the apk.daam family. There are three BibTeX CiteKey references in the data under library_entries. However, in the bibliographic data there are only two entries.

I'm using the apk.daam example here, but there are many others (e.g. win.appleseed, osx.bella, jar.strrat, elf.wellmess, and the list goes on). Only trying to illustrate that it is not a problem on just a few of the items.

family API response

GET /api/get/family/apk.daam responds with:

{
    "alt_names": [
        "BouldSpy"
    ],
    "attribution": [],
    "common_name": "DAAM",
    "description": "According to PCrisk, DAAM is an Android malware utilized to gain unauthorized access to targeted devices since 2021. With the DAAM Android botnet, threat actors can bind harmful code with a genuine application using its APK binding service.\r\n\r\nLookout refers to this malware as BouldSpy and assesses with medium confidence that this Android surveillance tool is used by the Law Enforcement Command of the Islamic Republic of Iran (FARAJA).",
    "library_entries": [
        "cyble:20230420:daam:8b46773",
        "labs:20230420:daam:0cdd5ef",
        "schmittle:20230427:lookout:3956976"
    ],
    "notes": [],
    "sources": [],
    "updated": "2023-05-30",
    "urls": [
        "https://blog.cyble.com/2023/04/20/daam-android-botnet-being-distributed-through-trojanized-applications/",
        "https://www.lookout.com/blog/iranian-spyware-bouldspy"
    ],
    "uuid": "37a3b62e-99da-47d7-81fb-78f745427b16"
}

family bib API response

GET /api/get/bib/family/apk.daam responds with:

@online{schmittle:20230427:lookout:3956976,
   author = {Kyle Schmittle and Alemdar Islamoglu and Paul Shunk and Justin Albrecht},
   title = {{Lookout Discovers Android Spyware Tied to Iranian Police Targeting Minorities: BouldSpy}},
   date = {2023-04-27},
   organization = {Lookout},
   url = {https://www.lookout.com/blog/iranian-spyware-bouldspy},
   language = {English},
   urldate = {2023-05-30}
}

@online{cyble:20230420:daam:8b46773,
   author = {Cyble},
   title = {{DAAM Android Botnet being distributed through Trojanized Applications}},
   date = {2023-04-20},
   organization = {Cybleinc},
   url = {https://blog.cyble.com/2023/04/20/daam-android-botnet-being-distributed-through-trojanized-applications/},
   language = {English},
   urldate = {2023-05-10}
}

So for these artefacts, it looks like when we created the library and later the automated generation of the bibtex file, we didn't address that the shortlinks are updated/removed in the individual JSON files whenever library entries are edited. We'll have to do a cross-validation of the DB vs. JSON files at some point in the future and with that it should be fixed. Can't give an estimate when that's gonna happen as Malpedia is just a side project for us and we are currently busy with regular projects in the institute but we will soon have a student assistant join our efforts so there's hope that the site will get some attention and some of the issues addressed.

malpedia / feedback