Update Conversion from Mapping.xlsx to Mapping-v15.1.json files

[allan-korol]

I would suggest that the name, description, url, etc would probably be best suited directly under the mitigation, with a 'list' of techniques it addresses. In the proposed form, it implies that given a mitigation, each technique would have a different name, description, etc for the mitigation.

{
    "M1234": {
        "name": "Software Configuration",
        "description": "Some description goes here.",
        "source": "MITRE",
        "techniques": {
            "T1234.001": {
                "use": "some use"
            },
            "T1234.002": {
                "use": "some use"
            },
        }
    },
    "ISM-1122": {
        "name": "Software Configuration",
        "description": "Some description goes here.",
        "source": "ISM",
        "techniques": [
            "T1223.001",
            "T4545.002",
            "T4326.001"
        ]
    }
}

This method would simplify the loading of data from the mappings spreadsheet as you will be able to load all the mitigations and their descriptions/details, and then enumerate the Technique sheet and simply find and attach the technique to the list for the mitigations' techniques field.

[glowbase]

Table maps Windows Event ID's to MITRE Techniques.

What would be the best way to implement this into the current working format for mappings-v15.1.json @allan-korol?

[allan-korol]

@glowbase, this is probably going to be extensive information but may tie into evidence locations and tools and really start to tie everything together.

My initial thought would be something like an "evidence source" (eg file and it's location, OS, etc).

We could then represent an "artefact" that is linked to an "evidence source" (and probably a tool), along with some extra details (where the event id would be represented.

Might even consider a specific "event" table (as opposed to an "artefact" table), but I think the important part would be to link to the source, technique, and tool.

I would be more inclined to explore an "event" table at this point, and consider merging into a generic "artefact" table later if appropriate.

[allan-korol]

@glowbase some mitigations within the newly generated mappings-v15.1json file have a null description (AC-13, AC-15). I assume this is from the xlsx file and not the import script. Unsure if this was intended or not and whether it should be allowed.

To continue with the DB import I have removed the NOT-NULL constraint from the description column in the database, but if they should have a description, it would be better to fix the data and add the constraint back in, to detect potential data issues like this (if it is an issue).

[glowbase]

@allan-korol , surprisingly the NIST documentation doesn't have a description for those two controls. I think the plan is to just remove those two controls from being used.

[allan-korol]

@glowbase the initial description of the issue discussed a new format and was extended for the uses.

I noticed that the actually convert_new.py script was bit of both, it was an array of objects with the technique as the key, and an object of use. I will adjust the script so it's in line with the first. An object of techniques, with the key being the Technique ID, and the value being an object of the use.

Current:

{
    "M1234": {
        "name": "Software Configuration",
        "description": "Some description goes here.",
        "source": "MITRE",
        "techniques": [
            {
               "T1234.001": {
                     "use": "some use"
               }
            },
            {
               "T1234.002": {
                     "use": "some use"
                }
            }
        ]
    }
}

Updated:

{
    "M1234": {
        "name": "Software Configuration",
        "description": "Some description goes here.",
        "source": "MITRE",
        "techniques": {
            "T1234.001": {
                "use": "some use"
            },
            "T1234.002": {
                "use": "some use"
            }
        }
    }
}

[allan-korol]

I recommend closing this issue now the new mapping is imported with PR #15 and opening new ones for things like evidence, artefacts, events, etc.

Should also remove the old convert.py as well.