Structure Question: Addressing specific skills, approaches, and broad categories

[joncamfield]

A structural challenge has been cropping up as SAFETAG is receiving more inputs from the community, and I'd like to have an open thread here to map it out.

TL;DR: Add a new structural level to SAFETAG, or add more non-"core"/MVA methods to capture complex, multi-activity/tool assessment processes?

The Minimal Viable Audit work led by @kakron has ended up focused mainly on the "method" level of SAFETAG, which makes sense; the true "MVA" should be a semi-flexible concept that applies to every org, no matter how large or small, no matter the threat actors, so a collection of core methods where an auditor would then select the relevant activities (based on org/context) meets this need.

In parallel, at the activity level, it's hard to get all the way down to the detail level without either cramming multiple tool walkthroughs into one activity, or making essentially duplicative activities to cover each tool (as a side-note, I strongly encourage the activities to give decent instructions that should not be exhaustive, but useful to peer practitioners and able - possibly with additional research/work -- to be followed by any on-staff technologists to verify fixes)

I've been trying to encourage more new activities and fewer methods, but we are straining that limit here with the work that @hackatom has been doing.

I see two(ish) options (note that these shouldn't totally break the current compile-to-PDF model, but should be more focused on the transition into the interactive interface Jun is creating at https://contentascode.github.io/safetag/@safetag/toolkit/

1) Add more Methods -- with the MVA concept, we can tack on non-MVA methods which are still at a methodology level in terms of them having a singular "goal" but multiple potential approaches, some of which would duplicate each other, some of which would support / help verify or deepen understanding of the processes and assets in play. This opens up the framework to include additional collections of more focused activities, see #262, #311 as potential things that could be turned into methods that don't fall into the MVA set.

2) Add a new structural level, or "activity variants". @jmatsushita has raised this idea, and you can also see it in some of the new "remote assessment" activities where there are simple variations the auditor can perform to adapt to different situations. By the same token, "web vuln assessment" could have variations for different tools (burp, OWASP/Wapiti, OpenVAS) with guidance on why to choose different tools.

3) Do nothing, cram everything in one activity, decide on a formatting approach to help track this.

[mfelaco]

This is not a simple issue, @cguerrave and I spent quite a while discussing it. We ended up choosing option 2 over 1: It provides a more elegant structure, and makes it so there isn't a huge list of methods. It also adds a step of complexity to the guide generation process, but it's a small price to pay. We envisioned the workflow for the interactive interface like this:

• Choose methods/categories.
• Choose activities inside those methods.
• Choose activity variants (activity modules might be a better name?) inside each activity selected.

The guidance part on why to choose different tools/modules would only belong in the interactive interface, the activity on the guide would explain what applies to all modules and it would be followed by each module selected.

[poser]

Agreed, assuming this isn't too much of a curveball for Jun.

you can also see it in some of the new "remote assessment" activities where there are simple variations the auditor can perform to adapt to different situations.

Can you send me a link on the toolkit prototype or a pointer so I can find an example of this once I get safetag init to work?

The trick to this sort of thing is making sure that you don't end up longing for a fourth level of abstraction six months down the road, right? We have a pretty clear sense of what belongs in a Method and what belongs in an Activity. The difference between an Activity and an Activity Module would need to be equally intuitive.

Even if we can only enforce these things with a styleguide, we should make it clear that:

• Contributors should try to avoid putting content into an Activity Module that might also be useful in some other Module within the same Activity, and
• An Activity Module should always be the "lowest level" of content.

We should also be clear about what to do if a contributor initially creates only a single variant. Should she just do the whole thing as an Activity? (In other words, is it the job of whoever writes up the second variation to find the appropriate point of abstraction?) Or should she try to figure out which paragraphs belong in which type of content unit?

(Random thought. This might be prematurely optimized for tool-specific Activities — and therefore too brittle to be useful — but...could we merge this notion of the Activity Module with the content that currently lives in the Walkthrough section of some Activities? Then it would at least have a coherent definition and contributors would be more likely to know when they do or do not need to create one.)

[seamustuohy]

Further agreement on #2 over #1 with one caveat. We should be willing to add new modules as needed to appropriately differentiate activities that are added.

We chose the current modules based upon our specific use case and activities. I know there are at least a few "modules" out there that will need to be added once we start including a broader set of activities. (forensics & incident response is just one example)

[hackatom]

I think the "activity variants" mentioned by @joncamfield can help in producing much richer content for each particular modules. One of the struggle I encountered during the documentation is that, putting different activities (burpsuite, FOCA, beef XSS etc) in one single activity.md makes it a bit difficult for auditor like us in the selection of activities, and the single file itself starts to get huge as contributors continue to add more. Not only that, @seamustuohy is correct, #2 may start introducing other great contents which may help auditors adapt to "different situations" (incident response and forensic) making SAFETAG more of a "complete" audit framework.

[seamustuohy]

Ohh, Yes! Now I'm excited.

Back when we had "Approaches" as an equivalent object to what we are calling "activity variants"[1] we used them to split up similar types of content according to the type of skills required for that activity. (i.e. facilitation approaches, technical approaches). The bloat you describe WAS a problem at that point. The tree based structure forced on us by using folders as our main organizing unit made content too hard to find for others. The nesting of method/approaches/->(link)->activity was too confusing for people who tried to explore the content in the repository. But, the meta-data based structure Jun has put in place will remove this obstacle.

It also opens up the opportunity for different activities to fall under different methods. (i.e. A process mapping activity while primarily used for risk modeling can also be a valuable training & awareness raising activity under responsive support.[2])

I wholeheartedly agree with the two points that @poser stated above. Activity content that could apply across all similar activity variants within a method would have to be moved up to the level of "activity variants." An "activity variant" would describe a specific approach that can be accomplished using one or more of the activities underneath it. Activities should be focused on the technique and technical aspects of that activity no matter what context it is used in.

For example:

Method: Recon

Activity Variant: Infrastructural OSINT

• Activities
• Recon-NG
• SpiderFoot

Activity Variant: People OSINT

• Activities
• Twitter Scraping
• Facebook Graph Search
• Harvester

Footnotes

[1] I actually vote that we adopt the name "approaches" instead of "activity variant" since it is already a concept within the SAFETAG framework. It was originally used as a synonym for this purpose. [2] I almost forgot how much time we spent debating and brainstorming the most appropriate name for "responsive support." The reason we chose it is so that we could nest, among other things, training, incident response, forensics, etc. underneath it as needed. So, if we want to avoid adding new modules/categories for a while that name was crafted to allow us to store almost any misc. direct service activities.

[joncamfield]

I like this model, and think when we were originally talking about "variants" that that terminology would live at the bottom-most level; so changing this to Method->Approach->Activity makes the most sense to me. To make sure we're sharing the same scoping here:

Method

Goal oriented, e.g. Recon: "Discover publicly/non-secret available information about an organization"

Approach

Ideally divided among multiple approaches to this information, grouped by general style (technical/scanning/verification, interview/facilitation process with staff, desk research, etc.). For Recon, one could see google dorking, reading through the org's social media and website(s) as "research", and using some of the automated tools as both supporting the research side as well as providing a bit more in-depth verification/exploration. I don't believe we have anything in recon currently that is in the facilitation/interview/interaction category, but one could imagine some additions to early stage conversations helping to scope with an org what is / isn't considered sensitive from their view.

Activity

Specific walkthrough guides where appropriate, or at least skeletal frames (to fit and be visible in the SAFETAG interface/PDF) that point out to a specific external resource (ideally using an archive.org link to protect against resources disappearing).

I am a bit nervous about relying too heavily on external resources to avoid the problem of someone using SAFETAG and then not having the content on hand, on site. This should be managed during the auditor's preparation, but that doesn't always happen. Perhaps a flag for the activity that it is "remote" would be sufficient to buffer against this?

[joncamfield]

To deal with current content before making a more massive change to files/naming/etc., I am proposing the following:

In the Exercise folder, a "normal" activity would simply have the instructions.md file with the complete walk-through. In activities with these more specific walkthroughs, we will include them from the instructions file. This will ensure they are visible in the current PDF creation process and can hopefully provide linkage/metadata in the Content As Code migration process (I imagine that will need some tweaks to work.

The instructions (and the per-activity approach.md) should also provide guidance to how a practitioner would choose which to run.