Abstract relationships as top-level constructs rather than embedded within other constructs

terrymacdonald commented 9 years ago

The current way relationships are conveyed (only in objects) has two issues.

Firstly, is that I cannot currently 'agree' with a relationship that someone has asserted between objects they have published. In some cases it is extremely valuable for someone to say they also think that object A and Object B are related. Currently I would need to produce my own object and assert the relationship from my own object to that object. Its for this reason that I would like to propose that Relationships should be able to be sent independently of the objects they refer to. In that case a third-party will be able to send their agreement that a relationship is valid. A subsequent consumer can then easily see that 14 others agree the suggested relationship is valid and only 2 say its not - therefore the relationship is more likely to be valid.

e.g. mycompany agrees completely with the assertion that thirdparty made that there is a relationship between the two objects that thirdparty published.

<stix:Relationships>
    <stix:Relationship id="mycompany:relationship-2c560b47-7b11-48a1-bcda-968dfbe0ceba" timestamp="2015-03-11T01:01:01+00:00" xsi:type='relationship:RelationshipType'>
        <stixCommon:Confidence>
            <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
        </stixCommon:Confidence>
        <stixCommon:Information_Source>
            <stixCommon:Identity id="mycompany:Identity-50790476-b684-11e3-9149-0800271e87d2">
                <stixCommon:Name>MyCompany, Inc.</stixCommon:Name>
            </stixCommon:Identity>
        </stixCommon:Information_Source>
        <relationship:Object>
            <indicator:Indicator id="thirdparty:Indicator-33fe3b22-0201-47cf-85d0-97c02164528d" xsi:type='indicator:IndicatorType' timestamp="2015-02-20T09:00:00.000000Z">
        </relationship:Object>
        <stixCommon:Relationship xsi:type="stixVocabs:AssertionAgreementVocab-1.0">Agrees</stixCommon:Relationship>
        <relationship:Related_Object>
            <indicator:Indicator id="thirdparty:Indicator-8cc8061e-2bed-496b-882e-2c4ad1b30f23" xsi:type='indicator:IndicatorType' timestamp="2015-02-27T13:00:00.000000Z">
        </relationship:Related_Object>
    </stix:Relationship>

^ That's effectively a [+1] from mycompany!.

The second use of the Top-level relationship object is to agree with the object that someone else sent. Note this is different from agreeing with the relationship between objects (the edges) that the previous example was describing. This use of the relationship object is agreeing with the object itself (the nodes in knowledge-graph-speak).

e.g. mycompany agrees completely that the assertion that thirdparty made in their info is correct.

<stix:Relationships>
    <stix:Relationship id="mycompany:relationship-2c560b47-7b11-48a1-bcda-968dfbe0ceba" timestamp="2015-03-11T01:01:01+00:00" xsi:type='relationship:RelationshipType'>
        <stixCommon:Confidence>
            <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
        </stixCommon:Confidence>
        <stixCommon:Information_Source>
            <stixCommon:Identity id="mycompany:Identity-50790476-b684-11e3-9149-0800271e87d2">
                <stixCommon:Name>MyCompany, Inc.</stixCommon:Name>
            </stixCommon:Identity>
        </stixCommon:Information_Source>
        <relationship:Object>
            <indicator:Indicator id="thirdparty:Indicator-33fe3b22-0201-47cf-85d0-97c02164528d" xsi:type='indicator:IndicatorType' timestamp="2015-02-20T09:00:00.000000Z">
        </relationship:Object>
        <stixCommon:Relationship xsi:type="stixVocabs:AssertionAgreementVocab-1.0">Agrees</stixCommon:Relationship>
    </stix:Relationship>
</stix:Relationships>

I think we need this functionality to crowdsource agreement from communities.

johnwunder commented 9 years ago

This is a duplicate of #70 but given that this write-up is better I'll close that one.

jordan2175 commented 9 years ago

I agree completely with Terry. This could add so many needed and interesting ways of expressing data. It goes without saying..... 1) The ID field or IDREF field here should be required 2) There should be NO inline content in these, just like the report object. 3) These should be first-class citizens and you should be able to send them by themselves in a STIX Package

terrymacdonald commented 9 years ago

oops. wrong button.

MarkDavidson commented 9 years ago

Here's a notional structure:

   <stix:Relationship 
                      id="example:Relationship-some-uuid-5555"
                      from_idref="example:Indicator-some-uuid-1234" 
                      to_idref="example:Campaign-some-other-uuid-0987">
       <stix:Relationship_Type>indicates</stix:Relationship_Type> <!-- Could be a [1,n] construct to permit multiple relationships -->
       <stix:Confidence>...</stix:Confidence>
       <stix:Information_Source>...</stix:Information_Source>
       <!-- Is other stuff necessary? -->
   </stix:Relationship>

I think there are some positives here:

There are currently a bunch (17? It's been a while since I looked) of locations in the schema where relationships can exist. This notional structure could help reduce that to a single location (provided it's a top level construct) and would hopefully simplify processing.
In STIX 1.1.1, modifications to relationships within the language require structural changes to the schema (e.g., adding/removing elements/attributes). With this notional structure, adding/removing relationships could be as straightforward as updating a controlled vocabulary (this is still a schema change, but it's not structural).
Relationships can be made without modification to the objects they relate. In STIX 1.1.1, adding a "Related_Campaign" (for instance) requires modifying the Indicator itself (aka versioning). A top level relationship construct would permit relationship modifications that don't require any modification to the related objects (e.g., Indicators).
My sense is that this can be implemented in a backward compatible manner, provided all existing relationships could be mapped into whatever form the "new" construct might take.

I think this idea hits all requirements from @jordan2175.

As for @terrymacdonald, you have an interesting problem statement:

[...] I cannot currently 'agree' with a relationship [...]

What about a "relationship assertion" where you specify the relationship you want to make an assertion about, plus an assertion? Something like

<stix:Relationship_Assertion
       id="example:assertion-uuid-9999"
       idref="example:Relationship-some-uuid-5555">
   <stix:Assertion_Type>+1</stix:Assertion_Type>
   <stix:Confidence>...</stix:Confidence>
   <stix:Information_Source>...</stix:Information_Source>
   <!-- Is other stuff necessary? -->
</stix:Relationship_Assertion>

I also toyed with the idea that a Relationship Assertion would just be a special type of relationship where the from_idref would be required (the idref refers to the relationship you are making the assertion about), the to_idref would be prohibited, and the Relationship_Type would contain the assertion. I wasn't sure which was better, but I wanted to concretely put a single one forward.

Thoughts?

-Mark

jordan2175 commented 9 years ago

I really like those ideas @MarkDavidson, brilliant. The +1 or -1 also GREATLY simplifies one of the major long term use cases. This would be so easy to implement in a Facebook/Instagram like APP for STIX. I also like the idea of getting rid of all of the other relationship gunk all throughout SITX. This would greatly clean up the data model and speed up processing and code logic. @terrymacdonald thanks for push for this. This is really going to make things better.

terrymacdonald commented 9 years ago

Yeah I like that a lot Mark. We need to do a lot of thinking about having a separate Relationship_Assertion object though. I went with the combined Relationship_Assertion and Relationship object, mainly because it would simplify the concept for new people learning the model. The idea was that there was one object to send relationships, and the rest are the 'data nodes'.

That said, on second thoughts there is merit in having a separate Assertion object....

Its not just relationships we want people to be able to assert. Its also the data in an object. We want others to be able to say yes that object is right, but the relationships with this other object is wrong. So how about making the relationship_assertion object into a generic assertion object? e.g.

<stix:Assertion
        id="example:assertion-uuid-9999"
        idref="example:Relationship-some-uuid-5555">
    <stix:Assertion_Type>+1</stix:Assertion_Type>
    <stix:Confidence>...</stix:Confidence>
    <stix:Information_Source>...</stix:Information_Source>
    <!-- Is other stuff necessary? -->
 </stix:Assertion>

That would cover the relationship case, but also the data node case.

Comments? Cheers Terry MacDonald

jordan2175 commented 9 years ago

Very interesting.... And I like where this is going. This would greatly simplify a lot of weirdness we currently have and allow much more interesting use cases to be solved.

MarkDavidson commented 9 years ago

@terrymacdonald, I'd like to discuss one of your paragraphs:

Its not just relationships we want people to be able to assert. Its also the data in an object. We want others to be able to say yes that object is right, but the relationships with this other object is wrong. So how about making the relationship_assertion object into a generic assertion object?

If we operate under the assumption that a relationship is a first class object, then a relationship wouldn't count as data in an object. You could disagree with a relationship using the assertion construct from above.

However, I think there is still a need to make assertions about data in objects. I reflected for a bit on what assertions you might want to make about data in an object, and I came to the conclusion that an assertion about data in an object is probably a sighting. (e.g., "I've seen that IP also!") Said another way, a sighting is a type of assertion.

Building on the assertion idea, there is certainly overlap across relationship/assertion concepts. I can think of three things that are relevant to this discussion:

Relationship - Assert that two objects have a specified relationship
Assertion - Convey an "agree", "disagree", "false positive" or similar assertion (sorry, circular definition) about an object
Sighting - Assert that an object has been seen by someone who didn't create the object.

I got tired of typing the XML, so I just went with a list of properties to discuss a notional assertion object (it's python color-coded, but don't read too much into that)

class Assertion:
    id_ # The ID of the assertion itself
    from_idref  # The ID of the "from" object
    to_idref  # The ID of the "to" object
    assertion_type  # The type of assertion: Relationship, Sighting, Assertion
    assertion  # The assertion being made: Indicates, +1, Disagree
    information_source  # Info source of this assertion
    confidence  # A confidence assertion of this assertion_type / assertion
    # Is other information needed? Rationale for the assertion might be useful

A relationship could look like this:

Assertion(
      id_ = 'Assertion-1',
      from_idref='Indicator-1',
      to_idref='Campaign-1',
      assertion_type='Relationship'
      assertion='Indicates'
      information_source='Threat Analyst 1',
      confidence='High',
      rationale='I saw this during my analysis')

A sighting could look like this. Note that sighting specific components of an object would get more complex. A possible tearline here is that you can only "sight" things that have IDs in order to keep the Sighting Assertion simple.

Assertion(
      id_ = 'Assertion-2',
      from_idref='Indicator-1',
      # to_idref  # Not populated in this scenario
      assertion_type='Sighting'
      # assertion  # May or may not need to be populated.
      information_source='Threat Analyst 2',
      # confidence # Is confidence irrelevant for a sighting?
      rationale='This was seen by our IDS'
     )

Lastly, an assertion could look like this:

Assertion(
      id_ = 'Assertion-3',
      from_idref='Assertion-1',
      # to_idref  # Not used for assertion_type == Assertion
      assertion_type='Assertion'
      assertion='+1'
      information_source='Threat Analyst 3',
      confidence='High',
      rationale='I confirmed the relationship during independent analysis'
     )

I think there's a potential discussion point for splitting Relationships (which have both 'from' and 'to' properties) and Assertions (which would only have a from property), but I'll end my message here.

-Mark

jordan2175 commented 9 years ago

Lets figure out 10-20 use cases and make sure this solves all the problems. The idea of doing relationships here and not inside of data objects, is very attractive.

terrymacdonald commented 9 years ago

Hi Mark,

Thanks for this. I'm not a fan of conflating these together. I believe all we will be doing is making it more difficult for people to understand. Technically we may be able to combine these all together, but we've seen the confusing that can cause before (Observable patterns and instances).

When it comes down to it, the entire STIX package we generate is an assertion. We could probably force all objects into the Assertion object if we wanted, but all that would do is confuse people.

I prefer to keep them separated (https://www.youtube.com/watch?v=XN32lLUOBzQ ).

I believe the three concepts we are trying to allow STIX to convey from producer to consumer are:

"These two things are related"
"I agree"/"I do not agree"
"I've seen that too"

In order to keep this simple to understand for new players, I still maintain we need to keep the separation of the above concepts into individual objects.

As mentioned on the STIX mailing list I believe the tweaked Sighting Object I've proposed here (https://github.com/STIXProject/schemas/issues/306) is NOT just "I've seen that too" (number 3), but is instead intended to simply convey "I've seen this interesting thing". This is a subtle but important difference. We need to separate the "stuff I've seen" from the "stuff I'm looking for", and recognize that the "stuff I've seen" is a superset of "I've seen that too".

So to describe each concept individually and how I propose we tackle them...

1. "These two things are related"

I believe that this should still be described using the relationship object Mark first proposed. It is simple, is easy to understand and works.

<stix:Relationship id="example:Relationship-some-uuid-5555" from_idref="example:Indicator-some-uuid-1234" to_idref="example:Campaign-some-other-uuid-0987"> stix:Relationship_Typeindicates/stix:Relationship_Type stix:Confidence.../stix:Confidence stix:Information_Source.../stix:Information_Source

/stix:Relationship

2. "I agree"/"I do not agree"

Assertion might not be the right word for the object we need to use here. Maybe attestation is a better word? Or Opinion?

After reading your text I have to disagree that the Assertion object (I'll keep calling it that) is likely only related to Sightings. I also see people sending out [+1]'s for Indicators (saying its a good or bad pattern) and potentially Courses of Action as well. So there is at least worth in having the ability to [+1] a node.

Here's Mark's object with the Attestation name instead (just because I like it)

<stix:Attestation id="example:assertion-uuid-9999" idref="example:Indicator-some-uuid-5555"> stix:Attestation_Typeaggree/stix:Attestation_Type stix:Confidence.../stix:Confidence stix:Information_Source.../stix:Information_Source

/stix:Attestation

3. "I've seen that too"

I think that this could be covered within the tweaked Sighting Object that I proposed I've proposed here ( https://github.com/STIXProject/schemas/issues/306). IMHO we need to separate the "stuff I've seen" (Sightings) from the "stuff I'm looking for" (Indicators). "I've seen that too" is covered by the 'stuff I've seen'' functionality that I think should be covered by the Sightings object.

The "stuff I've seen" idea can be conveyed with something like this. In this example we have detected an instance of something that we think is worth capturing for later. We can add it to our knowledge base, but it doesn't necessarily need to have any relationships at present, as we may be able to form them in the future when get more information. At this stage, we are just recording it because its a bit odd:

<stix:Sighting id="example:Sighting-0756a255-6623-4226-8356-015396918b38" timestamp="2014-08-22T20:17:38.959000+00:00" xsi:type='Sighting:SightingType' negate="false" version="2.1.1"> Sighting:Title192.168.1.1/Sighting:Title Sighting:Detected_Time2014-01-31T12:48:50Z/Sighting:Detected_Time

IPv4 Address/Sighting:Type 192.168.1.1/AddressObj:Address_Value /cybox:Properties /cybox:Object /Sighting:Observable /stix:Sighting The "I've seen that too" can be covered by this SIghting object as well. In this example we have detected an instance of a pattern that another Org shared through an Indicator at an earlier time: Sighting:Title192.168.1.1/Sighting:Title Sighting:Detected_Time2014-01-31T12:48:50Z/Sighting:Detected_Time IPv4 Address/Sighting:Type 192.168.1.1/AddressObj:Address_Value /cybox:Properties /cybox:Object /Sighting:Observable Sighting:Related_Indicators /Sighting:Related_Indicators /stix:Sighting I believe that they three objects as described above cover the scenarios that I've described, and give us the flexibility to cover any scenarios I can think of. It keeps things simple to understand which will help when new players come onboard, and won't require huge amounts of documentation to describe. Thoughts? Cheers Terry MacDonald On 7 April 2015 at 22:32, MarkDavidson notifications@github.com wrote: > @terrymacdonald https://github.com/terrymacdonald, I'd like to discuss > one of your paragraphs: > > Its not just relationships we want people to be able to assert. Its also > the data in an object. We want others to be able to say yes that object is > right, but the relationships with this other object is wrong. So how about > making the relationship_assertion object into a generic assertion object? > > If we operate under the assumption that a relationship is a first class > object, then a relationship wouldn't count as data in an object. You could > disagree with a relationship using the assertion construct from above. > > However, I think there is still a need to make assertions about data in > objects. I reflected for a bit on what assertions you might want to make > about data in an object, and I came to the conclusion that an assertion > about data in an object is probably a sighting. (e.g., "I've seen that IP > also!") Said another way, a sighting is a type of assertion. > > Building on the assertion idea, there is certainly overlap across > relationship/assertion concepts. I can think of three things that are > relevant to this discussion: > - Relationship - Assert that two objects have a specified relationship > - Assertion - Convey an "agree", "disagree", "false positive" or > similar assertion (sorry, circular definition) about an object > - Sighting - Assert that an object has been seen by someone who didn't > create the object. > > I got tired of typing the XML, so I just went with a list of properties to > discuss a notional assertion object (it's python color-coded, but don't > read too much into that) > > class Assertion: > id_ # The ID of the assertion itself > from_idref # The ID of the "from" object > to_idref # The ID of the "to" object > assertion_type # The type of assertion: Relationship, Sighting, Assertion > assertion # The assertion being made: Indicates, +1, Disagree > information_source # Info source of this assertion > confidence # A confidence assertion of this assertion_type / assertion > # Is other information needed? Rationale for the assertion might be useful > > A _relationship_ could look like this: > > Assertion( > id_ = 'Assertion-1', > from_idref='Indicator-1', > to_idref='Campaign-1', > assertion_type='Relationship' > assertion='Indicates' > information_source='Threat Analyst 1', > confidence='High', > rationale='I saw this during my analysis') > > A _sighting_ could look like this. Note that sighting specific components > of an object would get more complex. A possible tearline here is that you > can only "sight" things that have IDs in order to keep the Sighting > Assertion simple. > > Assertion( > id_ = 'Assertion-2', > from_idref='Indicator-1', > # to_idref # Not populated in this scenario > assertion_type='Sighting' > # assertion # May or may not need to be populated. > information_source='Threat Analyst 2', > # confidence # Is confidence irrelevant for a sighting? > rationale='This was seen by our IDS' > ) > > Lastly, an _assertion_ could look like this: > > Assertion( > id_ = 'Assertion-3', > from_idref='Assertion-1', > # to_idref # Not used for assertion_type == Assertion > assertion_type='Assertion' > assertion='+1' > information_source='Threat Analyst 3', > confidence='High', > rationale='I confirmed the relationship during independent analysis' > ) > > I think there's a potential discussion point for splitting Relationships > (which have both 'from' and 'to' properties) and Assertions (which would > only have a from property), but I'll end my message here. > > -Mark > > — > Reply to this email directly or view it on GitHub > https://github.com/STIXProject/schemas/issues/291#issuecomment-90533478.

jordan2175 commented 9 years ago

For the I have seen this too, I am thinking that it would be best to reference the original ID instead of creating a new ID.

athiasjerome commented 8 years ago

I agree with Terry (KISS)

athiasjerome commented 8 years ago

An --example-- of Relationship data model could be found in http://csrc.nist.gov/publications/nistir/ir7693/NISTIR-7693.pdf 6.4.1 Relationship Data Model (Page 21)

You could note, again just for example, the 'scope' (inclusive/exclusive)

athiasjerome commented 8 years ago

An interesting (easy to understand) description of the Relationship construct could be found in the "Entity-Relationship" chapter here http://www.adaptiveobjectmodel.com/WICSA3/ArchitectureOfAOMsWICSA3.pdf

athiasjerome commented 8 years ago

I suggest to name this new construct RelationshipStatement this will allow use of algorithms to handle multiple statements regarding a relationship

STIXProject / schemas

Abstract relationships as top-level constructs rather than embedded within other constructs #291