search

StephenOTT / STIX-Java

STIX 2.x Java Library
MIT License
27 stars 13 forks source link

Add Cyber Observables and related support #69

Closed anlklsim closed 5 years ago

anlklsim commented 5 years ago

added some more cyber observable objects and other fixes/changes

StephenOTT commented 5 years ago

@anlklsim What is the reasons for implementing all your cyber observables with @Value.Immutable pattern?

I would also like to explore creating custom annotations instead of the RegEx/Pattern. The Pattern is really unclear of its usage and its very messy.

Lets review what we think should be actually enforced for many of the values. You have @Pattern implemented in many places, but need to do a sanity check on if it should actually be enforced or left to the implementor to do the double check on that data. Binary data is a good example I think. It calls for Base64 but do we want to actually enforce this? Or do we want to accept the strings, and let someone enforce it at the implementation side if they choose.

StephenOTT commented 5 years ago

@anlklsim for NetworkSocketExtensionExt you have a enum such as:

    @JsonTypeName("network-socket-address-family-enum")
    public enum AddressFamily {
        AF_UNSPEC,
        AF_INET,
        AF_IPX,
        AF_APPLETALK,
        AF_NETBIOS,
        AF_INET_6,
        AF_IRDA,
        AF_BTH;
    }

What is the purpose of having the JsonTypeName annotation? How would it get used?

StephenOTT commented 5 years ago

@anlklsim take a look at the update so far. I have made some refactoring changes, and fixed a bunch of configs to be inline with rest of the project. Also see the Extensions section that was updated to match the spec.

I removed the JsonSubTypes annotations as this is covered by the StixParsers class to ensure easy runtime extendability without the need to recompile.

See the ProcessCooSpec test for a working example:

generated json:

{
  "type": "bundle",
  "id": "bundle--644342ee-690d-4161-8970-b1996fe7a1dd",
  "spec_version": "2.0",
  "objects": [
    {
      "type": "observed-data",
      "id": "observed-data--4ff75d5e-029b-4c61-a181-030597a55c80",
      "created": "2019-02-03T18:45:36.035Z",
      "modified": "2019-02-03T18:45:36.035Z",
      "revoked": false,
      "first_observed": "2019-02-03T18:45:36.013Z",
      "last_observed": "2019-02-03T18:45:36.028Z",
      "number_observed": 1,
      "objects": [
        {
          "type": "process",
          "extensions": {
            "windows-process-ext": {
              "aslr_enabled": true,
              "window_title": "some title"
            }
          },
          "is_hidden": true,
          "name": "Dog"
        }
      ]
    }
  ]
}

and then parsed back into a bundle BundleObject{type=bundle, id=bundle--644342ee-690d-4161-8970-b1996fe7a1dd, specVersion=2.0, objects=[ObservedData{firstObserved=2019-02-03T18:45:36.013Z, lastObserved=2019-02-03T18:45:36.028Z, numberObserved=1, objects=[Process{isHidden=true, name=Dog, arguments=[], environmentVariables={}, gpenedConnectionRefs=[], childRefs=[], type=process, extensions=[WindowsProcessExtension{aslrEnabled=true, windowTitle=some title, startupInfo={}, type=windows-process-ext, customProperties={}}], customProperties={}}], relationships=[], hydrated=true, type=observed-data, id=observed-data--4ff75d5e-029b-4c61-a181-030597a55c80, created=2019-02-03T18:45:36.035Z, externalReferences=[], objectMarkingRefs=[], granularMarkings=[], customProperties={}, labels=[], modified=2019-02-03T18:45:36.035Z, revoked=false}]}

More updates to come, but you should have enough to write a few more unit tests to test out compliance with spec.

anlklsim commented 5 years ago

@anlklsim for NetworkSocketExtensionExt you have a enum such as:

  @JsonTypeName("network-socket-address-family-enum")
  public enum AddressFamily {
      AF_UNSPEC,
      AF_INET,
      AF_IPX,
      AF_APPLETALK,
      AF_NETBIOS,
      AF_INET_6,
      AF_IRDA,
      AF_BTH;
  }

What is the purpose of having the JsonTypeName annotation? How would it get used?

I was using it since it was named that in the spec - I did not think it needed to be serialized, so probably can just keep it as a string or even just a comment, so it's documented.

StephenOTT commented 5 years ago
StephenOTT commented 5 years ago

@anlklsim See latest updates. I have added support for bean validation of the Extensions applied to COOs.

        ProcessCoo process = Process.builder()
                .isHidden(true)
                .name("Dog")
                .addExtensions(
                    WindowsProcessExtension.builder()
                        .aslrEnabled(true)
                        .windowTitle("some title")
                        .build())
                .build()
...
@AllowedParents({ProcessCoo.class})
public interface WindowsProcessExtensionExt extends CyberObservableExtension {
...
}

With a extension you define the interfaces that the extension is allowed to be used from.

So when a COO has a extension applied it will validate it.

I went with this model because I think it supports the most fluid add-on support for Custom Extensions. Allowing someone to build new extensions and add them to classpath without having to rebuild or modify the core SITX lib.

I will look at adding the same model support for SRO relationships and other Custom Object support once we determine if its a viable pattern.

(Annotation naming could use some work... so if you have better ideas, please share!)

StephenOTT commented 5 years ago

@anlklsim I am looking at create a "BusinessRule" Annotation to handle the various conditional field combinations found in COO and Extensions (and possibly used in other object types)

Would like your thoughts.

Imagine this style of function on a single interface. Where values like "some_field" and "some1", etc are the JsonProperty annotation values.

@BusinessRule(if = “some_field”, is = PRESENT, then = {“some1”, “some11”}, require = ALL)
@BusinessRule(if = “some_field”, is = PRESENT, then = {“some2”, “some3”}, require = MIN1)
@BusinessRule(if = “some_field123”, is = EMPTY, then = {“some5”, “some6”}, require = MIN1)

Any better usage or style thoughts? I was looking at using Spring Expression Lang but its missing support for optionals.

anlklsim commented 5 years ago

@anlklsim See latest updates. I have added support for bean validation of the Extensions applied to COOs.

        ProcessCoo process = Process.builder()
                .isHidden(true)
                .name("Dog")
                .addExtensions(
                    WindowsProcessExtension.builder()
                        .aslrEnabled(true)
                        .windowTitle("some title")
                        .build())
                .build()
...
@AllowedParents({ProcessCoo.class})
public interface WindowsProcessExtensionExt extends CyberObservableExtension {
...
}

With a extension you define the interfaces that the extension is allowed to be used from.

So when a COO has a extension applied it will validate it.

I went with this model because I think it supports the most fluid add-on support for Custom Extensions. Allowing someone to build new extensions and add them to classpath without having to rebuild or modify the core SITX lib.

I will look at adding the same model support for SRO relationships and other Custom Object support once we determine if its a viable pattern.

(Annotation naming could use some work... so if you have better ideas, please share!)

I do like this approach. How about the annotation be called @ApplicableTo({Process.class}) or @AppliedTo is shorter.

anlklsim commented 5 years ago

oops. clicked the wrong button

anlklsim commented 5 years ago

@anlklsim I am looking at create a "BusinessRule" Annotation to handle the various conditional field combinations found in COO and Extensions (and possibly used in other object types)

Would like your thoughts.

Imagine this style of function on a single interface. Where values like "some_field" and "some1", etc are the JsonProperty annotation values.

@BusinessRule(if = “some_field”, is = PRESENT, then = {“some1”, “some11”}, require = ALL)
@BusinessRule(if = “some_field”, is = PRESENT, then = {“some2”, “some3”}, require = MIN1)
@BusinessRule(if = “some_field123”, is = EMPTY, then = {“some5”, “some6”}, require = MIN1)

Any better usage or style thoughts? I was looking at using Spring Expression Lang but its missing support for optionals.

I find the if, then, within the annotation a bit awkward. I found this exampl on Class-leve Constraints: (http://in.relation.to/2008/04/01/bean-validation-sneak-peek-part-ii-custom-constraints/#H-ClasslevelConstraints). I think it would be more straight-forward to implement the custom annotation validations for the few classes that require them. I don't think we need a generic business rule processor - encoding the spec rules in the custom annotation should do the job well.

StephenOTT commented 5 years ago

I implemented a SPEL class level processor:

see the latest commits above (still testing with various scenarios, but looking good so far + will need to be configed to be highly compiled and cached for perf). I used SPEL because there was too many variations in the STIX spec for the conditionals.

@BusinessRule(ifExp = "isMultipart() == true", thenExp = "getBody().isPresent() == false", errorMessage = "Email Message cannot have Body when email is Multipart")
public interface EmailMessageCoo extends CyberObservableObject {
...
}
@BusinessRule(ifExp = "getAslrEnabled().isPresent() == true", thenExp = "getDepEnabled().isPresent() == false", errorMessage = "Dep and ASLR cannot both be enabled")
public interface WindowsProcessExtensionExt extends CyberObservableExtension {
@Documented
@Constraint(validatedBy = {StixValidateBusinessRuleValidator.class})
@Target( { ANNOTATION_TYPE, TYPE })
@Retention(RetentionPolicy.RUNTIME)
public @interface BusinessRule {
    String message() default "{io.digitalstate.stix.validation.contraints.businessrule.BusinessRule}";
    Class<?>[] groups() default {};
    Class<? extends Payload>[] payload() default {};
    String ifExp();
    String thenExp();
    String errorMessage() default "An error occurred";
    boolean expectedResult() default true;
}

(and again we can play with the specific annotation name and param naming)

StephenOTT commented 5 years ago

another example

@BusinessRule(ifExp = "getUrl().isPresent() == true", thenExp = "getHashes().isEmpty() == false && getPayloadBin().isPresent() == false", errorMessage = "When Url is used, Hashes have at least 1 value, and payload_bin cannot be used.")
@BusinessRule(ifExp = "getPayloadBin().isPresent() == true", thenExp = "getUrl().isPresent() == false && getHashes().isEmpty() == true", errorMessage = "When payload_bin is used, Url and hashes cannot be used.")
public interface ArtifactCoo extends CyberObservableObject {

http://docs.oasis-open.org/cti/stix/v2.0/cs01/part4-cyber-observable-objects/stix-v2.0-cs01-part4-cyber-observable-objects.html#_Toc496716218

StephenOTT commented 5 years ago

@anlklsim I have added business rule support to all COO related classes.

StephenOTT commented 5 years ago

@anlklsim Enums have been converted to Vocabs

StephenOTT commented 5 years ago

I do like this approach. How about the annotation be called @ApplicableTo({Process.class}) or @AppliedTo is shorter.

@anlklsim how about @AppliesTo(..) or UsedBy