PollResponse - empty messages

[c-x]

I wrote a simple test_client just like the example you provide in the script directory using libtaxii11.

When I run this client without time filters, I can see events returned and that's great. However, when I filter to have no events returned, I still have a PollResponse Message with no events in it but having the stix headers.

How can I identify this behavior (a response only containing headers) ?

<taxii_11:Poll_Response xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" message_id="32288" in_response_to="60368" collection_name="XXXXXX" more="false">
  <taxii_11:Inclusive_End_Timestamp>2014-07-24T18:53:44.071702+00:00</taxii_11:Inclusive_End_Timestamp>
  <taxii_11:Content_Block>
    <taxii_11:Content_Binding binding_id="urn:stix.mitre.org:xml:1.0"/>
    <taxii_11:Content>
      <stix:STIX_Package xmlns:CodeObj="http://cybox.mitre.org/objects#CodeObject-2" xmlns:ioc-tr="http://schemas.mandiant.com/2010/ioc/TR/" xmlns:GUIObj="http://cybox.mitre.org/objects#GUIObject-2" xmlns:ioc="http://schemas.mandiant.com/2010/ioc" xmlns:coa="http://stix.mitre.org/CourseOfAction-1" xmlns:WinDriverObj="http://cybox.mitre.org/objects#WinDriverObject-2" xmlns:ProcessObj="http://cybox.mitre.org/objects#ProcessObject-2" xmlns:PDFFileObj="http://cybox.mitre.org/objects#PDFFileObject-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:DiskObj="http://cybox.mitre.org/objects#DiskObject-2" xmlns:UserAccountObj="http://cybox.mitre.org/objects#UserAccountObject-2" xmlns:CustomObj="http://cybox.mitre.org/objects#CustomObject-1" xmlns:ARPCacheObj="http://cybox.mitre.org/objects#ARPCacheObject-1" xmlns:WinNetworkRouteEntryObj="http://cybox.mitre.org/objects#WinNetworkRouteEntryObject-2" xmlns:AddressObj="http://cybox.mitre.org/objects#AddressObject-2" xmlns:UserSessionObj="http://cybox.mitre.org/objects#UserSessionObject-2" xmlns:cvrf="http://www.icasi.org/CVRF/schema/cvrf/1.1" xmlns:WinPipeObj="http://cybox.mitre.org/objects#WinPipeObject-2" xmlns:ArtifactObj="http://cybox.mitre.org/objects#ArtifactObject-2" xmlns:PacketObj="http://cybox.mitre.org/objects#PacketObject-2" xmlns:incident="http://stix.mitre.org/Incident-1" xmlns:HTTPSessionObj="http://cybox.mitre.org/objects#HTTPSessionObject-2" xmlns:WinCriticalSectionObj="http://cybox.mitre.org/objects#WinCriticalSectionObject-2" xmlns:DeviceObj="http://cybox.mitre.org/objects#DeviceObject-2" xmlns:WinVolumeObj="http://cybox.mitre.org/objects#WinVolumeObject-2" xmlns:MutexObj="http://cybox.mitre.org/objects#MutexObject-2" xmlns:scap-core="http://scap.nist.gov/schema/scap-core/1.0" xmlns:WinSystemRestoreObj="http://cybox.mitre.org/objects#WinSystemRestoreObject-2" xmlns:WinMailslotObj="http://cybox.mitre.org/objects#WinMailslotObject-2" xmlns:WinFileObj="http://cybox.mitre.org/objects#WinFileObject-2" xmlns:DiskPartitionObj="http://cybox.mitre.org/objects#DiskPartitionObject-2" xmlns:ciq="urn:oasis:names:tc:ciq:xpil:3" xmlns:WinSystemObj="http://cybox.mitre.org/objects#WinSystemObject-2" xmlns:PipeObj="http://cybox.mitre.org/objects#PipeObject-2" xmlns:ttp="http://stix.mitre.org/TTP-1" xmlns:URLHistoryObj="http://cybox.mitre.org/objects#URLHistoryObject-1" xmlns:capecInstance="http://stix.mitre.org/extensions/AP#CAPEC2.5-1" xmlns:UnixUserAccountObj="http://cybox.mitre.org/objects#UnixUserAccountObject-2" xmlns:WinMemoryPageRegionObj="http://cybox.mitre.org/objects#WinMemoryPageRegionObject-2" xmlns:NetworkSubnetObj="http://cybox.mitre.org/objects#NetworkSubnetObject-2" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:UnixPipeObj="http://cybox.mitre.org/objects#UnixPipeObject-2" xmlns:WinProcessObj="http://cybox.mitre.org/objects#WinProcessObject-2" xmlns:ciqAddress="http://stix.mitre.org/extensions/Address#CIQAddress3.0-1" xmlns:SemaphoreObj="http://cybox.mitre.org/objects#SemaphoreObject-2" xmlns:cvrfVuln="http://stix.mitre.org/extensions/Vulnerability#CVRF-1" xmlns:WinEventObj="http://cybox.mitre.org/objects#WinEventObject-2" xmlns:EmailMessageObj="http://cybox.mitre.org/objects#EmailMessageObject-2" xmlns:WinWaitableTimerObj="http://cybox.mitre.org/objects#WinWaitableTimerObject-2" xmlns:WhoisObj="http://cybox.mitre.org/objects#WhoisObject-2" xmlns:WinEventLogObj="http://cybox.mitre.org/objects#WinEventLogObject-2" xmlns:capec="http://capec.mitre.org/capec-2" xmlns:UnixProcessObj="http://cybox.mitre.org/objects#UnixProcessObject-2" xmlns:genericTM="http://stix.mitre.org/extensions/TestMechanism#Generic-1" xmlns:oval="http://oval.mitre.org/XMLSchema/oval-common-5" xmlns:WinSemaphoreObj="http://cybox.mitre.org/objects#WinSemaphoreObject-2" xmlns:NetworkRouteEntryObj="http://cybox.mitre.org/objects#NetworkRouteEntryObject-2" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:CISCP="http://www.us-cert.gov/ciscp" xmlns:VolumeObj="http://cybox.mitre.org/objects#VolumeObject-2" xmlns:DNSQueryObj="http://cybox.mitre.org/objects#DNSQueryObject-2" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:WinUserAccountObj="http://cybox.mitre.org/objects#WinUserAccountObject-2" xmlns:campaign="http://stix.mitre.org/Campaign-1" xmlns:snortTM="http://stix.mitre.org/extensions/TestMechanism#Snort-1" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:PortObj="http://cybox.mitre.org/objects#PortObject-2" xmlns:UnixVolumeObj="http://cybox.mitre.org/objects#UnixVolumeObject-2" xmlns:SMSMessageObj="http://cybox.mitre.org/objects#SMSMessageObject-1" xmlns:maec="http://maec.mitre.org/XMLSchema/maec-package-2" xmlns:NetworkSocketObj="http://cybox.mitre.org/objects#NetworkSocketObject-2" xmlns:GUIDialogBoxObj="http://cybox.mitre.org/objects#GUIDialogboxObject-2" xmlns:LibraryObj="http://cybox.mitre.org/objects#LibraryObject-2" xmlns:WinThreadObj="http://cybox.mitre.org/objects#WinThreadObject-2" xmlns:cvrf-common="http://www.icasi.org/CVRF/schema/common/1.1" xmlns:NetworkConnectionObj="http://cybox.mitre.org/objects#NetworkConnectionObject-2" xmlns:ovalTM="http://stix.mitre.org/extensions/TestMechanism#OVAL5.10-1" xmlns:tlpMarking="http://data-marking.mitre.org/extensions/MarkingStructure#TLP-1" xmlns:WinKernelHookObj="http://cybox.mitre.org/objects#WinKernelHookObject-2" xmlns:WinComputerAccountObj="http://cybox.mitre.org/objects#WinComputerAccountObject-2" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:LinuxPackageObj="http://cybox.mitre.org/objects#LinuxPackageObject-2" xmlns:sch="http://purl.oclc.org/dsdl/schematron" xmlns:yaraTM="http://stix.mitre.org/extensions/TestMechanism#YARA-1" xmlns:prod="http://www.icasi.org/CVRF/schema/prod/1.1" xmlns:DNSCacheObj="http://cybox.mitre.org/objects#DNSCacheObject-2" xmlns:UnixFileObj="http://cybox.mitre.org/objects#UnixFileObject-2" xmlns:NetFlowObj="http://cybox.mitre.org/objects#NetworkFlowObject-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:openiocTM="http://stix.mitre.org/extensions/TestMechanism#OpenIOC2010-1" xmlns:WinRegistryKeyObj="http://cybox.mitre.org/objects#WinRegistryKeyObject-2" xmlns:ProductObj="http://cybox.mitre.org/objects#ProductObject-2" xmlns:WinHandleObj="http://cybox.mitre.org/objects#WinHandleObject-2" xmlns:oval-var="http://oval.mitre.org/XMLSchema/oval-variables-5" xmlns:HostnameObj="http://cybox.mitre.org/objects#HostnameObject-1" xmlns:a="urn:oasis:names:tc:ciq:xal:3" xmlns:cvssv2="http://scap.nist.gov/schema/cvss-v2/1.0" xmlns:DomainObj="http://cybox.mitre.org/objects#DomainObject-1" xmlns:vuln="http://www.icasi.org/CVRF/schema/vuln/1.1" xmlns:simpleMarking="http://data-marking.mitre.org/extensions/MarkingStructure#Simple-1" xmlns:APIObj="http://cybox.mitre.org/objects#APIObject-2" xmlns:DomainNameObj="http://cybox.mitre.org/objects#DomainNameObject-1" xmlns:genericStructuredCOA="http://stix.mitre.org/extensions/StructuredCOA#Generic-1" xmlns:ct="urn:oasis:names:tc:ciq:ct:3" xmlns:SystemObj="http://cybox.mitre.org/objects#SystemObject-2" xmlns:DNSRecordObj="http://cybox.mitre.org/objects#DNSRecordObject-2" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:WinServiceObj="http://cybox.mitre.org/objects#WinServiceObject-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cpe="http://cpe.mitre.org/language/2.0" xmlns:FileObj="http://cybox.mitre.org/objects#FileObject-2" xmlns:GUIWindowObj="http://cybox.mitre.org/objects#GUIWindowObject-2" xmlns:WinPrefetchObj="http://cybox.mitre.org/objects#WinPrefetchObject-2" xmlns:WinKernelObj="http://cybox.mitre.org/objects#WinKernelObject-2" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:WinFilemappingObj="http://cybox.mitre.org/objects#WinFilemappingObject-2" xmlns:marking="http://data-marking.mitre.org/Marking-1" xmlns:URIObj="http://cybox.mitre.org/objects#URIObject-2" xmlns:ta="http://stix.mitre.org/ThreatActor-1" xmlns:MemoryObj="http://cybox.mitre.org/objects#MemoryObject-2" xmlns:WinTaskObj="http://cybox.mitre.org/objects#WinTaskObject-2" xmlns:SocketAddressObj="http://cybox.mitre.org/objects#SocketAddressObject-1" xmlns:WinMutexObj="http://cybox.mitre.org/objects#WinMutexObject-2" xmlns:ASObj="http://cybox.mitre.org/objects#ASObject-1" xmlns:et="http://stix.mitre.org/ExploitTarget-1" xmlns:WinExecutableFileObj="http://cybox.mitre.org/objects#WinExecutableFileObject-2" xmlns:WinNetworkShareObj="http://cybox.mitre.org/objects#WinNetworkShareObject-2" xmlns:LinkObj="http://cybox.mitre.org/objects#LinkObject-1" xmlns:NetworkRouteObj="http://cybox.mitre.org/objects#NetworkRouteObject-2" xmlns:NetworkPacketObj="http://cybox.mitre.org/objects#NetworkPacketObject-2" xmlns:xnl="urn:oasis:names:tc:ciq:xnl:3" xmlns:X509CertificateObj="http://cybox.mitre.org/objects#X509CertificateObject-2" xmlns:maecInstance="http://stix.mitre.org/extensions/Malware#MAEC4.0-1" xmlns:UnixNetworkRouteEntryObj="http://cybox.mitre.org/objects#UnixNetworkRouteEntryObject-2" xmlns:oval-def="http://oval.mitre.org/XMLSchema/oval-definitions-5" xmlns:ciqIdentity="http://stix.mitre.org/extensions/Identity#CIQIdentity3.0-1" xmlns:AccountObj="http://cybox.mitre.org/objects#AccountObject-2" xmlns:xlink="http://www.w3.org/1999/xlink" id="repository:XXXXXXXXXXXXX" version="1.0"/>
    </taxii_11:Content>
    <taxii_11:Timestamp_Label>2014-07-24T18:53:44.075505+00:00</taxii_11:Timestamp_Label>
  </taxii_11:Content_Block>
</taxii_11:Poll_Response>

[MarkDavidson]

@c-x ,

Thank you for the report. I am able to duplicate what you're seeing (aka I think I know what TAXII Server you are using).

In terms of TAXII, this response is technically correct (e.g., the Content Binding ID matches the Content's format), but a "more correct" response might be to not have any Content Blocks in the response since there isn't any information to return.

I'll reach out to the owners of the server and see if it's actually their server and what (if any) comment they might have on this.

In the meantime, can you expand on what you mean by "identify" the behavior? I'm not sure I understand what you mean.

Thank you. -Mark

[usrlocalben]

fixed. ^_^ (no, it was not a libtaxii bug)

[c-x]

Maybe we can have a flag like content_block.only_headers=True or content_block.has_data=True ? I've no solution to provide you, it's more a feature request :-)

In my code I simply used the following function:

    def taxii_message_is_empty(self, content):
        if re.search("^<stix:STIX_Package\s+[^>]+>$", content) :
                return True
        return False

[usrlocalben]

no, it should not have a content block in this case. the self-terminating stix_package is a side effect that I need to clear up.

[MarkDavidson]

@benjamin9999, thanks for the update!

@c-x, I'd like to explore what you're asking for a little more to determine if it's something we should add to libtaxii. Regarding the function you mention (content_block.only_headers / content_block.has_data), I have a couple thoughts:

1. For "well-behaved applications", a content_block.has_data property should always be True, since ContentBlocks contain data. The content field within ContentBlock is required for this reason.
2. For a particular format (e.g., STIX) whether a particular piece of content (like the one you saw) is functionally equivalent to "no content" is within the scope of that particular format and normally not within the bounds of libtaxii. (Big caveat: I like to treat users nicely, so please let me know how useful that kind of function would be!) Presently, libtaxii has stayed away from "knowing" anything about the data that is contained in a ContentBlock other than how to serialize/deserialize it.
3. If that function was implemented, libtaxii would be bound slightly tighter to certain formats that the developers are more familiar with (like STIX) in the sense that libtaxii would provide some convenience functions that work for a subset of potential payloads but not others.

What do you think? Would that function only apply to the one (now addressed) case you ran into, or would it have broader utility?

Thank you. -Mark

[c-x]

@MarkDavidson:

I understood that content_block are data. To me, data is something which is either empty or either something containing a header and a real content. In other words, I should have the ability, if the content only contains the headers, to know that my real data is actually None (because I just have the headers). Today I can't know that directly (I had to use the regex I showed you).

Presently, libtaxii has stayed away from "knowing" anything about the data that is contained in a ContentBlock other than how to serialize/deserialize it.

Yes, but people writing application using libtaxii need to know, one way or the other, that the message they receive is only composed of headers without any real data.

I agree with you regarding the fact that libtaxii should stick to the protocol definition, but maybe the protocol should evolve to handle that particular case ?

[MarkDavidson]

@c-x,

Some thoughts so far (no real conclusions):

My biggest concern with this kind of functionality is that it would only work for Content Bindings that libtaxii "knows" about, and the application that calls libtaxii would have to be prepared for cases where libtaxii doesn't "know" about the Content Binding. e.g.,:

content_block = get_content_block_from_somwhere()
if libtaxii.can_check_content_binding(content_block.content_binding): #Test to see if libtaxii "knows" about the Content Binding
    if content_block.is_content_empty:
        pass # Nothing to do here
    else:
        process(content_block)
else:
    process(content_block) # ... maybe?

This could be partially mitigated by allowing libtaxii users to register their own is_content_empty checks for certain Content Bindings.

Out of curiosity, how do you parse your STIX documents? Maybe this is a feature that the python-stix library could implement?

-Mark

[c-x]

As we agreed in an offline conversation, this issue is actually a TAXII Provider issue. This issue can be closed.