Proposal - Combine Discovery Request/Response and Collection Information Request/Resopnse

I really do not see any value in having these two things be separate items. And I think could be easily combined in to one message group. If you wanted to get just the short answer and not the collection information in your discovery request, then I think we could add a field to the discovery request that is a boolean for extended_data: yes/no. But even then, I do not really see that as being of value.

The current work flow is: Client Sends Discovery Request -> <- Server Sends Discovery Response Client then needs to send Collection Information Request -> <- Server then Sends Collection Information Response

The extra back and forth is needed right now just so someone can do a Poll. I think it is an extra amount of work that is really not of value. People can still withhold sending the information if they choose.

I've just been re-reading the TAXII Service Specification v1.1 document, and I would have to agree with Bret. The two messages differ in their focus - The Discovery_Request (DR) tells the TAXII client what TAXII services are available, and the Collection_Information_Request (CIR) tells the TAXII Client what Data Colllections are available and what services are provided for each Data Collection. One shows services, and the other shows collections and the services within those. Granted, the DR does also show discovery services and collection management services which aren't shown within CIR, but the general gist applies.

It seems to me that we should pick one way or the other. We either make things discoverable via the data collections (i.e. CIR), or via the services (i.e. DR). It occurs to me that:

Repeating the services within collections (within CIR) will add a lot more content than just repeating the collection name within certain services
The DR already contains all services within TAXII. It would be minimal changes needed to add in the collection names within the poll and inbox services.

So, what I'm proposing is the following.

The Discovery Request is extended to include information about Collections i.e. the following is added to INBOX and POLL service instance descriptions.

    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>

The Discovery_Response would become:

<taxii_11:Discovery_Response xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" message_id="1752634358726784672" in_response_to="1050293259">
  <taxii_11:Service_Instance service_type="DISCOVERY" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/discovery/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="DISCOVERY" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/discovery/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="DISCOVERY" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/discovery/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>The default Discovery Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="DISCOVERY" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/discovery/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>The default Discovery Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="POLL" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/poll/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>This is the default Poll Service.</taxii_11:Message>
    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="POLL" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/poll/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>This is the default Poll Service.</taxii_11:Message>
    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/inbox/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>This is the default Inbox Service.</taxii_11:Message>
    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/inbox/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>This is the default Inbox Service.</taxii_11:Message>
    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="COLLECTION_MANAGEMENT" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/collection-management/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>The default Collection Management Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="COLLECTION_MANAGEMENT" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/collection-management/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>The default Collection Management Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
</taxii_11:Discovery_Response>

The Collection Information Request/Response is depricated.

I think this would provide the following benefits:

Reduce the number of messages passed between TAXII Client and TAXII Server
Reduce the overall number of bytes passed between TAXII Client and TAXII Server
Make it clearer that the directory service lists where you go to get the information, and that the collection management service just does collection management (not collection discovery as well as it presently does).

@terrymacdonald,

To me, the repetition of <Collection> elements signifies something that could be normalized.

What about having a separate set of <Collection> elements, and referring to them by collection name within the service descriptions? Maybe that's too much of a plain-ol' concatenation of the two message types.

IMO, this is definitely on the right track.

-Mark

@MarkDavidson,

The problem with normalizing the information separately is that not all collections will be available in all the services. Some collections would potentially be upload only (inbox only), and others download only (poll only). Some will be available over http (public) and others only over https (private). If we normalize them under a different separate collections object then it creates the issue of us needing to then delineate the differences within a separate filtering mechanism in order to describe these independently. I think the slight repetitiveness is outweighed by the flexibility we retain by keeping the collections described per service.

Cheers

Terry MacDonald | STIX, TAXII, CybOX Consultant

M: +61-407-203-026 E: terry.macdonald@threatloop.com W: www.threatloop.com

https://www.threatloop.com

Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.

On 27 May 2015 at 05:02, MarkDavidson notifications@github.com wrote:

@terrymacdonald https://github.com/terrymacdonald,

To me, the repetition of elements signifies something that could be normalized.

What about having a separate set of elements, and referring to them by collection name within the service descriptions? Maybe that's too much of a plain-ol' concatenation of the two message types.

IMO, this is definitely on the right track.

-Mark

— Reply to this email directly or view it on GitHub https://github.com/TAXIIProject/TAXII-Specifications/issues/74#issuecomment-105635806 .

I'm confused about Terry's sample response. Why are identical service instances repeated twice in the response? For example:

  <taxii_11:Service_Instance service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:http:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/inbox/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>This is the default Inbox Service.</taxii_11:Message>
    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1" available="true">
    <taxii_11:Protocol_Binding>urn:taxii.mitre.org:protocol:https:1.0</taxii_11:Protocol_Binding>
    <taxii_11:Address>/services/inbox/</taxii_11:Address>
    <taxii_11:Message_Binding>urn:taxii.mitre.org:message:xml:1.1</taxii_11:Message_Binding>
    <taxii_11:Message>This is the default Inbox Service.</taxii_11:Message>
    <taxii_11:Collections>
        <taxii_11:Collection collection_name="default" collection_type="DATA_FEED" available="true">
        <taxii_11:Collection collection_name="test" collection_type="DATA_SET" available="true">
    </taxii_11:Collections>
  </taxii_11:Service_Instance>

Http and https :).

Cheers Terry MacDonald

On 27 May 2015 at 06:13, Kurt Zettel notifications@github.com wrote:

I'm confused about Terry's sample response. Why are identical service instances repeated twice in the response? For example:
urn:taxii.mitre.org:protocol:http:1.0 /services/inbox/ urn:taxii.mitre.org:message:xml:1.1 This is the default Inbox Service. urn:taxii.mitre.org:protocol:https:1.0 /services/inbox/ urn:taxii.mitre.org:message:xml:1.1 This is the default Inbox Service. — Reply to this email directly or view it on GitHub https://github.com/TAXIIProject/TAXII-Specifications/issues/74#issuecomment-105652029 .

@MarkDavidson @terrymacdonald the thing I do not like about this representation is that it is based on the idea of the "service" being the thing people care about. And I would argue that what people really care about is the "collection"... They want to interact with a collection not with a service. So by sorting them by collection first, I believe, makes it easier for us to work with. See this layout, https://github.com/TAXIIProject/TAXII-Specifications/issues/77

@terrymacdonald @MarkDavidson Put another way, if you publish an IP-Watch-List collection, I will want to interact with it and will want to figure out all of the services that you offer for this IP-Watch-List, meaning can I share or only poll. The way I see people needing to interact with TAXII is to say: 1) What collections do you offer... 2) Based on the list of collections, (think a UX at this point), what services are available for them, meaning from the UX stand point, which ones can I subscript to by just clicking a checkbox. If we do it by service first then I need to parse the list of collections in the poll service and then parse the list of collections in the inbox service to see if I can a) poll data and b) share data with you for the same collections.

@jordan2175 - I understand that, but there is far more detail in the description of each service than there is in the description of each collection. Repeating the services would make the file bigger than having the service endpoints at the first layer, then the Collections inside them.

Another option I tried was to give each TAXII service endpoint an ID, and refer to that ID in a separate collections structure e.g:

MarkDavidson: Bret, I edited the response so that the XML highlighting would show up. Edit2: Hm.. the highlighting works in preview but not in reality. I give up.

<taxii_11:Discovery_Response xmlns:taxii="
http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="
http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="
http://taxii.mitre.org/query/taxii_default_query-1"
message_id="1752634358726784672" in_response_to="1050293259">
  <taxii_11:Service_Instances>
<taxii_11:Service_Instance service_type="DISCOVERY"
service_version="urn:taxii.mitre.org:services:1.1" available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:http:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/discovery/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
</taxii_11:Service_Instance>
<taxii_11:Service_Instance service_type="DISCOVERY"
service_version="urn:taxii.mitre.org:services:1.1" available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:https:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/discovery/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
</taxii_11:Service_Instance>
<taxii_11:Service_Instance service_type="DISCOVERY"
service_version="urn:taxii.mitre.org:services:1.1" available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:http:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/discovery/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>The default Discovery Service.</taxii_11:Message>
</taxii_11:Service_Instance>
<taxii_11:Service_Instance service_type="DISCOVERY"
service_version="urn:taxii.mitre.org:services:1.1" available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:https:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/discovery/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>The default Discovery Service.</taxii_11:Message>
</taxii_11:Service_Instance>
<taxii_11:Service_Instance id="c6c6dd97-bc6a-47f6-8637-f6986979234d"
service_type="POLL" service_version="urn:taxii.mitre.org:services:1.1"
available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:http:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/poll/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>This is the default Poll Service.</taxii_11:Message>
</taxii_11:Service_Instance>
<taxii_11:Service_Instance id="b4da3b39-7ed7-46a9-9276-e670fcb8a6c6"
service_type="POLL" service_version="urn:taxii.mitre.org:services:1.1"
available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:https:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/poll/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>This is the default Poll Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance id="e7b2f6a1-2ae1-42fa-a884-34ec49c166cf"
service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1"
available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:http:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/inbox/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>This is the default Inbox Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance id="836a2f78-8642-49e4-8ee8-fdb982d23724"
service_type="INBOX" service_version="urn:taxii.mitre.org:services:1.1"
available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:https:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/inbox/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>This is the default Inbox Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="COLLECTION_MANAGEMENT"
service_version="urn:taxii.mitre.org:services:1.1" available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:http:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/collection-management/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>The default Collection Management
Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
  <taxii_11:Service_Instance service_type="COLLECTION_MANAGEMENT"
service_version="urn:taxii.mitre.org:services:1.1" available="true">
<taxii_11:Protocol_Binding>urn:taxii.mitre.org:
protocol:https:1.0</taxii_11:Protocol_Binding>
<taxii_11:Address>/services/collection-management/</taxii_11:Address>
<taxii_11:Message_Binding>urn:taxii.mitre.org:
message:xml:1.1</taxii_11:Message_Binding>
<taxii_11:Message>The default Collection Management
Service.</taxii_11:Message>
  </taxii_11:Service_Instance>
<taxii_11:Service_Instances>
<taxii_11:Collections>
<taxii_11:Collection collection_name="default" collection_type="DATA_FEED"
available="true">
<taxii_11:Service_Instance idref="c6c6dd97-bc6a-47f6-8637-f6986979234d">
<taxii_11:Service_Instance idref="b4da3b39-7ed7-46a9-9276-e670fcb8a6c6">
<taxii_11:Service_Instance idref="e7b2f6a1-2ae1-42fa-a884-34ec49c166cf">
<taxii_11:Service_Instance idref="836a2f78-8642-49e4-8ee8-fdb982d23724">
</taxii_11:Collection>
<taxii_11:Collection collection_name="test" collection_type="DATA_SET"
available="true">
<taxii_11:Service_Instance idref="c6c6dd97-bc6a-47f6-8637-f6986979234d">
<taxii_11:Service_Instance idref="b4da3b39-7ed7-46a9-9276-e670fcb8a6c6">
<taxii_11:Service_Instance idref="e7b2f6a1-2ae1-42fa-a884-34ec49c166cf">
<taxii_11:Service_Instance idref="836a2f78-8642-49e4-8ee8-fdb982d23724">
</taxii_11:Collection>
</taxii_11:Collections>
</taxii_11:Discovery_Response>

But, doing it this way is more complicated (due to the referencing of the ids, and is longer than the original way I suggested.(6176 chars versus 6013). We are wanting to make things more efficient I still believe my original suggestion stands as the best option so far.

We could just mandate the location that all services are located, in the same way that HTTP is on TCP port 80 do, so everyone knows how to get to them i.e.

We mandate that TAXII Servers need to be on TCP port 3711 (unencrypted) and TCP port 13711 (encrypted) (note - random ports plucked out of air. We would need to submit something to IANA)
We mandate that the discovery path must be /d
We mandate that the collection mgmt path must be /cm
We mandate that the poll path must be /p
We mandate that the inbox path must be /i

Then we don't need the services part of the discovery, as a TAXII client knows where all the endpoints will be and then can just send the collections. We do lose flexibility to have discovery services on different servers to the poll, inbox and collection mgmt services, but how many people are actually separating their services onto different boxes? Also, if that is needed then they just need to run a reverse proxy - exactly as they already do when making a website that is across different servers (even using the same software to do it!)

Cheers

Terry MacDonald | STIX, TAXII, CybOX Consultant

M: +61-407-203-026 E: terry.macdonald@threatloop.com W: www.threatloop.com

https://www.threatloop.com

Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.

On 27 May 2015 at 06:22, jordan2175 notifications@github.com wrote:

@terrymacdonald https://github.com/terrymacdonald @MarkDavidson https://github.com/MarkDavidson Put another way, if you publish an IP-Watch-List collection, I will want to interact with it and will want to figure out all of the services that you offer for this IP-Watch-List, meaning can I share or only poll. The way I see people needing to interact with TAXII is to say: 1) What collections do you offer... 2) Based on the list of collections, (think a UX at this point), what services are available for them, meaning from the UX stand point, which ones can I subscript to by just clicking a checkbox. If we do it by service first then I need to parse the list of collections in the poll service and then parse the list of collections in the inbox service to see if I can a) poll data and b) share data with you for the same collections.

— Reply to this email directly or view it on GitHub https://github.com/TAXIIProject/TAXII-Specifications/issues/74#issuecomment-105654167 .

I agree that it makes sense to have services as the top level element as well. It is a known and limited quantity so repetition is less of a problem. Additionally, it is possible that some services don't have collections. Today taxii test (http://taxiitest.mitre.org/) does not even expose the collection service so there is a chance that other producers are not collection aware.

@kurtzettel The problem with that is everything in TAXII is done by a collection name, if you do not have the collection name, you really can not do anything.

@terrymacdonald Please remove the XML in your examples and just use space/tabbed structure so we can see it without all of the markup.

So with my example it would look something like this (1333 characters, not including spaces)

collection_response
    id: 12345
    in_response_to: 54321
    version: urn:taxii.mitre.org:services:1.1
    collections
        name: IP-Watch-List
            available: TRUE
            type: FEED
            poll_services
                protocol: urn:taxii.mitre.org:protocol:http:1.0
                address: http://foo.com/services/poll
                encodings: urn:taxii.mitre.org:message:json:1.1
                content_encodings: urn:stix.mitre.org:json:1.0
            inbox_services
                protocol: urn:taxii.mitre.org:protocol:http:1.0
                address: http://foo.com/services/inbox
                encodings: urn:taxii.mitre.org:message:json:1.1
                content_encodings: urn:stix.mitre.org:json:1.0
            subscription_services
                protocol: urn:taxii.mitre.org:protocol:http:1.0
                address: http://foo.com/services/subscription
                encodings: urn:taxii.mitre.org:message:json:1.1
                content_encodings: urn:stix.mitre.org:json:1.0
        name: URL-Watch-List
            available: TRUE
            type: FEED
            poll_services
                protocol: urn:taxii.mitre.org:protocol:http:1.0
                address: http://foo.com/services/poll
                encodings: urn:taxii.mitre.org:message:json:1.1
                content_encodings: urn:stix.mitre.org:json:1.0
            inbox_services
                protocol: urn:taxii.mitre.org:protocol:http:1.0
                address: http://foo.com/services/inbox
                encodings: urn:taxii.mitre.org:message:json:1.1
                content_encodings: urn:stix.mitre.org:json:1.0
            subscription_services
                protocol: urn:taxii.mitre.org:protocol:http:1.0
                address: http://foo.com/services/subscription
                encodings: urn:taxii.mitre.org:message:json:1.1
                content_encodings: urn:stix.mitre.org:json:1.0

So yes, I can see the overhead in the protocol, encodings, and content_encodings fields. Maybe we could use inheritance???

The other option as I see it from you would be (941 characters, not including spaces)

collection_response
    id: 12345
    in_response_to: 54321
    version: urn:taxii.mitre.org:services:1.1
    services
        type: INBOX
            protocol: urn:taxii.mitre.org:protocol:http:1.0
            address: http://foo.com/services/inbox
            encodings: urn:taxii.mitre.org:message:json:1.1
            content_encodings: urn:stix.mitre.org:json:1.0
            collections
                name: IP-Watch-List
                    available: TRUE
                    type: FEED
                name: URL-Watch-List
                    available: TRUE
                    type: FEED
        type: POLL
            protocol: urn:taxii.mitre.org:protocol:http:1.0
            address: http://foo.com/services/poll
            encodings: urn:taxii.mitre.org:message:json:1.1
            content_encodings: urn:stix.mitre.org:json:1.0
            collections
                name: IP-Watch-List
                    available: TRUE
                    type: FEED
                name: URL-Watch-List
                    available: TRUE
                    type: FEED
        type: SUBSCRIPTION
            protocol: urn:taxii.mitre.org:protocol:http:1.0
            address: http://foo.com/services/subsciprtion
            encodings: urn:taxii.mitre.org:message:json:1.1
            content_encodings: urn:stix.mitre.org:json:1.0
            collections
                name: IP-Watch-List
                    available: TRUE
                    type: FEED
                name: URL-Watch-List
                    available: TRUE
                    type: FEED

If we used inheritance in my example it could look like: (648 characters, not including spaces)

collection_response
    id: 12345
    in_response_to: 54321
    version: urn:taxii.mitre.org:services:1.1
    protocol: urn:taxii.mitre.org:protocol:http:1.0
    encodings: urn:taxii.mitre.org:message:json:1.1
    content_encodings: urn:stix.mitre.org:json:1.0
    collections
        name: IP-Watch-List
            available: TRUE
            type: FEED
            poll_services
                address: http://foo.com/services/poll
            inbox_services
                address: http://foo.com/services/inbox
            subscription_services
                address: http://foo.com/services/subscription
        name: URL-Watch-List
            available: TRUE
            type: FEED
            poll_services
                address: http://foo.com/services/poll
            inbox_services
                address: http://foo.com/services/inbox
            subscription_services
                address: http://foo.com/services/subscription

Your idea with inheritance could look like: (667 characters, not including spaces). A bit more then mine with inheritance

collection_response
    id: 12345
    in_response_to: 54321
    version: urn:taxii.mitre.org:services:1.1
    protocol: urn:taxii.mitre.org:protocol:http:1.0
    encodings: urn:taxii.mitre.org:message:json:1.1
    content_encodings: urn:stix.mitre.org:json:1.0
    services
        type: INBOX
            address: http://foo.com/services/inbox
            collections
                name: IP-Watch-List
                    available: TRUE
                    type: FEED
                name: URL-Watch-List
                    available: TRUE
                    type: FEED
        type: POLL
            address: http://foo.com/services/poll
            collections
                name: IP-Watch-List
                    available: TRUE
                    type: FEED
                name: URL-Watch-List
                    available: TRUE
                    type: FEED
        type: SUBSCRIPTION
            address: http://foo.com/services/subsciprtion
            collections
                name: IP-Watch-List
                    available: TRUE
                    type: FEED
                name: URL-Watch-List
                    available: TRUE
                    type: FEED

So without inheritance, your solution saves more space on the wire. However, with inheritance mine saves more space. In addition I believe my layout would be easier to process in code, IMO.

I'll throw some random thoughts against the wall:

Distinguishing TAXII Services by URL doesn't seem to be necessary. You could just have a "TAXII" URL that handles TAXII Messages (e.g., http://example.com/taxii/)
- This results in the term "TAXII Service" going away, it's just "TAXII" with message exchanges that are supported (or not)
Right now there's not any required functionality in TAXII. I think this is something that could change going forward.

If these thoughts were incorporated into a future revision, you could have something like:

The root TAXII URL is specified in DNS (yay auto-discovery!), perhaps in a TXT record.
Strongly bind TAXII to HTTP
- The current design allows for multiple protocols, which precludes using something like HTTP Options as a TAXII feature
HTTP Options (as Bret has suggested elsewhere) would be your entry into TAXII-land
- HTTP Options could be required functionality
- We'd have to think about what things belong in the initial HTTP Options request-response
Maybe have a "Data Collection first" methodology, where you (notionally) have this information:
- collection_name, collection_type, can_push, can_pull, supported_formats, available
- Since everything is off the same URL, you don't need to know protocol-specific things like URL

For fun, a notional response to an HTTP Options request:

supported_taxii_versions: urn:taxii.mitre.org:services:2.0
supported_message_bindings: urn:taxii.mitre.org:message:json:2.0, 
                            urn:taxii.mitre.org:message:xml:2.0
collections: 
    name: IP-Watch-List
    available: True
    type: FEED
    can_poll: True  # To poll, submit a poll request to the same URL
    can_push: True # To push, submit an inbox message to the same URL
    can_subscribe: True # To subscribe, submit a subscription request to the same URL
    content_encodings: urn:stix.mitre.org:json:1.0

    name: URL-Watch-List
    available: True
    type: FEED
    can_poll: True  # To poll, submit a poll request to the same URL
    can_push: True # To push, submit an inbox message to the same URL
    can_subscribe: True # To subscribe, submit a subscription request to the same URL
    content_encodings: urn:stix.mitre.org:json:1.0

There'd of course be a bunch of other stuff to nail down, like how authentication would work, how you'd handle presenting more information to authenticated users, whether you wanted to go http only or https only, but this might be a starting point.

Thank you. -Mark

@MarkDavidson That is a brilliant suggestion and makes so much sense. If we could simplify the messages and their structure and get tight binding to HTTP/s that would be great. I would love to start working on this. Can you create a page on the WIKI where we can start working through the messages, formats, etc?

@jordan2175, cc: @terrymacdonald,

I set it up here: https://github.com/TAXIIProject/TAXII-Specifications/wiki/idea-exploration.

Comments and additions welcome.

Thank you. -Mark

I initially didn't like the idea of binding TAXII strongly to HTTP per se, as it potentially precludes the use of a binary protocol in the future. I figured this could be mitigated by registering distinct IANA TCP ports for TAXII HTTP and TAXII HTTPS ports, and different ports in the future for binary TAXII protocol. Then the particular ports would be reserved specifically for HTTP, and you could strongly bind them together. This has the one BIG downside of requiring firewall ports to be opened to allow traffic flow - this would restrict the ease that new implementations could be rolled out, so on second thought I discounted it. Although we'll still probably need to do that when we get to the binary protocol implementation as it will likely require its own separate port.

If we did bind HTTP TAXII to a manadatory url (e.g., http://example.com/taxii/) then that would make it easier to find. As I mentioned in another github post if we created a new '_taxii' DNS TXT record containing the URL to the TAXII services (e.g. http://example.com/taxii/ ) then auto-discovery would work well.

TAXII Client would then be able to either:

Look up DNServer for a domain
Check for a _taxii DNS TXT record
Go directly to the TAXII URL and continue with the discovery process as Mark outlined earlier.

Go directly to the domain on a well-known TAXII URL http:///taxii

I like it!

And the hookup with DNS potentially allows us to use DANE in the future as well ( http://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities). The plan is coming together nicely ( https://i.chzbgr.com/maxW500/5057315840/h31215519/).

Cheers

Terry MacDonald | STIX, TAXII, CybOX Consultant

M: +61-407-203-026 E: terry.macdonald@threatloop.com W: www.threatloop.com

https://www.threatloop.com

Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.

On 27 May 2015 at 23:25, MarkDavidson notifications@github.com wrote:

I'll throw some random thoughts against the wall:

Distinguishing TAXII Services by URL doesn't seem to be necessary. You could just have a "TAXII" URL that handles TAXII Messages (e.g., http://example.com/taxii/)

This results in the term "TAXII Service" going away, it's just "TAXII" with message exchanges that are supported (or not)

Right now there's not any required functionality in TAXII. I think this is something that could change going forward.

If these thoughts were incorporated into a future revision, you could have something like:

The root TAXII URL is specified in DNS (yay auto-discovery!), perhaps in a TXT record.

Strongly bind TAXII to HTTP

The current design allows for multiple protocols, which precludes using something like HTTP Options as a TAXII feature

HTTP Options (as Bret has suggested elsewhere) would be your entry into TAXII-land

HTTP Options could be required functionality

We'd have to think about what things belong in the initial HTTP Options request-response

Maybe have a "Data Collection first" methodology, where you (notionally) have this information:

collection_name, collection_type, can_push, can_pull, supported_formats, available

Since everything is off the same URL, you don't need to know protocol-specific things like URL

For fun, a notional response to an HTTP Options request:

supported_taxii_versions: urn:taxii.mitre.org:services:2.0 supported_message_bindings: urn:taxii.mitre.org:message:json:2.0, urn:taxii.mitre.org:message:xml:2.0 collections: name: IP-Watch-List available: True type: FEED can_poll: True # To poll, submit a poll request to the same URL can_push: True # To push, submit an inbox message to the same URL can_subscribe: True # To subscribe, submit a subscription request to the same URL content_encodings: urn:stix.mitre.org:json:1.0
name: URL-Watch-List
available: True
type: FEED
can_poll: True  # To poll, submit a poll request to the same URL
can_push: True # To push, submit an inbox message to the same URL
can_subscribe: True # To subscribe, submit a subscription request to the same URL
content_encodings: urn:stix.mitre.org:json:1.0
There'd of course be a bunch of other stuff to nail down, like how authentication would work, how you'd handle presenting more information to authenticated users, whether you wanted to go http only or https only, but this might be a starting point.

Thank you. -Mark

— Reply to this email directly or view it on GitHub https://github.com/TAXIIProject/TAXII-Specifications/issues/74#issuecomment-105906855 .

This thread is now migrating away from the initial issue. But let me make two comments.. @terrymacdonald I agree with you, I would love to see a binary version in the future and my hope is that the JSON version will get us 80% there. We can still run a binary version of TAXII over TCP 443, there is no issue with that. We can also run a binary version of TAXII inside of HTTPS and all of that over TCP port 443, no issue with that either. Lots of applications these days hijack the old defined port mapping and those port mappings do not mean a lot anymore. On a firewall or IPS, about all you can say is if it is TCP 443, it is probably HTTPS, but you will need to check to make sure. Most products no longer trust the port numbers and instead use deep application analysis.

TAXIIProject / TAXII-Specifications

Proposal - Combine Discovery Request/Response and Collection Information Request/Resopnse #74