Consider changing 'Timestamp Label' to 'Sequence ID'

[terrymacdonald]

The Timestamp Label (section 3.1.5 of The TAXII Services Specification v1.0 PDF) seems overly onerous on producers and clients, as it requires the servers to keep track of time.For a producer that deals with a high volume of data, it may be very difficult to accurately create a timestamp of up to 6 digits while under load without some good quality RTC on board.

The purpose of the Timestamp Label field seems to be:

• Keep track of order
• Allow for a consumer to request for a slice of the data by potentially specifying the last part of data they were sent.
• Ensure that each new entry in the data set/feed is atomic

In my mind, a better solution would be to replace the Timestamp Label field with a Sequence ID field. The Sequence ID would just be a sufficiently large sequence number that increments each time that data is added to the set/feed in a similar way that TCP uses sequence numbers to keep track of order in a connection. A producer would need to keep track of Sequence IDs for each data set/feed, and it would not require any special clock accuracy. It would also be atomic within that dataset/feed,allowing any consumer to determine if it has missed part of the feed, and to request that to be sent.

Comments?

Terry

[MarkDavidson]

Hello Terry,

Thank you for the comment. I've reposted your question to the TAXII Discussion List so that it can reach the whole community.

[MarkDavidson]

Another comment, per a post to the TAXII Discussion List by Dave Cridland:

3) Timestamps which are not timestamps.

§4.1.4 I found deeply confusing. Because a timestamp label need not bear any relationship to chronological time, it looks like a very complex way of expressing a strictly monotonically increasing value. Worse, because back ends are required, in effect, to support arbitrary precision this seems like a recipe for some nasty bugs.

Unlike most of the other issues, it's not clear to me this can be fixed since it's relatively low-level; but I'd hope it can be.

[JasonKeirstead]

The problem with sequence ID, is I am not sure how to correlate it back to time.

Fetching documents from a server that were added in a specific time range is going to be a very common use case, and if all documents do not contain a timestamp how do you enable this.

TAXII has no search/filtering mechanism as part of the protocol, the only way you can do any filtering at all today is by time... removing this means you can't do any filtering - it's an 'all or nothing' proposal, not something all clients are going to want.

[MarkDavidson]

@JasonKeirstead,

I'll attempt a clarification. I wasn't thinking this issue was for removing Timestamp Labels, but instead for renaming them while retaining current functionality. The Timestamp portion of Timestamp Label is just a formatting exercise.

E.g.,

1. Rename Timestamp Label to Monotonic Counter (or sequence ID, or ...)
2. Instead of Timestamp Label, use a number (one could do datetime.to_seconds() or an equivalent thing)

Everything should still work as originally designed - e.g., filtering sections of a Data Collection based on Monotonic Counters (nee Timestamp Label).

Thoughts? -Mark

[JasonKeirstead]

Not sure I understand.. if the counter is to contain only a UNIX timestamp, then shouldn't that be part of the spec?

On the other hand, if the counter can contain a non-timestamp derrived counter (for example, start at 1 and go up from there...), then it will not be able to be used to fetch documents within a time period.