Add instructions on how to modify TTL after the Cassandra schema is applied

[codefromthecrypt]

Does anyone know how to adjust the TTL after the schema has been installed? We should help folks understand this in two scopes:

• change all TTLs change (requested by @pdraper)
• change TTL for one or more traces (requested by @maryoush)

Once we understand how, it would be best to test it (in our integration tests) and document it in the readmes, such as what was started in #1533

@pdraper tried this..

cqlsh> alter table zipkin.traces with default_time_to_live = 60480;
ServerError: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError

[codefromthecrypt]

I assigned mick (as a polite request) but if anyone @openzipkin/cassandra can help, would be great. One way to help is to ask on one of the cassandra forums as I don't expect this to be zipkin specific (except the table names)

[codefromthecrypt]

from gitter @maryoush Mar 29 23:51

in my case only what effectively allows me to get rid of old traces is to run globally script on cassandra to adjust the ( gc_grace_seconds , default_time_to_live )

[maryoush]

The proper cql could look like - AA arbitrary TTL in seconds

ALTER TABLE zipkin.traces
    WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy', 'max_window_size_seconds': 'AA'}
    AND gc_grace_seconds = AA
    AND default_time_to_live =  AA;

[michaelsembwever]

I'm pretty sure in all versions that TTL is only stored in the cell.

The table setting is a default value that gets added to the write. Each column or cell of data contains a timestamp of when it was written and a TTL value.

Therefore changing the table's TTL setting will have no impact on data already on in C*. It will have an impact on subsequent writes though and @maryoush example is correct.

You can reduce gc_grace_seconds though and trigger a user defined compaction on those sstables that then have high tombstone ratios in them. This will help alleviate disk space issues.

Given the nature of zipkin's data you may also want to reduce the gc_grace_seconds. There are some consequences to too low a gc_grace_second value, see http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateTable.html#tabProp__cqlTableGc_grace_seconds ,but for the case of Zipkin where the need for durability isn't critical it probably isn't an issue.

[michaelsembwever]

Each column or cell of data contains a timestamp of when it was written and a TTL value. Therefore changing the table's TTL setting will have no impact on data already on in C*.

To elaborate a bit on this…

You can't "change" TTL of existing data. The data on disk is immutable, so updating data with a new TTL effectively means duplicating it with only the newer version having the new TTL. Further updating old data is a bad idea when using the TimeWindowCompactionStrategy, as it's expecting data written in chronological order (at least not across days).