TTL Remover not working in Cassandra 4.

[kmrmanish23]

Run.sh for Cassandra 4:

For Cassandra 3 and 4.0

CLASSPATH=$CLASSPATH./impl/target/ttl-remover.jar:./cassandra-4/target/ttl-remover-cassandra-4.jar

change versions of jars on classpath to target 3 or 4

change --cassandra-version if necessary

java -javaagent:/opt/cassandra-ttl-remover/buddy-agent/target/byte-buddy-agent.jar \ -cp "/opt/cassandra-ttl-remover/impl/target/ttl-remover.jar:/opt/cassandra-ttl-remover/cassandra-4/target/ttl-remover-cassandra-4.jar" \ $JVM_OPTS \ com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI \ --cassandra-version=4 \ --sstables \ /var/lib/cassandra/data/cycling \ --output-path \ /var/lib/cassandra/data/cycling/stripped \ --cql \ 'CREATE TABLE IF NOT EXISTS test.test (id uuid, name text, surname text, PRIMARY KEY (id)) WITH default_time_to_live = 10;'

============================================================================= Connected to Test Cluster at 127.0.0.1:9042 [cqlsh 6.0.0 | Cassandra 4.0.7 | CQL spec 3.4.5 | Native protocol v5] Use HELP for help. cqlsh> show version; [cqlsh 6.0.0 | Cassandra 4.0.7 | CQL spec 3.4.5 | Native protocol v5] cqlsh>

Below is the error Log while running run.sh.

root@:/opt/cassandra-ttl-remover# sh run.sh SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.util.ServiceConfigurationError: com.instaclustr.cassandra.ttl.SSTableTTLRemover: com.instaclustr.cassandra.ttl.Cassandra4TTLRemover Unable to get public no-arg constructor at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582) at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265) at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.getTTLRemover(TTLRemoverCLI.java:134) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.run(TTLRemoverCLI.java:100) at picocli.CommandLine.executeUserObject(CommandLine.java:1919) at picocli.CommandLine.access$1100(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332) at picocli.CommandLine$RunLast.handle(CommandLine.java:2326) at picocli.CommandLine$RunLast.handle(CommandLine.java:2291) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159) at picocli.CommandLine.execute(CommandLine.java:2058) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.execute(TTLRemoverCLI.java:119) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.main(TTLRemoverCLI.java:77) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.main(TTLRemoverCLI.java:73) Caused by: java.lang.NoClassDefFoundError: org/apache/cassandra/db/lifecycle/ILifecycleTransaction at java.base/java.lang.Class.getDeclaredConstructors0(Native Method) at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) at java.base/java.lang.Class.getConstructor0(Class.java:3342) at java.base/java.lang.Class.getConstructor(Class.java:2151) at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:660) at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:657) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:668) ... 23 more Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.db.lifecycle.ILifecycleTransaction at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)

================================================================= echo $JAVA_HOME /opt/zulu11.60.19-ca-jdk11.0.17-linux_x64 root@# echo $CASSANDRA_HOME

root@:/opt/cassandra-ttl-remover# echo $CLASSPATH /opt/cassandra-ttl-remover-1.1.2/impl/target/ttl-remover.jar:/opt/cassandra-ttl-remover-1.1.2/cassandra-4/target/ttl-remover-cassandra-4.jar

[kmrmanish23]

To reproduce the issue , below are the Steps.

1) git clone https://github.com/instaclustr/cassandra-ttl-remover 2) mvn clean install -DskipTests 3) Modify the Run.sh as attached. run.sh.txt

Getting the Below Error after Run:

root@:/opt/cassandra-ttl-remover# sh run.sh SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

............ Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.db.lifecycle.ILifecycleTransaction.

Kindly suggest if something is missed.

[smiklosovic]

Hi @kmrmanish23 ,

it seems like you are not propagating $CLASSPATH like here:

https://github.com/instaclustr/cassandra-ttl-remover/blob/master/run.sh#L72

You have only this:

    -cp "/opt/cassandra-ttl-remover/impl....... \

where is $CLASSPATH?

[kmrmanish23]

Thanks for the quick response . I modified the run.sh file .PFA the same . modified run.sh.txt

While running the run.sh now , A new directory is created as below with system & debug.log file. "drwxr-xr-x 2 root root 4096 Jan 31 08:55 cassandra.logdir_IS_UNDEFINED

Below is the content of the debug & system.log file .However no action is taken on the STABLES.

Kindly suggest if I missed something.

root@:/opt/cassandra-ttl-remover# cd cassandra.logdir_IS_UNDEFINED/ root@:/opt/cassandra-ttl-remover/cassandra.logdir_IS_UNDEFINED# cat system.log INFO [main] 2023-01-31 08:48:12,600 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown INFO [main] 2023-01-31 08:52:37,201 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown INFO [main] 2023-01-31 08:54:50,229 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown root@:/opt/cassandra-ttl-remover/cassandra.logdir_IS_UNDEFINED# cat debug.log INFO [main] 2023-01-31 08:48:12,600 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown INFO [main] 2023-01-31 08:52:37,201 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown INFO [main] 2023-01-31 08:54:50,229 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown root@:/opt/cassandra-ttl-remover/cassandra.logdir_IS_UNDEFINED#

[smiklosovic]

You are the most probably hitting this issue, I can not do anything about it.

https://issues.apache.org/jira/browse/CASSANDRA-17773

What is ".PFA the same" ?

What you mean by no action is taken? What is in /var/lib/cassandra/data/cycling/stripped ?

[kmrmanish23]

Really sorry for asking the basic Question regaridng the CLASSPATH.

After running the command " mvn clean install -DskipTests" , All the related jars are generated in the respective "target" folder under "impl", "buddy-agent" and Cassandra- X " folders.
Below is the extracted files & folders.

root@localhost:/opt/cassandra-ttl-remover# pwd /opt/cassandra-ttl-remover root@qolsysjci-Latitude-5530:/opt/cassandra-ttl-remover# ls -lrt total 56 -rw-r--r-- 1 root root 7749 Jan 30 12:37 README.adoc -rw-r--r-- 1 root root 4854 Jan 30 12:37 pom.xml -rwxr-xr-x 1 root root 7112 Jan 30 12:37 mvnw drwxr-xr-x 4 root root 4096 Jan 30 12:38 buddy-agent drwxr-xr-x 4 root root 4096 Jan 30 12:38 cassandra-2 drwxr-xr-x 4 root root 4096 Jan 30 12:38 cassandra-3 drwxr-xr-x 4 root root 4096 Jan 30 12:38 cassandra-4 drwxr-xr-x 4 root root 4096 Jan 30 12:38 cassandra-4.1 drwxr-xr-x 4 root root 4096 Jan 31 13:20 impl drwxr-xr-x 2 root root 4096 Jan 31 14:06 cassandra.logdir_IS_UNDEFINED -rwxr-xr-x 1 root root 2732 Jan 31 14:58 run.sh root@qolsysjci-Latitude-5530:/opt/cassandra-ttl-remover#

My CASSANDRA_HOME=/home/manish/cassandra/apache-cassandra-4.0.7 JAVA_HOME=/opt/zulu11.60.19-ca-jdk11.0.17-linux_x64

What should be my "$CLASSPATH" value?

Currently it set to

echo $CLASSPATH /opt/cassandra-ttl-remover-1.1.2/impl/target/ttl-remover.jar:/opt/cassandra-ttl-remover-1.1.2/cassandra-4/target/ttl-remover-cassandra-4.jar

[kmrmanish23]

You are the most probably hitting this issue, I can not do anything about it.

https://issues.apache.org/jira/browse/CASSANDRA-17773 This link is not working .

What is ".PFA the same" ? Please find attached . I have attached the modified run.sh .

What you mean by no action is taken? What is in /var/lib/cassandra/data/cycling/stripped ? This folder is empty . this is for cresting the copy of SStables as the output.

[kmrmanish23]

Based on the issue reported in https://issues.apache.org/jira/browse/CASSANDRA-17773

I changed the CASSANDRA_LOG_DIR to point to "/var/log/cassandra" .Rest looks fine .

getting below error now :

root@0:/var/log/cassandra# cd /opt/cassandra-ttl-remover/ root@:/opt/cassandra-ttl-remover# sh run.sh INFO [main] 2023-01-31 17:14:13,099 JarManifestVersionProvider.java:57 - ttl-remove version: ttl-remove development build, Build time: unknown, Git commit: unknown Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.cassandra.io.sstable.Descriptor.fromFilenameWithComponent(Descriptor.java:301) at org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:227) at org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:205) at com.instaclustr.cassandra.ttl.Cassandra4TTLRemover.executeRemoval(Cassandra4TTLRemover.java:45) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.run(TTLRemoverCLI.java:101) at picocli.CommandLine.executeUserObject(CommandLine.java:1919) at picocli.CommandLine.access$1100(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332) at picocli.CommandLine$RunLast.handle(CommandLine.java:2326) at picocli.CommandLine$RunLast.handle(CommandLine.java:2291) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159) at picocli.CommandLine.execute(CommandLine.java:2058) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.execute(TTLRemoverCLI.java:119) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.main(TTLRemoverCLI.java:77) at com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI.main(TTLRemoverCLI.java:73) Caused by: java.lang.NullPointerException at org.apache.cassandra.config.DatabaseDescriptor.getNonLocalSystemKeyspacesDataFileLocations(DatabaseDescriptor.java:1920) at org.apache.cassandra.db.Directories.(Directories.java:96) ... 15 more

[smiklosovic]

Hi @kmrmanish23 ,

why do not you use the run.sh script and modify only what you need?

All you need to change is this:

https://github.com/instaclustr/cassandra-ttl-remover/blob/master/run.sh#L61

and this

https://github.com/instaclustr/cassandra-ttl-remover/blob/master/run.sh#L63

Why do you care what is CLASSPATH set to? That is taken care of automatically based on your CASSANDRA_HOME. You need to set CASSANDRA_HOME and CLASSPATH will be set automatically.

I tried the script on the local installation of Cassandra, my Cassandra is in /tmp/c/cassandra and my stripped sstables will be saved in /tmp/stripped

So the command looks like this (BUT YOU NEED TO EXECUTE WHOLE run.sh SCRIPT!!!!)

java -Dcassandra.storagedir=$CASSANDRA_HOME/data -Dcassandra.config=file:///$CASSANDRA_HOME/conf/cassandra.yaml \
    -cp "$CLASSPATH./impl/target/ttl-remover.jar:./cassandra-4.1/target/ttl-remover-cassandra-4.1.jar" \
    $JVM_OPTS \
    com.instaclustr.cassandra.ttl.cli.TTLRemoverCLI \
    --cassandra-version=4 \
    --sstables \
    /tmp/c/cassandra/data/data/test/test-cfb929d0a16911eda1626bfeab53946b \
    --output-path \
    /tmp/stripped \
    --cql \
    'CREATE TABLE IF NOT EXISTS test.test (id uuid, name text, surname text, PRIMARY KEY (id)) WITH default_time_to_live = 10;'

Notice --sstables and --output-path That is all I changed! run.sh script takes care of the rest based on what your CASSANDRA_HOME is.

I am using Cassandra 4.2-SNAPSHOT (4.1 is compatible with that).

If you use Cassandra 4.0.7, you need to comment java command for 4.1 and you need to uncomment this

https://github.com/instaclustr/cassandra-ttl-remover/blob/master/run.sh#L69-L81

Then you need to change --stables and --output-path as I showed above.

If you run Cassandra 4, then you need to modify this

https://github.com/instaclustr/cassandra-ttl-remover/blob/master/run.sh#L72

to reflect Cassandra 4, so change jars instead of 3 to 4.

[kmrmanish23]

Thanks For the Update .

For the Cassandra 3 & 4 , there is no option to set CASSANDRA_HOME inside the run.sh. the same is there for version 4.1 and 2. If I am running without setting the CLASSPATH , its throwing error to set the CLASSPATH.

see the below code from run.sh.

if [ -z "$CLASSPATH" ]; then echo "You must set the CLASSPATH var" >&2 exit 1 fi

Let me try out this again.

[kmrmanish23]

I re-installed the entire cassandra with version 4.1 and used the same run.sh file now .

the script existed now after reading the cassandra.yaml file .Please find below the logs.

INFO [main] 2023-02-01 11:06:20,430 DatabaseDescriptor.java:460 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO [main] 2023-02-01 11:06:20,431 DatabaseDescriptor.java:514 - Global memtable on-heap threshold is enabled at 1987MiB INFO [main] 2023-02-01 11:06:20,431 DatabaseDescriptor.java:518 - Global memtable off-heap threshold is enabled at 1987MiB INFO [main] 2023-02-01 11:06:20,431 DatabaseDescriptor.java:585 - Native transport rate-limiting disabled. INFO [main] 2023-02-01 11:06:20,596 FBUtilities.java:166 - InetAddress.getLocalHost() was used to resolve listen_address to qolsysjci-Latitude-5530/127.0.1.1, double check this is correct. Please check your node's config and set the listen_address in cassandra.yaml accordingly if applicable.

Please note that there is no error message in the Log .It simply existed without creating any files in the "/tmp/stripped" folder.

[kmrmanish23]

Please note that listen_address: localhost in the cassandra.yaml file.

Do i need to change it ?

[smiklosovic]

I run run.sh as described against Cassandra 4.1 and all just went fine and SSTables are stripped from ttls.

My modified run.sh is here

https://gist.github.com/smiklosovic/358bc53ed7de6b6599f5a336228e93e8

CASSANDRA_HOME is set to /tmp/c/cassandra by export CASSANDRA_HOME=/tmp/c/cassandra in the shell.

The script is meant to be run from the repository's root directory.

[kmrmanish23]

Hi Štefan Miklošovič,

Thanks a lot for your guidance. The tool is finally working for me.

Coming to the tool functionality, I have a single Table in Cassandra spread across 10 nodes. So , I have to run this for individual SStables on all the 10 nodes ?

Thanks, Manish

[smiklosovic]

:+1: Yes, obviously.