search

kroxylicious / kroxylicious-junit5-extension

JUnit5 extension and helpers for writing tests parameterised over Kafka clusters
Apache License 2.0
9 stars 10 forks source link

Kafka tmp folder cleanup #346

Open fvaleri opened 2 months ago

fvaleri commented 2 months ago

Is your feature request related to a problem? Please describe. When running tests that spin up Kafka clusters using this extension, a Kafka folder is created in the system's tmp folder (eg /tmp). For example, this is what I see after a few test runs:

$ ls -lh | grep kafka
drwx------.  3 fvaleri fvaleri   60 Jun 27 17:12 kafka10646175192143534245
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:09 kafka10731974920315473506
drwx------.  3 fvaleri fvaleri   60 Jun 27 10:56 kafka11605476241959957962
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:54 kafka11816154945662167115
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:15 kafka13022225278978666876
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:48 kafka1377296571130768089
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:52 kafka14267337757722455836
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:18 kafka14845665989614847982
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:32 kafka15128511401280566632
drwx------.  3 fvaleri fvaleri   60 Jun 27 18:16 kafka15248911086120805686
drwx------.  3 fvaleri fvaleri   60 Jun 27 18:48 kafka16171744576385138765
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:51 kafka16767144956370632249
drwx------.  3 fvaleri fvaleri   60 Jun 27 12:50 kafka17254428017328026648
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:21 kafka18036041767700724727
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:23 kafka3814424952145888592
drwx------.  3 fvaleri fvaleri   60 Jun 27 10:49 kafka4279816655104873005
drwx------.  3 fvaleri fvaleri   60 Jun 27 10:47 kafka4811556531083866635
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:16 kafka4883273034852846820
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:52 kafka6402978026332173233
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:34 kafka7183057001088643918
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:31 kafka7868694988658798292
drwx------.  3 fvaleri fvaleri   60 Jun 27 10:57 kafka8191923141184461507
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:50 kafka8538252842719455691
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:38 kafka8980994420226963248
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:22 kafka9184883345137399347
drwx------.  3 fvaleri fvaleri   60 Jun 27 11:25 kafka9308172393486580856
drwx------.  3 fvaleri fvaleri   60 Jun 27 19:00 kafka9460072113763535598

Describe the solution you'd like As default behavior, it would be good to automatically delete these folders when the test completes. An option to disable this would be good in case one needs to inspect that folder (eg dump segments).

k-wall commented 2 months ago

Agreed, I would expect the default behaviour to clean up as it goes. Are you using Kafka IN_VM or CONTAINERS mode?

fvaleri commented 2 months ago

Agreed, I would expect the default behaviour to clean up as it goes. Are you using Kafka IN_VM or CONTAINERS mode?

This is happening IN_VM mode.

SamBarker commented 2 months ago

Ok this is doubly strange we have: https://github.com/kroxylicious/kroxylicious-junit5-extension/blob/c25b46779a977b967928b7869a959962fa055c54/impl/src/main/java/io/kroxylicious/testing/kafka/invm/InVMKafkaCluster.java#L95-L96 which enables deleteOnExit and https://github.com/kroxylicious/kroxylicious-junit5-extension/blob/c25b46779a977b967928b7869a959962fa055c54/impl/src/main/java/io/kroxylicious/testing/kafka/invm/InVMKafkaCluster.java#L396-L405 which attempts to walk the folder and delete everything it created.

@fvaleri Are the test runs completing ? Do you have tests failing? Just trying to understand why the cleanup is not being executed.

fvaleri commented 2 months ago

@SamBarker @k-wall thanks for looking into this.

Are the test runs completing ?

Effectively, I doing some test refactoring and I often interrupt a running test with Ctrl-C. This is when the /tmp/kafka* dirs are left. I guess it could also happen if the Java code calls System.exit for some reason. A shutdown hook may help to avoid the issue.