Running 09-demo-aggregate.yaml using script results in Linux ownership issues

[tonvanbart]

Found this while trying to reproduce #35 on my private (Linux) laptop - fails with yet another exception:

Platform: Ubuntu 20.04 LTS Steps

• build ksml runner image: docker build -t axual/ksml -f Dockerfile-build-runner . --build-arg runner=ksml-runner
• start local environment: docker-compose up -d
• modify examples/ksml-runner.yml so that it points to 09-demo-aggregate.yaml
• start the local run: ./examples/run.sh

Expected result: same exception as #35 Actual result: the runner fails with the following stacktrace:

2022-04-22T20:55:17,973Z [system] [main] INFO  i.a.k.g.TopologyGeneratorImpl -
                  owner_count: retention=PT3M, key=BINARY:String, value=BINARY:Long, caching=false
Exception in thread "main" org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: state directory [/ksml/io.ksml.example.processor] doesn't exist and couldn't be created
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:807)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:782)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:692)
    at io.axual.ksml.runner.backend.kafka.KafkaBackend.<init>(KafkaBackend.java:82)
    at io.axual.ksml.runner.backend.kafka.KafkaBackendProvider.create(KafkaBackendProvider.java:44)
    at io.axual.ksml.runner.config.KSMLRunnerConfig.getConfiguredBackend(KSMLRunnerConfig.java:80)
    at io.axual.ksml.runner.KSMLRunner.main(KSMLRunner.java:57)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: state directory [/ksml/io.ksml.example.processor] doesn't exist and couldn't be created
    at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:134)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:804)
    ... 6 more
.../ksml on  main [$!?] 

[tonvanbart]

Tried to work around by trying to create this directory by hand in the examples/ subdir, gave it chmod 777 for good measure. Result: I do get the expected exception now, but also some nio exceptions before that:

2022-04-22T21:20:25,221Z [system] [main] ERROR o.a.k.s.p.internals.StateDirectory -
                Error changing permissions for the directory /ksml 
java.nio.file.FileSystemException: /ksml: Operation not permitted
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)
    at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)
    at org.apache.kafka.streams.processor.internals.StateDirectory.configurePermissions(StateDirectory.java:154)
    at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:144)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:804)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:782)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:692)
    at io.axual.ksml.runner.backend.kafka.KafkaBackend.<init>(KafkaBackend.java:82)
    at io.axual.ksml.runner.backend.kafka.KafkaBackendProvider.create(KafkaBackendProvider.java:44)
    at io.axual.ksml.runner.config.KSMLRunnerConfig.getConfiguredBackend(KSMLRunnerConfig.java:80)
    at io.axual.ksml.runner.KSMLRunner.main(KSMLRunner.java:57)

Maybe there should be some additional instructions?

[jeroenvandisseldorp]

You should probably point to a user directory somewhere, potentially relative to the working directory. If you leave out the initial '/' then Kafka Streams creates a cache directory underneath the directory where you run.

[tonvanbart]

I tried changing the value from /ksml to ksml in ksml-runner.yml (so, leaving out the initial /, however this does not work:

Exception in thread "main" io.axual.ksml.runner.exception.KSMLRunnerConfigurationException: The provided path does not exists or is not a directory
Configuration Key   : 'workingDirectory'
Configuration Value : 'ksml' 
        at io.axual.ksml.runner.config.KSMLConfig.validate(KSMLConfig.java:75)
        at io.axual.ksml.runner.config.KSMLRunnerConfig.validate(KSMLRunnerConfig.java:51)
        at io.axual.ksml.runner.KSMLRunner.main(KSMLRunner.java:55)

Is that the correct property to change?

[tonvanbart]

Looking in run.sh I see that the runner image is started like this;

docker run --name ksml-example --rm -ti -v "${BASEDIR}":/ksml -w /ksml --network ksml_example axual/ksml:latest

So it resolved the directory where run.sh resides as $BASEDIR and mounts this as /ksml in the container and sets it as the working directory. It's expectingksml-runner.yml in that location, and the runner config points to /ksml as well. The process inside the image is running as USER 1024 but this is not present on the host; this is the cause of the ownership issue.

Tried to make a workaround with the following steps:

1. create a subdirectory work/ inside examples/
2. change the ownership of that chown 1024 work
3. copy the definition to run (in this case 09-demo-aggregate.yaml into work/ OR change the definitions: entry in ksml-runner.yml to point to ../09-demo-aggregate.yaml (so one directory up from work/)
4. change workingDirectory in ksml-runner.yml to point to /ksml/work so the process gets a directory that it owns

This seems to avoid the ownership issue, however still no joy:

2022-04-26T19:50:24,257Z [system] [main] INFO  i.a.k.g.TopologyGeneratorImpl -
                Reading KSML from source file(s): [../09-demo-aggregate.yaml]
2022-04-26T19:50:24,289Z [system] [main] WARN  io.axual.ksml.avro.AvroSchemaLoader -
                Could not find/load Avro schema: SensorData
2022-04-26T19:50:24,289Z [system] [main] WARN  io.axual.ksml.avro.AvroSchemaLoader -
                Could not find/load Avro schema: SensorData
Exception in thread "main" io.axual.ksml.exception.KSMLParseException: Could not load Avro schema: SensorData
    at io.axual.ksml.avro.AvroSchemaLoader.load(AvroSchemaLoader.java:54)
    at io.axual.ksml.schema.SchemaLibrary.getSchema(SchemaLibrary.java:46)
    at io.axual.ksml.parser.UserTypeParser.parseAvroType(UserTypeParser.java:137)
    at io.axual.ksml.parser.UserTypeParser.parseTypeAndNotation(UserTypeParser.java:121)
    at io.axual.ksml.parser.UserTypeParser.parseListOfTypesAndNotation(UserTypeParser.java:77)
(...continues...)

After this copied SensorData.avsc into work/ and restarted the script --> ownership issue is solved and we're back at issue #37 .