search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
119 stars 1.04k forks source link

Warning: failed to write offset checkpoint file to... #2744

Open pimpelsang opened 5 years ago

pimpelsang commented 5 years ago

KSQL server in locked down mode running on docker container without permanent storage. After restart seem to get this warning...

Could it be that the parent directories for checkpoint file are not automatically created for each partition?

[2019-04-29 08:24:28,703] WARN task [0_8] Failed to write offset checkpoint file to [/tmp/kafka-streams/_confluent-ksql-proactive-ksqlquery_CTAS_STAGE_AGGR_12/0_8/.checkpoint] (org.apache.kafka.streams.processor.internals.ProcessorStateManager:327) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:910) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:804) at org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:79) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:773) at org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:286) at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:474) at org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:325) at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:1056) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.kafka.streams.processor.internals.TaskManager.commitAll(TaskManager.java:412) at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:459) java.io.FileNotFoundException: /tmp/kafka-streams/_confluent-ksql-proactive-ksqlquery_CTAS_STAGE_AGGR_12/0_8/.checkpoint.tmp (No such file or directory) at java.io.FileOutputStream.open0(Native Method)

jurgispods commented 5 years ago

I'm having the same problem. My application runs as a Kubernetes deployment of two instances. When the pod is restarted, the above error occurs.

the4thamigo-uk commented 5 years ago

We are seeing this as well

aperrot42 commented 5 years ago

also having the same issue in kubernetes deployment (containerized kafka stream job) :

2019-07-01 00:12:28,538 WARN  [peps-jobs-processors-job-7cf45f7777-knqlb] [o.a.k.s.p.i.ProcessorStateManager|checkpoint|l.327] - task [0_3] Failed to write offset checkpoint file to [/tmp/kafka-streams/enercal-gcf-peps-processors-job/0_3/.checkpoint]
java.io.FileNotFoundException: /tmp/kafka-streams/enercal-gcf-peps-processors-job/0_3/.checkpoint.tmp (No such file or directory)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:79)
    at org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:325)
    at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:474)
    at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:459)
    at org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:286)
    at org.apache.kafka.streams.processor.internals.TaskManager.commitAll(TaskManager.java:412)
    at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:1057)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:911)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
sleccese commented 5 years ago

We are seeing this problem as well. Running with Kafka 2.11_2.0.0, 2.0.0 Stream Client Version and on Kubernetes as StatefulSet with a CentOS BaseImage with OpenJDK8 task [1_9] Failed to write offset checkpoint file to /tmp/kafka-streams-global-tables/stream_no1/1_9/.checkpoint: {} j.i.FileNotFoundException: /tmp/kafka-streams-global-tables/stream_no1/1_9/.checkpoint.tmp (No such file or directory) at j.i.FileOutputStream.open0(FileOutputStream.java) at j.i.FileOutputStream.open(FileOutputStream.java:270) at j.i.FileOutputStream.<init>(FileOutputStream.java:213) at j.i.FileOutputStream.<init>(FileOutputStream.java:162) at o.a.k.s.s.i.OffsetCheckpoint.write(OffsetCheckpoint.java:78) at o.a.k.s.p.i.ProcessorStateManager.checkpoint(ProcessorStateManager.java:315) at o.a.k.s.p.i.StreamTask.commit(StreamTask.java:383) at o.a.k.s.p.i.StreamTask.commit(StreamTask.java:368) at o.a.k.s.p.i.AssignedTasks$1.apply(AssignedTasks.java:67) at o.a.k.s.p.i.AssignedTasks.applyToRunningTasks(AssignedTasks.java:362) at o.a.k.s.p.i.AssignedTasks.commit(AssignedTasks.java:352) at o.a.k.s.p.i.TaskManager.commitAll(TaskManager.java:401) at o.a.k.s.p.i.StreamThread.maybeCommit(StreamThread.java:1035) at o.a.k.s.p.i.StreamThread.runOnce(StreamThread.java:845) at o.a.k.s.p.i.StreamThread.runLoop(StreamThread.java:767) at o.a.k.s.p.i.StreamThread.run(StreamThread.java:736)

pimpelsang commented 5 years ago

I think it's related to that one https://issues.apache.org/jira/browse/KAFKA-5998

So hopefully fixed in next major release?

sm4rk0 commented 4 years ago

I'd say it's fixed. In that case this issue can be closed.

Reeebuuk commented 4 years ago

Wouldn't say it's fixed, still see it in 2.4.1