search

Yolean / kafka-keyvalue

The sequel to https://github.com/Yolean/kafka-cache - over HTTP this time
Apache License 2.0
3 stars 4 forks source link

ConcurrentModificationException when an update completed out of order #19

Open solsson opened 5 years ago

solsson commented 5 years ago 
2019-03-05T07:58:53,530 DEBUG s.y-Key Got onupdate completion for topic2-0-127 previous offset 126   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:53,629 DEBUG s.y-Key Got onupdate completion for topic2-0-128 previous offset 127   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:54,229 WARN  s.y-OnU Unrecognized HTTP error for topic2-0-102[a8149657-4160-4b25-966c-a9921f19a100],5000,5000,http://target-service/kafka-keyvalue/v1/updates: MessageBodyWriter not found for media type=application/octet-stream, type=class se.yolean.kafka.keyvalue.UpdateRecord, genericType=class se.yolean.kafka.keyvalue.UpdateRecord.   [main]se.yolean.kafka.keyvalue.onupdate.OnUpdateWithExternalPollTrigger
2019-03-05T07:58:54,230 WARN  s.y-Key On-update completed out of order, topic2-0-102 before 101   [main]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:54,232 WARN  s.y-Key On-update failed for topic2-0-102[a8149657-4160-4b25-966c-a9921f19a100]   [main]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
Exception in thread "main" java.util.ConcurrentModificationException
    at java.base/java.util.LinkedHashMap$LinkedHashIterator.remove(LinkedHashMap.java:732)
    at se.yolean.kafka.keyvalue.onupdate.OnUpdateWithExternalPollTrigger.checkCompletion(OnUpdateWithExternalPollTrigger.java:93)
    at se.yolean.kafka.keyvalue.cli.Main.pollOnupdate(Main.java:87)
    at se.yolean.kafka.keyvalue.cli.Main.main(Main.java:30)
2019-03-05T07:58:54,323 DEBUG s.y-Key Got onupdate completion for topic2-0-129 previous offset 128   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:54,332 DEBUG s.y-Key Got onupdate completion for topic2-0-130 previous offset 129   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
solsson commented 5 years ago

I don't think this has anything to do with out of order updates. It's just that we add onupdate items from the streams thread and remove them from the main thread that polls onupdate.

I've seen documentation on streams API support for scheduled operations, which could be an alternative to polling form main, but I can't find it now. Anyhow there could be many threads in streams so concurrent updates wouldn't be ruled out anyway.

Using a ConcurrentHashMap would be possible, but I'd like iteration order to be that of inertion as with LinkedHashMap. Using synchronized is another option.

solsson commented 5 years ago

e7736fecdd131afe120a84a688bf8ed70ef73f03 introduced a punctuation that we might want to move polling to. There's some info on the threading model at https://docs.confluent.io/current/streams/architecture.html#threading-model