flipp-oss / deimos

Framework to work with Kafka, Avro and ActiveRecord
Other
59 stars 22 forks source link

DB Producer can fail if all messages in batch were unique #75

Closed colinmroberts closed 3 years ago

colinmroberts commented 4 years ago

In the DB producer, if a compacted topic has a batch with all-unique keys, it will fail when calling compact_messages with this error:

NoMethodError:
       undefined method `reverse!' for nil:NilClass

E.g. consider this sample test which should pass:

      it 'should no make any changes when there are no duplicates' do
        Deimos.configure { |c| c.db_producer.compact_topics = :all }
        expect(producer.compact_messages(deduped_batch)).to eq(deduped_batch)
      end

This appears to be because compact batch is calling reverse! on uniq!. uniq! will return nil when no changes were made to the array, i.e. it was unique already. This method should use uniq instead