search

snowplow-incubator / common-streams

Other
1 stars 0 forks source link

BatchUp should be compatible with ListOfList #32

Closed istreeter closed 9 months ago

istreeter commented 9 months ago

See #31 where I introduce ListOfList as being the perfect data structure for snowplow streaming apps.

Currently, our BatchUp methods required a combine method for combining smaller batches into larger batches. If we implemented this for a ListOfList then it would look something like this:

def combine(x: ListOfList[A], y: ListOfList[A]): ListOfList[A] = ListOfList.of(x.value ::: y.value)

But that does not make the best use of a ListOfList, because it uses the fairly slow ::: operator, whereas the real benefit of ListOfList is it has fast prepend.