Avoiding repartitions when not transforming keys

[madstap]

Problem

When using ::willa/xform kafka streams will always repartition the topic, since we might have changed the key. There should be some way of transforming only the values, to make it possible to avoid repartitioning.

Suggested solution

Add ::willa/xform-values which takes a transducer of the shape (map (fn [[k v]] v)).

[kschltz]

I believe fn signature would be better like (fn[v]...), then willa could compose it in something like (fn[[k v]] [k (your-fn v)])

[madstap]

I left the key in as an argument because it seems useful to be able to know the key inside the function. In my current use-case the key is also a field in the value, so I don't strictly need it. The reasoning behind including it is that if we do it is easy to ignore, but if we don't include it, there's nothing we can do if we need it.

An alternative could be to have 2 different ones, (map (fn [v] v)) and (map (fn [[k v]] v)), which would also cover both use-cases.

[DaveWM]

hey @madstap, apologies for taking so long to respond - I only just noticed this issue :man_facepalming:.

I am aware of this issue, but actually I'd like to avoid adding this functionality to Willa if possible. Although it would allow you to avoid unnecessary repartitions, it comes at the cost of making Willa's API larger and requiring the user to have more knowledge of the underlying Kafka Streams library. My aim is to keep Willa's API as simple as possible, so I decided to leave this functionality out. However I'm making the assumption that unnecessary repartitions are a minor annoyance rather than a serious problem, if that's not the case for you please let me know and I'll have a think about how Willa could support this.

For now, if you need to call map-values there is an (undocumented) "escape hatch" built in to Willa. Instead of calling build-topolgy!, you can call build-topology-unsafe! which accepts an extra parameter called overrides. This is a map of entity id -> builder fn, where the builder function is responsible for calling methods on the KStreamBuilder object. It takes in 4 arguments, but I think you'll only need the first 2: entity (the entity map defined in your topology) and builder (the KStreamBuilder object). In your case, you should be able to add an override for the relevant :kstream entity, which will create the KStream object and then call map-values on it.

[DaveWM]

I'll close this issue for now, if you really need this functionality please let me know and I'll reopen the issue

[DaveWM]

@madstap I've added a way to do this in version 0.3.0. You can now set the willa.overrides/prevent-repartition option on the kstream entity to prevent KS creating a repartition.