Open radekosmulski opened 1 year ago
I think this is because repartition
creates a brand new Dataset
object which then tries to infer a schema from the raw data all over again, but it shouldn't be too hard to maintain the existing schema in this case.
Does it work if you supply schema=self.schema
as a Dataset
constructor argument in the definition of repartition
?
@radekosmulski any update based on Karl's comment above? thanks.
I think @sararb fixed this issue in #192
@radekosmulski can you pls test this again with the latest branches pulled and see if this issues was fixed or not? Sara made a fix but not sure it solves your issue as well.
Reproducer code: