search

kite-sdk / kite

Kite SDK
http://kitesdk.org/docs/current/
Apache License 2.0
394 stars 263 forks source link

CDK-1016: Fix OutputFormat writing directly to datasets. #379

Closed rdblue closed 9 years ago

rdblue commented 9 years ago

This happens only when a dataset instance is passed to the configuration methods. The fix is to verify that the target dataset is not the root by inspecting whether the partition key has any values.

rdblue commented 9 years ago

@joey or @tomwhite could you review this for the 1.1.0 release? Thanks!

tomwhite commented 9 years ago

Where did you see this happening? It looks fine to me, but probably could do with a test.

rdblue commented 9 years ago

This was causing replace to fail because the tasks were writing directly to the dataset instead of the job dataset that replaces partitions.

rdblue commented 9 years ago

Thanks Tom! I've merged this for now. #381 has tests that catch this case, which is how I found it. I'll merge those shortly.