Add inplace param to flatten operation.

ducleminh commented 8 years ago

Description and related tickets, documents

Add inplace param to flatten operation, moved flatten (array type) unit tests to DDF core.

https://adatao.atlassian.net/browse/PE-1902

Reviewers: @hai-adatao @Huandao0812 @nhanitvn @phvu

Breaking changes & backward compatible issues

Is this PR a breaking change or has backward compatible issue (e.g: changes in API names, interfaces, signature / remove something...)? If yes, please state out what and tag, and if it can possibly affect BA, please tag aht, tuananh, khangpham and baonguyen

How to test

Describe how this PR is tested. In case manual testing is required, describe how to do so.

PR Progress

Make sure all checkboxes below are checked before merged

[x] Branch is in format prefix/description (see this)
[ ] Merge check has no conflicts. PR checks passed.
[ ] Code review is done.

hai-adatao commented 8 years ago

What is the difference between flatten and flattenArrayColumn @binhmop @ducleminh ? I guess one for object type and one for array type, right? If that's the case, can you help put in some tests for flatten @ducleminh ?

ducleminh commented 8 years ago

yeah, flattenDDF applies to structtype whereas the other, as the name suggest will flatten array-type column. unit tests for flatten added. @hai-adatao

hai-adatao commented 8 years ago

It's not in the scope of this ticket, but I want to discuss 2 things, and possibly we can create tickets to track it too:

Can we move implementation of flatten to core instead of spark?
Add tests for flatten array of struct and struct of array, if we can cover these 2 then we can be confident that flatten and flattenArrayTypeColumn works most of the cases

@nhanitvn @Huandao0812

ddf-project / DDF