[WIP] Issue #69: Support for DSV2

[karuppayya]

Whats new

1. Uses DSV2 apis of Spark 3.0
2. Leverage the ability of Redshift to unload data in Parquet format
   • Avoid row wise type conversion done for CSV. This gives good performance improvement(Benchmark yet to be done)
   • Columnar reads
3. Use the Spark's DSV2 apis of CSV, Parquet for reads. (No need for custom reader)

Note: this PR adds only read support, since DSV2 write support is broken

• [x] Re- verify after 3.0 artifacts are published to central mvn repo
• [x] This implemenation doesnt perform a count effieciently(dumps a(first) column data to a file and gets the count of number of rows.) This needs to be optimized.
• [ ] Code cleanup - Some code has been copied from DSv1 implementation. Check if they are really required
• [ ] Fix build.sbt
• [ ] Add Unit tests

[smoy]

I will start taking a look this week or next week. My organization hasn't prepped for Spark 3.0, so its in a lot of flux. I am setting up my personal development machine to check this out.

[karuppayya]

@smoy @lucagiovagnoli I have fixed compilation errors and ran tests. Given that 3.0.0 is out, can you please review the changes, when you get some cycles. We can also come up with a plan on how to make sure the code is compatible with 2.x and 3.x(currently the repo has become incompatible with my changes. We can fix them through the review process). Thanks

[gfeldman]

Anything I can do to help?

[karuppayya]

Thanks @gfeldman. Any help with the review is much appreciated.

[smoy]

close since it has been inactive for some time.