[WIP] Write support

[sadikovi]

This is WIP, to test and discuss the API and implementation.

Once design has been approved, I will split work into small chunks and submit them separately.

PR adds low-level write support with the following features:

• Writing pretty much any schema, as long as column writers are provided correct values, definition and repetition levels.
• Support for writing data pages v1 and data pages v2.
• All encodings that are supported for reading are supported in writes.
• All compression levels that are supported for reading are supported in writes.

Limitations:

• Statistics are not supported for writes. This includes sort orders (which is part of statistics API).
• New logical type is not supported, we just write None. But for that we will need to update format first.

Testing:

• Not a lot was done for testing, I was just hacking it to make sure that some basic stuff works.
• Tested manually, I added a simple binary to try out the API and write small files, including empty ones. Tested against our parquet-schema + parquet-read to check metadata and records + Spark.

[coveralls]

Pull Request Test Coverage Report for Build 535

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 309 unchanged lines in 7 files lost coverage.
• Overall coverage decreased (-8.1%) to 87.108%

---

Files with Coverage Reduction	New Missed Lines	%
encodings/encoding.rs	1	94.64%
errors.rs	2	25.0%
file/reader.rs	13	96.64%
util/io.rs	28	60.0%
column/page.rs	39	61.76%
schema/types.rs	72	90.41%
file/metadata.rs	154	45.82%
<!--	Total:	309		-->

Totals
Change from base Build 533:	-8.1%
Covered Lines:	10473
Relevant Lines:	12023

---

💛 - Coveralls

[sunchao]

Thanks @sadikovi ! Very nice work 👍 . I'll try to allocate some time to review this soon.

[sadikovi]

It would be good to check the copy/clone calls that I make in the code.

[sadikovi]

I think it might be worth spending some time and building the code to write Row into Parquet, so we can test write-read roundtrip.

[sunchao]

Hi @sadikovi . Sorry for the delay. I think the WIP PR looks great! Could you split it up into several sub-PRs and submit them for separate review?

It would be good to check the copy/clone calls that I make in the code.

Will check this in the sub-PRs.

I think it might be worth spending some time and building the code to write Row into Parquet, so we can test write-read roundtrip.

Hmm not sure if this is necessary, since we already have encoder level round-trip tests. Also, we can do round-trip test for column reader/writer - will that be enough?

[sadikovi]

Thanks. I will outline and link issues on what needs to be completed. Yes, column reading/writing should be enough.