Open allisonport-db opened 11 months ago
Hello @allisonport-db , I would like to take this issue. I am new to the Delta and worked on Spark/Java earlier.
Can I request you to guide me on this ?
Hi @nitinmahawadiwar, currently DataTypeParserSuite.scala
has test cases for JSON string
to DataType
. We need to test the other direction DataType
to JSON string
. One way is to modify the tests in the suite like below. Basically make the tests round trip "JSON string" -> DataType -> "JSON string"
- private def checkDataType(dataTypeString: String, expectedDataType: DataType): Unit = {
+ private def checkDataTypeRoundTrip(dataTypeString: String, expectedDataType: DataType): Unit = {
test(s"parseDataType: parse ${dataTypeString.replace("\n", "")}") {
- assert(parse(dataTypeString) === expectedDataType)
+ val dataType = parse(dataTypeString)
+ assert(dataType === expectedDataType)
+ assert(dataType.toJson === dataTypeString)
}
}
Let me know if you any further qns. At the end of this, we may want to rename DataTypeParser
to DataTypeSerDe
, but lets get the tests added first.
Hi @vkorukanti , thanks for briefing me. I am going through the project setup & code understanding in my local environment. I will connect here with you again.
Feature request
Which Delta project/connector is this regarding?
Overview
All of our
DataType
classes providetoJson
which serializes the type following the delta protocol schema serialization rules https://github.com/delta-io/delta/blob/master/PROTOCOL.md#schema-serialization-format. We use this method to serialize the physical and logical schemas inScanStateRow
.toJson
is only tested by the 2 round-trip tests inDefaultJsonHandlerSuite
. We should add more thorough tests fortoJson
.