Surface error more clearly for `push_validity`

[kszlim]

When this function panics, it's not clear exactly what is being pushed that violates its invariants. Ideally you'd see what arrow field is failing and what the data looks like.

[chmp]

Thanks a lot for the report! The error messages are definitely something to improve.

If you have a concrete example, it would be quite helpful, so I can directly include it in the test suite.

[kszlim]

I worked around the issue I encountered by mapping my schema directly, but basically the issue is that if tracing determines the schema incorrectly, there's not really a good way to determine what about the inferred schema clashes with your data that you're trying to push. You can either encounter this issue or one of the panics on the SimpleSerializer for unimplemented methods.

[chmp]

My current plan is to include the relevant fields (and potentially other metadata) in the error message, e.g.,

Cannot push null for non-nullable array (field: "$.nested.field")

Would this have helped?

[kszlim]

Yeah something like that would help immensely