BQ Table Schema - Maintaining Original Case of Spark DF Schema

[kurtmaile]

Hi Sam,

Is it possible to retain the original case for the BQ Table schema generated? Currently it makes all fields lowercase (customeraccountid) when the original is camel (customerAccountId) - this makes it quite hard for some of the business people to read. Is this possible at all to retain the original case?

Cheers K

[samelamin]

Hey kurt

You can probably send a pr to the schema converter It should be a small one and help you understand the code base

On Tue, 23 May 2017 at 11:52, kurtmaile notifications@github.com wrote:

Hi Sam,

Is it possible to retain the original case for the BQ Table schema generated? Currently it makes all fields lowercase (customeraccountid) when the original is camel (customerAccountId) - this makes it quite hard for some of the business people to read. Is this possible at all to retain the original case?

Cheers K

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/23, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLmwFMDA9GgCI1GWO-SDvbRoEEz3LNks5r8rpjgaJpZM4Njg0U .

[kurtmaile]

cool will try - just looking at the code nothing jumps out as where this is explicitly happening - assume its implicitly (json4s library?) happening somewhere I cant see - a pointer?

fieldToJson - would appear the where this would happen?

Cheers

[kurtmaile]

Hey Sam any hints as to why the schema casing is all lower case? Nothing obvious jumps out, I may be missing the missing the obvous too :)

[samelamin]

Hey @kurtmaile I believe its here https://github.com/samelamin/spark-bigquery/blob/master/src/main/scala/com/samelamin/spark/bigquery/converters/BigQueryAdapter.scala#L17

Give it a test locally after you do the pr to ensure it still saves to bq :)

[kurtmaile]

Oh cool thanks - what is the reason you initially made this lower case though so that I dont break things? Is there a BQ constraint to adhere to? Thanks for the tip ;)

[samelamin]

I think initially BQ didnt allow uppercase schemas but this has now been fixed, send a PR over when your free :)

[samelamin]

This has now been released on v 0.1.7

[kurtmaile]

Awesome thanks!

On Thu, 22 Jun 2017 at 22:52, Sam Elamin notifications@github.com wrote:

This has now been released on v 0.1.7

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/23#issuecomment-310512903, or mute the thread https://github.com/notifications/unsubscribe-auth/ATIvhR6M-vvTkCo1vbEDb23ORISmfxrrks5sGuH6gaJpZM4Njg0U .