CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Exception when using register as temp view if label has hyphens #29

Closed alexjbush closed 5 years ago

alexjbush commented 6 years ago

The behaviour for labels when using either the SQL action or debugAsTable is currently incorrect as it will allow you to create a label with hyphens.

Expected Behavior

Two options:

  1. Prevent any labels being created that contain hyphens
  2. Warn users if the label they choose will fail when using SQL-based actions

Actual Behavior

When using the following action and debugAsTable you get the following error:

.openFileCSV("/path/to/my/files/*.csv", "my-new-label", Map("header" -> "true")) 

//Exception:
org.apache.spark.sql.AnalysisException: Invalid view name: my-new-label;

Doing this on plain Spark gets the same exception when registering the temp views:

scala> val res = spark.read.parquet("...")
res: org.apache.spark.sql.DataFrame = [... 122 more fields]

scala> res.createOrReplaceTempView("people-ww")
org.apache.spark.sql.AnalysisException: Invalid view name: people-ww;
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$createTempViewCommand(Dataset.scala:2687)
  at org.apache.spark.sql.Dataset$$anonfun$createOrReplaceTempView$1.apply(Dataset.scala:2640)
  at org.apache.spark.sql.Dataset$$anonfun$createOrReplaceTempView$1.apply(Dataset.scala:2640)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2872)
  at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:2639)
  ... 48 elided

Specifications