search

databricks / spark-redshift

Redshift data source for Apache Spark
Apache License 2.0
605 stars 349 forks source link

TableName assumes PUBLIC schema when no schema is set #295

Open emmanueljob opened 7 years ago

emmanueljob commented 7 years ago

https://github.com/databricks/spark-redshift/blob/master/src/main/scala/com/databricks/spark/redshift/TableName.scala#L42

Shouldn't this follow the rules of Redshift and respect the search_path of the logged in user:

http://docs.aws.amazon.com/redshift/latest/dg/r_search_path.html

JoshRosen commented 7 years ago

Good catch. If I'm reading the linked documentation correctly, it sounds like the right fix is to pass through non-schema-qualified table names as-is without assuming PUBLIC.

Which version of spark-redshift are you using / against which releases should I target a patch?

JoshRosen commented 7 years ago

Ping @emmanueljob, would be great to know in which maintenance branches you would like this fixed.

emmanueljob commented 7 years ago

I have no specific urgent need, we can work around the issue for now. If it's in whatever your next release is whenever that is we'll take advantage of it.

sboora commented 7 years ago

@JoshRosen any chance this can be patched up in v2.0.1.

sboora commented 7 years ago

I am taking a stab at this here, https://github.com/databricks/spark-redshift/pull/324

venkatvb commented 5 months ago

We're also running into the same bug, working around it by explicitly specifying the schema name. It'll be reasonable to default to search_path if no schema is set.