Closed yogesh-0586 closed 5 years ago
On spark-shell running DML query run fast, but on spark-submit running in yarn and cluster mode it took almost 2-3 minutes for single query
This isn't a bug but more of a performance issue, behaviour is very different between a spark shell and spark submit depending on how you are using spark submit
Best to add some form of telemetry to this ticket to validate that it's the connector not yarn or spark adding this overhead
@samelamin Thanks for your reply, I found that where was issues, it's from spark side
I tried to drop and create query using
runDMLQuery()
but for drop and create tables it takes more than 2 minutes please check following log:Is possible to create and drop table execute in less time?