Hi
I'm getting the following error in glue 3.0 and 4.0 while writing on Redshift:
"An error occurred while calling o441.count. Packet for query is too large (5,584,499 > 4,194,304). You can change this value on the server by setting the 'max_allowed_packet' variable."
Everything works on glue 2.0. The same happens to me on some S3 write jobs.
I'm suspecting the mysql driver is the problem even though I don't expect it to be used in these cases
Is there a way to set that variable via pyspark to work around the problem ?
Hi I'm getting the following error in glue 3.0 and 4.0 while writing on Redshift: "An error occurred while calling o441.count. Packet for query is too large (5,584,499 > 4,194,304). You can change this value on the server by setting the 'max_allowed_packet' variable." Everything works on glue 2.0. The same happens to me on some S3 write jobs. I'm suspecting the mysql driver is the problem even though I don't expect it to be used in these cases Is there a way to set that variable via pyspark to work around the problem ?