search

datamill-co / target-redshift

A Singer.io Target for Redshift
MIT License
23 stars 17 forks source link

CRITICAL cursor already closed / connection already closed #48

Open gbachRM opened 3 years ago

gbachRM commented 3 years ago

Hello all! I have been receiving an error while using target-redshift where the connection is established at the start of a run, but once the data has been gathered and preparing to upload the run breaks due to the connection already being closed.

CRITICAL cursor already closed
psycopg2.OperationalError: SSL SYSCALL error: EOF detected
During handling of the above exception, another exception occurred:
psycopg2.InterfaceError: cursor already closed
During handling of the above exception, another exception occurred:
psycopg2.InterfaceError: connection already closed

If anyone could shine some light on how to resolve this issue that would be great!

AlexanderMann commented 3 years ago

@gbachRM this is usually some issue with SSL settings, firewalls, keep alives, and whatnot. A good way to debug this is to get a psql client setup from the same env where you're planning to run target-redshift, then debug getting the connections happy etc.

Target Redshift should expose all of the necessary configuration options to get this working. Be sure to check the older issues in here as this comes up...a lot. This might be something you need to setup a bastion for.

connorflyn commented 2 years ago

Hi @AlexanderMann, we are again facing the error with a loader which inherits from target-redshift.

We have tried to set the max_batch_size to as low as 50. But still face the error...

psycopg2.OperationalError: SSL connection has been closed unexpectedly 
 psycopg2.InterfaceError: cursor already closed

It would appear despite changing the max_batch_size to extremely low values, it still will not load batches of data in smaller increments!

If you could shine some light on how to resolve this issue that would be great!