Closed chrislungu closed 3 months ago
ok, I wasn't thinking clearly :). this works:
copy {schema}.{table} from stdin with csv delimiter as '|' null as '\\N';
Cool, no problem.
Please take a look at psycopg 3 copy, because you can pass it Python values instead of formatting your string manually, so you shouldn't have this problem.
Please complete the following information:
Describe the bug
I am working on a ELT pipeline that uses psycopg2 as the main (and only) connection between python and my database (postgres). I am not using sqlalchemy or panda's native to_sql() as they're not fit for purpose. My data sometimes has missing values (SQL [null] equivalent), which I obviously need reflected in my staging table. From that angle, it's the most simple thing I want to do: have data, have the same data in db.
When copying the data into my staging table (unlogged, all data types text), postgres shows the Null values as a string ("Null") as opposed to the "empty" SQL standard [null]. I tried everything, including converting NaN to None in pandas and converting Nan to \N in my iterator, but whatever I do, psycopg2 assumes everything is a string. I can't input any other data types, as it will only accept strings.
copy_from() might work (haven't tried) because it allows you to specify the [null] character (null="\\N"), but this option is not available in copy_expert(). using copy_from() is also not a possibility, because it won't accept schema's and table names in one go.
how do I go around this issue?
thanks!
If possible, provide a script reproducing the issue.