palewire / django-postgres-copy

Quickly import and export delimited data with Django support for PostgreSQL's COPY command
https://palewi.re/docs/django-postgres-copy/
MIT License
180 stars 48 forks source link

No dynamic encoding for adapted params #158

Open osvill opened 1 year ago

osvill commented 1 year ago

All adapted query parameters in copy_to.py line 45 have the default encoding of latin1. This comes from the psycopg2.extensions.adapt function.

This behavior leads to an implicit error if you try something like MyModel.objects.filter(name="björn").to_csv(export_path) and your database's encoding is set to UTF8, because of the ö in björn. The generated SQL looks like: COPY (SELECT "mymodel"."id", "mymodel"."name", "mymodel"."num", "mymodel"."dt", "mymodel"."parent_id" FROM "mymodel" WHERE "mymodel"."name" = 'bj�rn ') TO STDOUT DELIMITER ',' CSV HEADER and there will be no match with name='bj�rn' for björn.

So my suggestion is to add following code to copy_to.py line 49:

48    with connections[self.using].cursor() as c:
49        # set client encoding to adapted params    # new
50        client_encoding = c.connection.encoding    # new
51        for p in adapted_params:    # new
52            p.encoding = client_encoding if client_encoding else p.encoding    # new

This will generate the right SQL: COPY (SELECT "mymodel"."id", "mymodel"."name", "mymodel"."num", "mymodel"."dt", "mymodel"."parent_id" FROM "mymodel" WHERE "mymodel"."name" = 'björn') TO STDOUT DELIMITER ',' CSV HEADER