Incorrectly treating binary field as text

[tired-engineer]

Issue Description

• Description of the issue: Binary fields are being passed as text when replicating from Postgres to Postgres. I've made an attempt to fix the issue, for example there: https://github.com/slingdata-io/sling-cli/commit/263ffc1d55baf4cbd5c5ee6f5fde1999be6a9182, but the rabbit hole got much deeper than I expected.

• Sling version (sling --version): 1.2.2

• Operating System (linux, mac, windows): mac, arm

• Replication Configuration: source: Crt.sh Postgres (MY_PG=postgresql://guest:@crt.sh:5432/certwatch?sslmode=disable) target: local postgres (IS_PG=...) sling run --src-conn MY_PG --src-stream public.certificate --tgt-conn IS_PG --tgt-object "public.certificate" -d -m truncate

• Log Output (please run command with -d):

2024-04-03 21:39:57 INF execution failed

Perhaps the 'transforms' source option could help with encodings? Also try `replace_non_printable`. See https://docs.slingdata.io/sling-cli/run/configuration#source

fatal:
--- sling_cli.go:418 func1 ---
--- sling_cli.go:474 cliInit ---
--- cli.go:284 CliProcess ---
~ failure running task (see docs @ https://docs.slingdata.io/sling-cli)
--- sling_logic.go:224 processRun ---
--- sling_logic.go:371 runTask ---
~ execution failed
--- task_run.go:138 Execute ---

--- task_run.go:99 func1 ---
--- task_run.go:522 runDbToDb ---
--- task_run_write.go:303 WriteToDb ---
--- database.go:2402 BulkImportFlow ---
~ could not bulk import
--- database.go:2389 func1 ---
~ could not copy data
--- database_postgres.go:211 BulkImportStream ---
~ could not execute statement
--- database_postgres.go:196 func2 ---
pq: invalid byte sequence for encoding "UTF8": 0x82

context canceled

--- task_run.go:99 func1 ---
~ Could not WriteToDb
--- task_run.go:522 runDbToDb ---
~ could not insert into "public"."certificate_tmp"
--- task_run_write.go:309 WriteToDb ---

--- task_run.go:99 func1 ---
--- task_run.go:522 runDbToDb ---
--- task_run_write.go:303 WriteToDb ---
--- database.go:2402 BulkImportFlow ---
--- database.go:2389 func1 ---
--- database_postgres.go:211 BulkImportStream ---
~ could not COPY into table "public"."certificate_tmp"
--- database_postgres.go:190 func2 ---
pq: invalid byte sequence for encoding "UTF8": 0x82

context canceled

--- task_run.go:99 func1 ---
--- task_run.go:522 runDbToDb ---
--- task_run_write.go:303 WriteToDb ---
--- database.go:2402 BulkImportFlow ---
~ could not bulk import
--- database.go:2389 func1 ---
~ could not copy data
--- database_postgres.go:211 BulkImportStream ---
~ could not execute statement
--- database_postgres.go:196 func2 ---
pq: invalid byte sequence for encoding "UTF8": 0x82

[flarco]

I appreciate the attempt! I'll look into it, thanks. For now, you could try to cast that binary column as string? Let me know if that works.

sling run --src-conn MY_PG --src-stream public.certificate \
 --tgt-conn IS_PG --tgt-object "public.certificate" -d -m truncate \
 --src-options '{"columns": { "my_col": "string" } }'

[tired-engineer]

Unfortunately, the same error happens.

[flarco]

Should be good in v1.2.3

[tired-engineer]

Thanks for the quick fix, I'd like to confirm that the issue is solved.