Open M-JULIANI opened 3 months ago
just chipping in, I get the same error using the all-minilm
ollama embedder and 384 dimensions. the pdf is an invoice so contains a simple table and text
@alez007 Could you add more context, eg. code example or something.
Insert this string into a text
column to reproduce:
value = "\x1c\x00Þé`\U00104103Æ\U000eadac\U00018d30I"
I'm moving this issue to the core-integrations repository because I think we should address this problem in the integration and not at the Haystack level, at least initially.
Describe the bug
Running a simple embedding/indexing pipeline based on the pgvector integration (example).
Having no problem when running
markdown
files, when running the pipeline throughpdf
files, I get the following error on some pdfs:psycopg.DataError: PostgreSQL text fields cannot contain NUL (0x00) bytes
.I wonder whether you all have come across this before and whether this might be a common case to solve for in
document_writer
.I added a patch to that file to replace null bytes before attempting to write to
pg vector
Error message
psycopg.DataError: PostgreSQL text fields cannot contain NUL (0x00) bytes
.Expected behavior Would expect the pipeline to write valid content to the db, and perhaps ignore null bytes.
Additional context The PDF includes images, tables, etc. This is the type of PDF I will likely be ingesting, as opposed to a super clean and simple one. The PDF is 28MB.
To Reproduce
FAQ Check
System: