search

altaurog / pgcopy

fast data loading with binary copy
https://pgcopy.readthedocs.io/
Other
113 stars 33 forks source link

Help debugging, meaningless error message #31

Closed galinyc1 closed 2 years ago

galinyc1 commented 2 years ago

Thank you for all the help, I'm using your solution all the time!

My code fails in the current project on the following line (worked in many other projects): worker.copy(row, BytesIO)

And the error that I get is: ERROR: Something went wrong with XXXXXX table.

I tried reviewing all the code and PostgreSQL logs and I can't find anything.

Any recommendation, where or what to look for?

Thanks!

altaurog commented 2 years ago

Thanks for your interest in pgcopy! I am glad this library has been useful for you.

Can you provide a traceback? This error is indeed quite vague.

Can you eliminate the possibility of a data-related issue by trying with only one row? If you can get it to work for one row, can you identify specific rows/values that are problematic? I implemented once a sort of binary search that broke the data set up into successively smaller segments, running copy on each one until failing segments were all reduced to a single row, but that would only be useful if you are certain the problem is with the data (and you have a very large data set).

galinyc1 commented 2 years ago

I did so much debugging, sorry for the delay. I think I found the problem When querying the data and getting the output from deepcopy, I'm getting 6 figures for the milliseconds' value in a timestamp column. Since milliseconds should be between 0-999. The Source data type is timestamp The target data type is timestamp

And I'm getting values like: datetime.datetime(2022, 9, 6, 19, 3, 40, 209000)

To be inserted in the target.

Thank you again!

altaurog commented 2 years ago

Thanks for the update, @galinyc1 . Just curious, where was the error raised?

galinyc1 commented 2 years ago

ERROR: Something went wrong with XXXXXX table.

altaurog commented 2 years ago

That’s the error, which you sent. Where in the code was this raised? Can you send me a traceback?

galinyc1 commented 2 years ago

Sorry, I did more debugging and it was my fault. The problem was actually that the target data type was NUMERIC and I had to convert the data in python to Decimal to get it working.

The error happened when writing to the database "worker.copy(chunk, BytesIO)"

To my original request in this issue, I don't know how to produce a traceback.

altaurog commented 2 years ago

In any case, I am glad you found the solution! I’m closing this issue, feel free to re-open it if necessary.