search

CartoDB / odbc_fdw

PostgreSQL Foreign-data Wrapper for ODBC
Other
66 stars 22 forks source link

Interpret 01000 sqlstate as string truncation #138

Closed jgoizueta closed 3 years ago

jgoizueta commented 3 years ago

See https://app.clubhouse.io/cartoteam/story/116537

We've found a case where when reading a text field containing 178665 chars (a BigQuery Geography in WKT form), the text was being truncated to 8192 characters (MAXIMUM_BUFFER_SIZE, the chunk size in this case) because after the SQLGetData call, which was returning SQL_SUCCESS_WITH_INFO, the SQLState obtained with SQLGetDiagRec was "01000". The value used by all drivers we've been using for string truncation is "01004", but MS Docs say drivers can do whatever they wish:

Unlike return codes, the SQLSTATEs in this manual are guidelines, and drivers are not required to return them. Therefore, while drivers should return the proper SQLSTATE for any error or warning they are capable of detecting, applications should not count on this always occurring ... Because SQLSTATEs are not returned reliably ... ... When applications do base programming logic on SQLSTATEs, they should be prepared for the SQLSTATE not to be returned or for a different SQLSTATE to be returned. Exactly which SQLSTATEs are returned reliably can be based only on experience with numerous drivers.

🤦

Note that according to SQLGetData documentation 01000 is a General Warning ("Driver-specific informational message").

In the case found it seems clear that the meaning is string truncation and the full string is read if we interpret it so. So this will interpret both the usual 01004 and 01000 as string truncation. It seems unlikely that we find a driver which uses 01000 with other meaning here 🤞

jgoizueta commented 3 years ago

I looked into the Simba driver documentation and found no reference to 01000, but I need to check if there's some additional documentation

jgoizueta commented 3 years ago

Maybe it's better to restore the string truncation detection we had before #131 where in the case of SQLGetData returning SQL_SUCCESS_WITH_INFO and the SQLState being different from 01S07 (fractional truncation) we interpret it as string truncation.

jgoizueta commented 3 years ago

Hey, I'm going to merge this to test a new release package and fix this ASAP, but I still welcome CR about this, for awareness and discussing this problem.