pacman82 / arrow-odbc

Fill Apache Arrow record batches from an ODBC data source in Rust.
MIT License
52 stars 10 forks source link

feat: handle invalid UTF-8 bytes received from ODBC drivers #110

Open avhz opened 1 week ago

avhz commented 1 week ago

I am not 100% sure if this is an optimal solution, but I think it would be good to handle invalid UTF-8 from the ODBC drivers somehow, and lossily converting them was the best I could think of so far.

Or, at least not panic, as it will kill a Python session when reading from a database and the table contains some non-UTF-8 bytes.

Note: I also included some rustflags in the Cargo.toml for Apple Silicon users who've installed unixodbc via Homebrew.

Edit: I haven't tested it with the test suite yet, as the MSSQL image doesn't run on my laptop.