DbSource Performance Fix for Oracle Databases - set the statement fetch size

zavtech / morpheus-core

The foundational library of the Morpheus data science framework

Apache License 2.0

238 stars 22 forks source link

DbSource Performance Fix for Oracle Databases - set the statement fetch size #86

Closed dgunning closed 6 years ago

dgunning commented 6 years ago

DataFrame.read().db() is very slow against Oracle databases because the default fetch size for Oracle is 10 records. For example reading 30000 records takes over a minute instead of less than a second.

Set the statement fetch size to be at least 1000 before running the SQL query. Allow the statement fetch size to be set as a DbSourceOption

Zavster commented 6 years ago

Thanks mate, this has been addressed by the merge linked below. While I was at it, added support for auto commit and read only settings on the connection...

https://github.com/zavtech/morpheus-core/pull/87/files