Performance issues reading many columns/rows simultaneously

[daviessm]

First off, I don't think this is the fault of rust-oracle. It seems to be either OCI or the way ODPI-C uses OCI but I'd appreciate your opinion.

I have an application which, using many CPU threads, reads all data from all tables in a database schema to generate a SHA256 checksum of each table. I would expect the CPU to be maxing out with the SHA256 functions but when profiling the application most of the time seems to be taken up with allocating memory for table data and freeing it after use.

I've made a small reproduction program at https://github.com/daviessm/rust_oracle_performance which demonstrates the issue. Although it discards all the data rather than hashing it, I would hope for much better performance. The flamegraph at https://raw.githubusercontent.com/daviessm/rust_oracle_performance/master/flamegraph.svg shows 28% of the runtime is in dpiGen__release and 31% is in dpiConn_newVar.

To run the program, use the parameters -d for database connection string (e.g. 127.0.0.1/orcl), -u for username and -p for password. On the first run it'll create a table called test1 which contains 50 columns and 2,000,000 rows of VARCHAR2 data, then reads from it using one thread for each CPU on the client. Subsequent runs will use the existing table.

[kubo]

Sorry for not replying for so long. The performance improves at version 0.6.1, which will be released soon.

I measured performance of https://github.com/daviessm/rust_oracle_performance by time command for version 0.5.7, 0.6.0 and 0.6.1-dev. Speed up roughly twice.

command:

cargo build
time cargo run -- --username xxx --password xxx --dbname xxx

time command output:

version 0.5.7

real    0m10.373s
user    2m19.586s
sys 0m1.388s

version 0.6.0

real    0m10.838s
user    2m24.656s
sys 0m1.450s

version 0.6.1-dev

real    0m5.438s
user    1m7.024s
sys 0m1.320s

flamegraph:

version 0.5.7

version 0.6.0

version 0.6.1-dev