search

tpilz / lumpR

Landscape Unit Mapping Program for R
GNU General Public License v3.0
11 stars 12 forks source link

database operations are slow #33

Closed tpilz closed 6 years ago

tpilz commented 7 years ago

Database operation using RODBC are very slow which becomes significant when processing large amounts of datasets. I also noticed that on Linux it is slower than on Windows (for whatever reason) and there are also differences regarding the employed DBMS (e.g., sqlite is slower than MariaDB/MySQL).

There are a few discussions around regarding this issue. However, it seems to be necessary to employ a different R package (and adapt lumpR accordingly) to speed up database processing.

A solution could be the package RJDBC, see http://stackoverflow.com/questions/30943748/r-painfully-slow-read-performance-using-rodbc-sql-server.

tpilz commented 7 years ago

... maybe a workaround with package dplyr is possible as well. But not yet sure if it will be possible to integrate MS Access as well.

For use of dplyr with databases see https://cran.r-project.org/web/packages/dplyr/vignettes/databases.html

tpilz commented 7 years ago

Solved for SQLite by commit 443b77b88dd40981474596c43eb57b451e8f24e2