Open gijzelaerr opened 7 years ago
ah only now I see the absolute path error, oops. Still this is not very user friendly.
Ok so this actually only fails the first time. So if you init/connect a database with a absolute path first time, the second time you can access DB's with a relative path.
realize now this is probably related to #193 and I guess you can only operate on one db connection per Python session.
You can have multiple connections but not different db directories at the same time. In the R frontend we catch this, perhaps also a good idea for Python @Mytherin
You mention having multiple connections but not different db directories at the same time. It looks like a single R session can have multiple connections, but different R sessions (or different threads) cannot simultaneously connect to the same embedded database.
It would be useful to allow this for certain situations, such as when using threading (leaving aside the potential for data corruption if a table is modified simultaneously---that could be handled with a table lock upon modification)
Right now, multiple connections work, for example:
library(MonetDBLite)
library(DBI)
dir.create("monetdb_tmp")
dbdir <- "monetdb_tmp"
con <- ml(dbdir)
con2 <- ml(dbdir)
dbWriteTable(con, "mtcars", mtcars)
dbGetQuery(con, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8")
dbGetQuery(con2, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8")
But if you start a second R session, say in a new terminal, connecting to the database throws the error:
# in second R session
library(MonetDBLite)
library(DBI)
dbdir <- "monetdb_tmp"
con <- ml(dbdir)
# Error in monetdb_embedded_startup(embedded, !getOption("monetdb.debug.embedded", :
# Failed to initialize embedded MonetDB !FATAL: BBPaddfarm: bad rolemask
A similar thing happens if you attempt multiple connections in separate threads:
library(parallel)
cl <- makeCluster(3)
parLapply(cl = cl, 1:3, function(i, dbdir) {
library(MonetDBLite)
library(DBI)
new_con <- ml(dbdir)
out <- dbGetQuery(new_con, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8")
dbDisconnect(new_con)
out
}, dbdir = dbdir)
# 3 nodes produced errors; first error: Failed to initialize embedded MonetDB !FATAL:
# GDKlockHome: Database lock 'monetdb_tmp/.gdk_lock' denied
Note that the mclapply family functions work (with forking), probably due to the shared memory.
Multiple threads works, multiple processes does not. Its non-trivial to allow multiple process access to the same data directory. I suggest using a stand-alone MonetDB server in those cases.
I think a good start would be to give a warning or error, it is now not obvious what is happening. It is not a huge problem to have the limitation of one connection for now, as long as it is obvious.
2017-09-09 17:00 GMT+02:00 caewok notifications@github.com:
You mention having multiple connections but not different db directories at the same time. It looks like a single R session can have multiple connections, but different R sessions (or different threads) cannot simultaneously connect to the same embedded database.
It would be useful to allow this for certain situations, such as when using threading (leaving aside the potential for data corruption if a table is modified simultaneously---that could be handled with a table lock upon modification)
Right now, multiple connections work, for example:
library(MonetDBLite) library(DBI) dir.create("monetdb_tmp")dbdir <- "monetdb_tmp"con <- ml(dbdir)con2 <- ml(dbdir) dbWriteTable(con, "mtcars", mtcars) dbGetQuery(con, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8") dbGetQuery(con2, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8")
But if you start a second R session, say in a new terminal, connecting to the database throws the error:
in second R session
library(MonetDBLite) library(DBI)dbdir <- "monetdb_tmp"con <- ml(dbdir)# Error in monetdb_embedded_startup(embedded, !getOption("monetdb.debug.embedded", : # Failed to initialize embedded MonetDB !FATAL: BBPaddfarm: bad rolemask
A similar thing happens if you attempt multiple connections in separate threads:
library(parallel)cl <- makeCluster(3) parLapply(cl = cl, 1:3, function(i, dbdir) { library(MonetDBLite) library(DBI)new_con <- ml(dbdir)out <- dbGetQuery(new_con, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8") dbDisconnect(new_con)out }, dbdir = dbdir)# 3 nodes produced errors; first error: Failed to initialize embedded MonetDB !FATAL:# GDKlockHome: Database lock 'monetdb_tmp/.gdk_lock' denied
Note that the mclapply family functions work (with forking), probably due to the shared memory.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hannesmuehleisen/MonetDBLite/issues/193#issuecomment-328282501, or mute the thread https://github.com/notifications/unsubscribe-auth/AAT6pMpZ9r_RoQfYuZpKmjL0gyuSb8A5ks5sgqgegaJpZM4PN-ik .
-- Gijs Molenaar http://pythonic.nl