DataLoader interface problem for Greenplum

zhangce commented 10 years ago

It involves some code refactoring, unit tests and load function. I tested it under postgres, and Sen tested under Greenplum.

First, are you sure this runs on GP?

115 GPLOAD: 116 INPUT: 122 OUTPUT:

Why OUTPUT is in the same level of indentation with GPLOAD? YAML is indentation-sensitive. I am expecting this line gives

czhang@raiders4:~$ gpload -f gpload.yaml
2014-09-13 21:53:47|ERROR|unexpected key: "output"

DId you guys really run the code you wrote on GP? Or am I missing anything?

111 DATABASE: ${dbSettings.dbname} USER: ${dbSettings.user} HOST: ${dbSettings.host} PORT: ${dbSettings.port}

You miss the case where dbSettings.dbname matches null. See line 68 in ExtractorRunner.scala Also, shouldn't the password also be here?

139 val sql = """COPY """ + s"${tablename} FROM STDIN"

You are assuming the DB client and server are on the same machine. On line 85 of the same file, YOURSELF uses a different one: \COPY

128 val cmd = s"gpload -f ${loadyaml.getAbsolutePath()}"

Can we assume gpload is directly callable? Shouldn't there be an command line option specify where is the binary?

senwu commented 10 years ago

It involves some code refactoring, unit tests and load function. I tested it under postgres, and Sen tested under Greenplum.

First, are you sure this runs on GP?

115 GPLOAD: 116 INPUT: 122 OUTPUT:

Why OUTPUT is in the same level of indentation with GPLOAD? YAML is indentation-sensitive. I am expecting this line gives

czhang@raiders4:~$ gpload -f gpload.yaml
2014-09-13 21:53:47|ERROR|unexpected key: "output"

DId you guys really run the code you wrote on GP? Or am I missing anything?

I fixed this bug and tested it using following code: import scala.sys.process._ ("cp /lfs/local/0/senwu/deepdive_loader/app/kbp-mintz/a.txt /lfs/local/0/senwu/develop/grounding/a.txt").!! import org.deepdive.datastore.DataLoader val du = new DataLoader du.load(s"/lfs/local/0/senwu/develop/grounding/a.txt", s"ccc", settings.dbSettings, true)

The load function insert the data into table ccc successfully. Please review and double check the code.

111 DATABASE: ${dbSettings.dbname} USER: ${dbSettings.user} HOST: ${dbSettings.host} PORT: ${dbSettings.port}

You miss the case where dbSettings.dbname matches null. See line 68 in ExtractorRunner.scala

We checked the null dbname in the Deepdive.scala line 44-50 val dbSettings = settings.dbSettings dbSettings.dbname match { case "" => log.error(s"parsing dbname failed") Context.shutdown() case _ => } Also, shouldn't the password also be here?
I have no idea. Feiran, can you answer this question?

139 val sql = """COPY """ + s"${tablename} FROM STDIN"

You are assuming the DB client and server are on the same machine. On line 85 of the same file, YOURSELF uses a different one: \COPY

128 val cmd = s"gpload -f ${loadyaml.getAbsolutePath()}"

Can we assume gpload is directly callable? Shouldn't there be an command line option specify where is the binary?

feiranwang commented 10 years ago

please see my email for my answers.

rionda commented 10 years ago

I think this was fixed, can we please close it? @feiranwang @zhangce @SenWu .

feiranwang commented 10 years ago

Should have been solved in Sep.

HazyResearch / deepdive