search

uwescience / myria

Myria is a scalable Analytics-as-a-Service platform based on relational algebra.
myria.cs.washington.edu
Other
112 stars 46 forks source link

Running postgres locally with multiple databases does not work because we use the same database #393

Closed domoritz closed 10 years ago

domoritz commented 10 years ago

A deployment with

# Deployment configuration
[deployment]
path = /tmp/myria
name = myria
dbms = postgresql
database_name = myria
database_password = ''
rest_port = 8753

# Compute nodes configuration
[master]
0 = localhost:8001

[workers]
1 = localhost:9001
2 = localhost:9002

Does not work because when we insert into the same database, it causes conflicts. 

edu.washington.escience.myria.DbException: edu.washington.escience.myria.DbException: Query #1 failed.
    at edu.washington.escience.myria.util.concurrent.OperationFutureBase.rethrowIfFailed0(OperationFutureBase.java:316)
    at edu.washington.escience.myria.util.concurrent.OperationFutureBase.sync0(OperationFutureBase.java:283)
    at edu.washington.escience.myria.parallel.DefaultQueryFuture.sync(DefaultQueryFuture.java:97)
    at edu.washington.escience.myria.parallel.Server.ingestDataset(Server.java:1137)
    at edu.washington.escience.myria.api.DatasetResource.doIngest(DatasetResource.java:294)
    at edu.washington.escience.myria.api.DatasetResource.newDataset(DatasetResource.java:265)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    at com.sun.jersey.server.impl.container.grizzly2.GrizzlyContainer._service(GrizzlyContainer.java:222)
    at com.sun.jersey.server.impl.container.grizzly2.GrizzlyContainer.service(GrizzlyContainer.java:192)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
    at org.glassfish.grizzly.http.server.HttpHandlerChain.service(HttpHandlerChain.java:196)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:175)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:815)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)
    at java.lang.Thread.run(Thread.java:724)
Caused by: edu.washington.escience.myria.DbException: Query #1 failed.
    at edu.washington.escience.myria.parallel.MasterQueryPartition$WorkerExecutionInfo$2.operationComplete(MasterQueryPartition.java:103)
    at edu.washington.escience.myria.parallel.QueryFutureListener.operationComplete(QueryFutureListener.java:43)
    at edu.washington.escience.myria.util.concurrent.OperationFutureBase.notifyListener(OperationFutureBase.java:521)
    at edu.washington.escience.myria.util.concurrent.OperationFutureBase.notifyListeners(OperationFutureBase.java:502)
    at edu.washington.escience.myria.util.concurrent.OperationFutureBase.wakeupWaitersAndNotifyListeners(OperationFutureBase.java:146)
    at edu.washington.escience.myria.util.concurrent.OperationFutureBase.setFailure0(OperationFutureBase.java:476)
    at edu.washington.escience.myria.parallel.DefaultQueryFuture.setFailure(DefaultQueryFuture.java:67)
    at edu.washington.escience.myria.parallel.MasterQueryPartition.workerFail(MasterQueryPartition.java:383)
    at edu.washington.escience.myria.parallel.Server$MessageProcessor.run(Server.java:166)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at edu.washington.escience.myria.util.concurrent.RenamingThreadFactory$1.run(RenamingThreadFactory.java:33)
    Suppressed: edu.washington.escience.myria.DbException: Worker #2 failed: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
  Detail: Key (typname, typnamespace)=(public MyriaSysTemp TwitterK, 2200) already exists.
        at edu.washington.escience.myria.accessmethod.JdbcAccessMethod.execute(JdbcAccessMethod.java:184)
        at edu.washington.escience.myria.accessmethod.JdbcAccessMethod.createTableIfNotExists(JdbcAccessMethod.java:225)
        at edu.washington.escience.myria.operator.DbInsert.init(DbInsert.java:193)
        at edu.washington.escience.myria.operator.Operator.open(Operator.java:351)
        at edu.washington.escience.myria.parallel.QuerySubTreeTask.init(QuerySubTreeTask.java:614)
        at edu.washington.escience.myria.parallel.WorkerQueryPartition.init(WorkerQueryPartition.java:229)
        at edu.washington.escience.myria.parallel.WorkerQueryPartition.init(WorkerQueryPartition.java:216)
        at edu.washington.escience.myria.parallel.WorkerShortMessageProcessor.processQueryMessage(WorkerShortMessageProcessor.java:84)
        at edu.washington.escience.myria.parallel.WorkerShortMessageProcessor.processMessage(WorkerShortMessageProcessor.java:48)
        at edu.washington.escience.myria.parallel.ipc.IPCMessageHandler.receiveRegisteredData(IPCMessageHandler.java:180)
        at edu.washington.escience.myria.parallel.ipc.IPCMessageHandler.messageReceived(IPCMessageHandler.java:246)
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
        ... 3 more
    Caused by: edu.washington.escience.myria.DbException: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
  Detail: Key (typname, typnamespace)=(public MyriaSysTemp TwitterK, 2200) already exists.
        ... 20 more
    Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
  Detail: Key (typname, typnamespace)=(public MyriaSysTemp TwitterK, 2200) already exists.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
        at edu.washington.escience.myria.accessmethod.JdbcAccessMethod.execute(JdbcAccessMethod.java:181)
        ... 19 more

Is there a way to define a different connection per worker in the config file?

dhalperi commented 10 years ago

Check out jsonQueries/scalability_valmeida/create_deployment.py. I think that's where Victor put his script to create a Postgres-enabled deployment file.

Here's a snip from the production Myria config that does this:

 % grep aldebaran myria-production.cfg.psql 
1 = aldebaran.cs.washington.edu:1778:/disk1/myria-production:myria1
19 = aldebaran.cs.washington.edu:1779:/disk2/myria-production:myria2
37 = aldebaran.cs.washington.edu:1780:/disk3/myria-production:myria3
55 = aldebaran.cs.washington.edu:1781:/disk4/myria-production:myria4
dhalperi commented 10 years ago

May be worth moving that .py file to myriadeploy and possibly reorganizing myriadeploy itself. I think it's a hodgepodge of scripts made by and modified by several different authors.

domoritz commented 10 years ago

Agreed. A single script that delegates work and does some sanity checks might be useful as well.

domoritz commented 10 years ago

Hardcoding the postgres port to 5401 doesn't seem to be a good idea anyway.

I found this comment:

        // TODO: Allow using the parameters to create the connection info.
        // Now it is hardcoded to use a specific connection info, which allows only one
        // myria instance per machine in the cluster

Sounds like it is not even possible to have multiple workers on one machine with pg (and probably mysql or monet as well).

dhalperi commented 10 years ago

There are four instances on aldebaran per the snip I sent you. They just have to use different databases. This is a single pg instance, just different databases.

On Mon, Feb 3, 2014 at 1:15 PM, Dominik Moritz notifications@github.comwrote:

Hardcoding the postgres port to 5401 doesn't seem to be a good idea anyway.

I found this comment:

    // TODO: Allow using the parameters to create the connection info.
    // Now it is hardcoded to use a specific connection info, which allows only one
    // myria instance per machine in the cluster

Sounds like it is not even possible to have multiple workers on one machine with pg (and probably mysql or monet as well).

Reply to this email directly or view it on GitHubhttps://github.com/uwescience/myria/issues/393#issuecomment-34000771 .

domoritz commented 10 years ago

For reference

WORKER_NUMBER = SERVER:PORT:DIRECTORY:DATABASE
dhalperi commented 10 years ago

(closable?)