ODatabaseException: Database is closed when retrieve item from DB

valenpo commented 6 years ago

OrientDB Version: 2.2.33 embedded

Java Version: build 1.8.0_144-b01

OS: OSX

Expected behavior

Normally should return the Vertex. Connection to graph success, and some items are retrieved. But when working with multi-threading (import PST file with folder structure) this is not happens all the time.

graphFactory = new OrientGraphFactory(settings.getUrl(), settings.getLogin(), settings.getPassword());
        graphFactory.setupPool(1, Runtime.getRuntime().availableProcessors() * 2);
 private Vertex getVertex(ItemType itemType, IndexedProperty key, String value) throws DatabaseException, ItemNotFoundException {
        Graph graph = graphFactory.getTx();
        try {
            Iterable<Vertex> foundVertices = graph.getVertices(itemType.getClassName() + "." + key.getIndexKey(), value);
            Vertex vertex = Iterables.getFirst(foundVertices, null);
            if (vertex == null)
                throw new ItemNotFoundException("vertex not found {property='" + key + "',value='" + value + "'}");
            return vertex;
        } finally {
            graph.shutdown(); //line 313
        }
    }

Actual behavior

Some time's throw Database is closed exception when working with multithreading (pst import with folder structure).

com.orientechnologies.orient.core.exception.ODatabaseException: Database is closed
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.makeActive(OrientBaseGraph.java:309)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.pollGraphFromStack(OrientBaseGraph.java:1872)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.shutdown(OrientBaseGraph.java:1125)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.shutdown(OrientBaseGraph.java:1080)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.shutdown(OrientBaseGraph.java:1073)
    at com.stimulus.archiva.database.blueprints.BluePrintDatabase.getVertex(BluePrintDatabase.java:313)

Steps to reproduce

package com.stimulus.archiva.database.blueprints;

import com.google.common.collect.Collections2;
import com.google.common.collect.Iterables;
import com.google.common.util.concurrent.MoreExecutors;
import com.orientechnologies.orient.client.db.ODatabaseHelper;
import com.orientechnologies.orient.core.index.OIndex;
import com.orientechnologies.orient.core.metadata.schema.OClass;
import com.orientechnologies.orient.core.metadata.schema.OType;
import com.stimulus.archiva.database.ConnectionSettings;
import com.stimulus.archiva.database.DatabaseMock;
import com.stimulus.archiva.domain.Application;
import com.stimulus.archiva.domain.Database;
import com.stimulus.archiva.domain.database.TreeItem;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.impls.orient.OrientGraph;
import com.tinkerpop.blueprints.impls.orient.OrientGraphFactory;
import com.tinkerpop.blueprints.impls.orient.OrientGraphNoTx;
import com.tinkerpop.blueprints.impls.orient.OrientVertexType;
import org.junit.Assert;
import org.junit.FixMethodOrder;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.runners.MethodSorters;

import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import static org.mockito.Mockito.when;

/**
 * Created by Valentin Popov popov@mailarchiva.ru on 27/03/2018.
 */
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class BluePrintDatabaseTest {

    private static final String domain = "@domain.com";

    @Test
    public void multipleConnections() {
        OrientGraphFactory factory = new OrientGraphFactory("memory:tinkerpop");
        factory.setupPool(0, 16);
        String name = "User";
        String userName = "userName";

        OrientGraphNoTx tx = factory.getNoTx();
        try {
            OrientVertexType v = tx.createVertexType(name, "V");
            v.createProperty(userName, OType.STRING);
            OIndex<?> index = v.createIndex(userName, OClass.INDEX_TYPE.UNIQUE, userName);
            index.getDefinition().setNullValuesIgnored(true);
            tx.createEdgeType("friends", "E");
        } finally {
            tx.shutdown();
        }

        String className = "class:" + name;

        Vertex root;
        Graph graph = factory.getTx();
        try {
            root = graph.addVertex(className);
            root.setProperty(userName, "root");
        } finally {
            graph.shutdown();
        }

        ExecutorService executorService = Executors.newFixedThreadPool(64);

        for (int i = 0; i < 100; i++) {
            int friendID = i;
            executorService.submit(() -> {
                Graph graph1 = factory.getTx();
                try {
                    Vertex friend = graph1.addVertex(className);
                    String friendName = "friend" + friendID;
                    friend.setProperty(userName, "friend" + friendID);
                    graph1.addEdge("root" + friendName, root, friend, "friends");
                } catch (Exception e) {
                    e.printStackTrace();
                } finally {
                    graph1.shutdown();
                }
            });
        }

        MoreExecutors.shutdownAndAwaitTermination(executorService, 1, TimeUnit.MINUTES);
    }

}

valenpo commented 6 years ago

Here is a test case how reproduce such behavior. We have a user root, and other 100 users in same time wan't to be root user friends. Unfortunately for the user root, he will be alone. ;(

package com.stimulus.archiva.database.blueprints;

import com.google.common.collect.Collections2;
import com.google.common.collect.Iterables;
import com.google.common.util.concurrent.MoreExecutors;
import com.orientechnologies.orient.client.db.ODatabaseHelper;
import com.orientechnologies.orient.core.index.OIndex;
import com.orientechnologies.orient.core.metadata.schema.OClass;
import com.orientechnologies.orient.core.metadata.schema.OType;
import com.stimulus.archiva.database.ConnectionSettings;
import com.stimulus.archiva.database.DatabaseMock;
import com.stimulus.archiva.domain.Application;
import com.stimulus.archiva.domain.Database;
import com.stimulus.archiva.domain.database.TreeItem;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.impls.orient.OrientGraph;
import com.tinkerpop.blueprints.impls.orient.OrientGraphFactory;
import com.tinkerpop.blueprints.impls.orient.OrientGraphNoTx;
import com.tinkerpop.blueprints.impls.orient.OrientVertexType;
import org.junit.Assert;
import org.junit.FixMethodOrder;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.runners.MethodSorters;

import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import static org.mockito.Mockito.when;

/**
 * Created by Valentin Popov popov@mailarchiva.ru on 27/03/2018.
 */
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class BluePrintDatabaseTest {

    private static final String domain = "@domain.com";

    @Test
    public void multipleConnections() {
        OrientGraphFactory factory = new OrientGraphFactory("memory:tinkerpop");
        factory.setupPool(0, 16);
        String name = "User";
        String userName = "userName";

        OrientGraphNoTx tx = factory.getNoTx();
        try {
            OrientVertexType v = tx.createVertexType(name, "V");
            v.createProperty(userName, OType.STRING);
            OIndex<?> index = v.createIndex(userName, OClass.INDEX_TYPE.UNIQUE, userName);
            index.getDefinition().setNullValuesIgnored(true);
            tx.createEdgeType("friends", "E");
        } finally {
            tx.shutdown();
        }

        String className = "class:" + name;

        Vertex root;
        Graph graph = factory.getTx();
        try {
            root = graph.addVertex(className);
            root.setProperty(userName, "root");
        } finally {
            graph.shutdown();
        }

        ExecutorService executorService = Executors.newFixedThreadPool(64);

        for (int i = 0; i < 100; i++) {
            int friendID = i;
            executorService.submit(() -> {
                Graph graph1 = factory.getTx();
                try {
                    Vertex friend = graph1.addVertex(className);
                    String friendName = "friend" + friendID;
                    friend.setProperty(userName, "friend" + friendID);
                    graph1.addEdge("root" + friendName, root, friend, "friends");
                } catch (Exception e) {
                    e.printStackTrace();
                } finally {
                    graph1.shutdown();
                }
            });
        }

        MoreExecutors.shutdownAndAwaitTermination(executorService, 1, TimeUnit.MINUTES);
    }

}

and a trace

com.orientechnologies.orient.core.exception.ODatabaseException: Database is closed
    DB name="tinkerpop"
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.makeActive(OrientBaseGraph.java:306)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.addEdge(OrientBaseGraph.java:580)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.addEdge(OrientBaseGraph.java:75)
    at com.stimulus.archiva.database.blueprints.BluePrintDatabaseTest.lambda$writeRead$0(BluePrintDatabaseTest.java:80)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

luigidellaquila commented 6 years ago

Hi @valenpo

Thank you for reporting, probably the problem is due to the fact that you are sharing the same vertex (root) between threads, could you please try to share the RID only and let each thread reload the root?

Thanks

Luigi

valenpo commented 6 years ago

Hi @luigidellaquila thanks for response. I tried implement this test case, that cover more complex logic in our application that already doing reload etc...

I'll redo test case, that cover retry logic.

Thanks.

orientechnologies / orientdb