Closed AlexanderRay closed 9 years ago
There's a thread on working around this issue here: http://stackoverflow.com/questions/5839359/java-lang-outofmemoryerror-gc-overhead-limit-exceeded
I recommend trying with the GC limit turned off first, then trying one of the other options.
I don't know Scala, but FWIW in Java we've had up the heap memory in the orient server (and turn down the off-heap cache memory accordingly) to support multi-threaded clients writing to the database.
I am getting the same error in my inserts too. I am using OrientDB 2.0.1. Our application uses embedded server and access data through java API.
Thanks, Jing
more infos after some recharge:
ca. 11500 records per second. its ok. not 150.000 -) records per second. but its ok.
* second run (= at first delete the records from the first run):
after 20 minutes :
its create a huge file and breaks because of insufficient free space. ok, I have only 110 GB free on the test drive. but I don't think it is a right behavior
if it helps, I can create an java-test-project or upload my scala-test-project to github ...
Hi, I suppose you insert 10^6 Nodes in single transaction, do not you ?
On Wed, Feb 4, 2015 at 4:31 PM, Alexander Ray notifications@github.com wrote:
if it helps, I can create an java-test-project or upload my scala-test-project to github ...
— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/3517#issuecomment-72863125 .
Best regards, Andrey Lomakin.
yes. I do it in a single transaction. because I need that a database is always in a consistent state. in my workflows I need to import/create till to 10^8 records in a single transaction and be able to rollback if something going wrong ...
That is reason of huge wal file. We need to log whole tx operation as single unit. That is reason of OOM too. Could you split your tx in few smaller txs.
On Wed, Feb 4, 2015 at 4:54 PM, Alexander Ray notifications@github.com wrote:
yes.
— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/3517#issuecomment-72867713 .
Best regards, Andrey Lomakin.
the huge wal file was created only on the second run. after the Vertexes und Edges from the first run were deleted and the transaction was commited
before {
graph = factory.getTx
graph.getRawGraph.declareIntent( new OIntentMassiveInsert() );
}
after {
graph.commit()
graph.shutdown()
}
"Graph" can "remove everything" in {
graph.command(new OCommandSQL("delete Vertex Structure")).execute()
graph.command(new OCommandSQL("delete Vertex StructureNode")).execute()
graph.command(new OCommandSQL("delete Edge childNode")).execute()
graph.command(new OCommandSQL("delete Edge rootNode")).execute()
assert(true)
}
It is created when tx is commited, but before this all data is collected in heap.
On Wed, Feb 4, 2015 at 5:07 PM, Alexander Ray notifications@github.com wrote:
the huge wal file was created only on the second run. after the Vertexes und Edges was deleted and the transaction was commited
before { graph = factory.getTx graph.getRawGraph.declareIntent( new OIntentMassiveInsert() ); }
after { graph.commit() graph.shutdown() }
"Graph" can "remove everything" in { graph.command(new OCommandSQL("delete Vertex Structure")).execute() graph.command(new OCommandSQL("delete Vertex StructureNode")).execute() graph.command(new OCommandSQL("delete Edge childNode")).execute() graph.command(new OCommandSQL("delete Edge rootNode")).execute() assert(true) }
— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/3517#issuecomment-72870669 .
Best regards, Andrey Lomakin.
i see, it was created in the test "create 6-level structure" during commit.
ok. I see your point. therefore the question:
what are the transaction usage constraints, so that orientdb-engine can handle the commitment properly?
Hi Alexander, I think in 2.1 we will implement early OOM detection mechanics, if we have risk of OOM we will rollback tx will suggest to decrease amount of operations and repeat tx, it will allow to avoid puzzled OOM errors.
On Wed, Feb 4, 2015 at 5:40 PM, Alexander Ray notifications@github.com wrote:
ok. I see your point. therefore the question:
what are the transaction limits, so that orientdb-engine can handle the commitment properly?
— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/3517#issuecomment-72876952 .
Best regards, Andrey Lomakin.
ok.
Can you share the piece of code that write to the database? Are you executing SQL statements to create vertices and edges?
Lvc@
On 5 February 2015 at 09:43, Alexander Ray notifications@github.com wrote:
ok.
- what are the guidelines for using transactions?
- some tests result without transaction (graph = factory.getNoTx)
- first run [image: wotx - 1 run] https://cloud.githubusercontent.com/assets/786006/6057120/a72fa4ea-ad1a-11e4-988a-2455c0e1cf6b.png it is slower as with Tx ca. 7200 records per second
- second run [image: screen shot 2015-02-05 at 09 35 06] https://cloud.githubusercontent.com/assets/786006/6057138/d7a74466-ad1a-11e4-9033-718e0c46b4c6.png [image: screen shot 2015-02-05 at 09 34 40] https://cloud.githubusercontent.com/assets/786006/6057140/dab65c28-ad1a-11e4-97aa-5db76c2e304c.png
- more slower as the first run
- creates about 700 wal files, each ca 150 MB
- and BREAKS BECAUSE OG INSUFFICIENT FREE SPACE
— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/3517#issuecomment-73012354 .
here is the last version of the test file
function createTestClasses - database initialisation on the first run test "Graph" can "remove everything" - clear database on the second run function createStructure - creates a simple n-level tree structure
graph.commit are placed in the function createStructure and not in after (each) routine because of time measurements
if you need a complete test-project, I can cleanup it and upload to github
/**
* User: Alexander Ray
*/
package model.orientdb
import com.orientechnologies.orient.core.intent.OIntentMassiveInsert
import com.orientechnologies.orient.core.sql.OCommandSQL
import com.tinkerpop.blueprints.impls.orient.{OrientBaseGraph, OrientVertex, OrientGraphFactory, OrientGraph}
import org.scalatest._
import de.softamrhein.Logger
import scalax.file.Path
class OrientDBGraphAPISpec extends FlatSpecLike
with BeforeAndAfter
with BeforeAndAfterAll
with Matchers
with Logger
{
var factory: OrientGraphFactory = null
var graph: OrientBaseGraph = null
val dbPath = "/Users/alexanderray/Develop/orientdb/databases/test"
val createNewDatabase = false
val localDatabase = true
val useTx = false
def createTestClasses() = {
val g = factory.getNoTx
g.command(new OCommandSQL("create class Structure extends V")).execute()
g.command(new OCommandSQL("create class StructureNode extends V")).execute()
g.command(new OCommandSQL("create property StructureNode.ident STRING")).execute()
g.command(new OCommandSQL("create class rootNode extends E")).execute()
g.command(new OCommandSQL("create class childNode extends E")).execute()
g.shutdown()
}
override def beforeAll() = {
if (createNewDatabase && localDatabase) {
val path: Path = Path.fromString (dbPath)
path.deleteRecursively(continueOnFailure = false)
}
factory = new OrientGraphFactory(if (localDatabase) "plocal:/" + dbPath else "remote:localhost/test")
if (createNewDatabase && localDatabase) createTestClasses()
}
override def afterAll() = {
factory.close()
}
before {
graph = if (useTx) factory.getTx else factory.getNoTx
graph.getRawGraph.declareIntent( new OIntentMassiveInsert() );
}
after {
graph.shutdown()
}
"Graph" can "remove everything" in {
graph.command(new OCommandSQL("delete Vertex Structure")).execute()
graph.command(new OCommandSQL("delete Vertex StructureNode")).execute()
graph.command(new OCommandSQL("delete Edge childNode")).execute()
graph.command(new OCommandSQL("delete Edge rootNode")).execute()
assert(true)
}
def createStructure(ident: String, nodePrefix: String, levelCount: Int, nodesPerLevel: Int) = {
val structureVertex = graph.addVertex("class:Structure", "ident", ident)
def loop (currentLevel: Int, parentVertex: OrientVertex, edgeClass: String): Unit = {
if (currentLevel < levelCount) {
(1 to nodesPerLevel).foreach { idx =>
val nodeVertex = graph.addVertex("class:StructureNode", "ident", nodePrefix + '-' + currentLevel + '-' + idx)
parentVertex.addEdge(edgeClass, nodeVertex, edgeClass)
loop(currentLevel + 1, nodeVertex, "childNode")
}
}
}
loop(0, structureVertex, "rootNode")
if (useTx) graph.commit()
}
it can "create small 1-level structure" in {
createStructure("structure-1-level", "Node1l", 1, 10)
}
it can "create 2-level structure" in {
createStructure("structure-2-level", "Node2l", 2, 10)
}
it can "create 3-level structure" in {
createStructure("structure-3-level", "Node3l", 3, 10)
}
it can "create 4-level structure" in {
createStructure("structure-4-level", "Node4l", 4, 10)
}
it can "create 5-level structure" in {
createStructure("structure-5-level", "Node5l", 5, 10)
}
it can "create 6-level structure" in {
createStructure("structure-6-level", "Node6l", 6, 10)
}
}
Ok, seems in your loop you have too many objects in transactions (that is RAM consuming). Try rather to put this inside the loop:
if( useTx && currentLevel % 500 == 0 )
graph.commit()
In this way you commit every 500 items.
transactions in the last test were disabled (val useTx = false)
what is the difference between small transactions and no transactions? after first small transaction database is in a "dirty-inconsistent-state". and I can't properly rollback it anymore to a clean-begin-state in case on some failure ...
test with small - 500 items - transactions
new code
def createStructure(ident: String, nodePrefix: String, levelCount: Int, nodesPerLevel: Int) = {
val structureVertex = graph.addVertex("class:Structure", "ident", ident)
var cnt = 0
def loop (currentLevel: Int, parentVertex: OrientVertex, edgeClass: String): Unit = {
if (currentLevel < levelCount) {
(1 to nodesPerLevel).foreach { idx =>
val nodeVertex = graph.addVertex("class:StructureNode", "ident", nodePrefix + '-' + currentLevel + '-' + idx)
parentVertex.addEdge(edgeClass, nodeVertex, edgeClass)
cnt += 1
if( useTx && cnt % 500 == 0 )
graph.commit()
loop(currentLevel + 1, nodeVertex, "childNode")
}
}
}
loop(0, structureVertex, "rootNode")
if (useTx) graph.commit()
}
same behavior as before
It is already fixed in 2.0.2 could you try it ?
On Thu, Feb 5, 2015 at 8:14 PM, Alexander Ray notifications@github.com wrote:
test with small - 500 items
new code
def createStructure(ident: String, nodePrefix: String, levelCount: Int, nodesPerLevel: Int) = {
val structureVertex = graph.addVertex("class:Structure", "ident", ident) var cnt = 0 def loop (currentLevel: Int, parentVertex: OrientVertex, edgeClass: String): Unit = { if (currentLevel < levelCount) { (1 to nodesPerLevel).foreach { idx => val nodeVertex = graph.addVertex("class:StructureNode", "ident", nodePrefix + '-' + currentLevel + '-' + idx) parentVertex.addEdge(edgeClass, nodeVertex, edgeClass) cnt += 1 if( useTx && cnt % 500 == 0 ) graph.commit() loop(currentLevel + 1, nodeVertex, "childNode") } } } loop(0, structureVertex, "rootNode") if (useTx) graph.commit()
}
-
first run [image: screen shot 2015-02-05 at 18 44 41] https://cloud.githubusercontent.com/assets/786006/6066120/e3399c8e-ad6a-11e4-86f1-d9a3b602a0e6.png
second run [image: screen shot 2015-02-05 at 19 10 15] https://cloud.githubusercontent.com/assets/786006/6066123/ef5d7b5c-ad6a-11e4-9df9-4aa76c893746.png
same behavior as before
- creates 600 wal files for 100 GB
- breakes because of low disc space
— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/3517#issuecomment-73096709 .
Best regards, Andrey Lomakin.
without TX - same behavior as before
with small (500 records) Tx - same behavior as before
may be the test-project is linked with wrong libraries, how can I log the current library version at runtime?
with one big Tx - same behavior as before
Alexander did you try on 2.0.2 snapshot ?
It should be INFO OrientDB Server v2.0.2 is active. Did you see such message ?
You may try to download distribution from there https://www.dropbox.com/s/c7l62cd7fgdshy7/orientdb-community-2.0.2-SNAPSHOT-distribution.zip?dl=0
so, I have created a war file, it contains right (2.0.2-snapshot) libraries ..
ok. if test will fail again I will try to repeat the same on my side.
it is. I have tested it already with the snapshot from dropbox. same results ... on a second run test 6 fails because of many/huge wal files
no way, ok I will try it locally
I ran several times and had only one wal file after test ends.
ok, should I upload my test-project? or can you upload yours?
few seconds I have found reason
Could you try now same link to distribution.
which one?
also, it creates many wal files as before ... in test 5. for commit in about 100.000 records, orientdb creates about 60 wal files each ca 150MB, also about 10GB of wal files for commitment of 100.000 records. why?
test 6 runs, for now 219 wal files ...
also, nothing changes
I repeated your test and all work, it means that you use old distribution. Is it possible for you to build from sources ? branch hotfix-2.0.2
Guys, I can force upload on maven if needed.
I use that one:
ehh .. ok, i'll try it ...
it fails' don't know why ..
I have created an java-test project.
you can download it here: https://www.dropbox.com/s/v6c8ymb5n87vxk8/test-n-level-tree.zip?dl=0
It seems you never recreated database, just replaced jar. Did you ?
Hi,
I got an OutOfMemoryError while inserting about 10^6 Nodes in a OrientGraph.
orientdb.err
a test class in scala: