Memory leak issue

[scampi]

I use Virtuoso v07.20.3212

I am experiencing memory leaks which cause the server hosting the endpoint to crash. I have set the parameters NumberOfBuffers and MaxDirtyBuffers for a server of 15Gb.

However, the memory consumption goes way beyond. This is confirmed by setting the previous parameters to work with 8Gb, and I do see the memory consumption steadily increasing.

The logs do not show anything relevant, although I have run trace_on('errors');.

Specifically, I am sending queries from a multi-threaded application (4 threads), each query being a simple lookup about the attributes of a resource.

[scampi]

This is the output of status();:

OpenLink Virtuoso  Server
Version 07.20.3212-pthreads for Linux as of Mar 13 2015 
Started on: 2015-03-13 14:28 GMT+0

Database Status:
  File size 0, 2007296 pages, 803232 free.
  680000 buffers, 452427 used, 2 dirty 0 wired down, repl age 0 0 w. io 0 w/crsr.
  Disk Usage: 452558 reads avg 0 msec, 0% r 0% w last  0 s, 487 writes flush          0 MB,
    3187 read ahead, batch = 132.  Autocompact 0 in 0 out, 0% saved.
Gate:  4329 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap. 
Log = /spaziodati-triplestore/virtuosodb/var/lib/virtuoso/db/virtuoso.trx, 5700 bytes
1202808 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 3 connects, max 2 concurrent
RPC: 16 calls, 0 pending, 1 max until now, 0 queued, 0 burst reads (0%), 0 second 0M large, 33M max
Checkpoint Remap 1178 pages, 0 mapped back. 0 s atomic time.
    DB master 2007296 total 803232 free 1178 remap 0 mapped back
   temp  256 total 251 free

Lock Status: 0 deadlocks of which 0 2r1w, 1 waits,
   Currently 1 threads running 0 threads waiting 0 threads in vdb.
Pending:

Client 1111:2:  Account: dba, 112 bytes in, 256 bytes out, 0 stmts.
PID: 21846, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 0 threads.
Locks: 

Client 1111:3:  Account: dba, 203 bytes in, 256 bytes out, 1 stmts.
PID: 24211, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks: 

Running Statements:
 Time (msec) Text
          76 status()

Hash indexes

[scampi]

Some information about the memory consumption: http://pastebin.com/ZCZzCmuf Htop outputs the following VIRT: 9284M and RES: 8144M.

[HughWilliams]

@scampi: What is the triple count of the Virtuoso database in use ? Are you querying via the Virtuoso SPARQL endpoint i.e. http://hostname/sparql or other interface ?

If a copy of the database can be provided and querying via the SPARQL endpoint then a HTTP recording can be made as detailed below which can be replayed to recreate the problem:

http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksRecording#HTTP%20Recording

Note also, the following document on monitoring Virtuoso memory consumption:

http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtMonitorMemoryConsumption

[remyajose]

Does this issue solved? I am also facing a similar issue. There is a steady increase in memory conception while inserting data into Virtuoso from a multi threaded application. I am using JDBC connector for Virtuoso DB.

[scampi]

I didn't yet experimented with it, but I will report when I have time allocated to it.

[andyjenkinson]

We also see this: see issue #128

[cbdavis]

I've also been struggling with this problem for a while and have tried all the various suggestions listed here and on #128. None of these has worked for me. In my case this isn't a slow memory leak, but one that quickly gets out of control within about a minute, although it's been difficult to debug since everything may be well behaved for hours or even days.

I have NumberOfBuffers and MaxDirtyBuffers set based on the recommendation of 16 GB of memory free. Using the suggested memcheck-virtuoso.sh, I see this:

2015-11-30T17:10 VmSize: 14908244kB 419
2015-11-30T17:15 VmSize: 14908244kB 419
2015-11-30T17:20 VmSize: 45726712kB 419
2015-11-30T17:25 VmSize: 48197992kB 419

I have been able to locate a sort of "query of doom" featuring unions, property paths and subqueries, which allows me to consistently replicate the issue. I've been using this query in a template for about two years now, which means that previous versions of Virtuouso handled this ok.

select sum(?out) as ?total_MWh where {
     ?title rdf:type cat:Powerplant .
     ?title prop:Annual_Energyoutput_MWh ?out .
     {?title prop:Ownercompany <http://enipedia.tudelft.nl/wiki/Eni_Spa>}
     UNION {?title prop:Operator <http://enipedia.tudelft.nl/wiki/Eni_Spa>}
     UNION { 
          <http://enipedia.tudelft.nl/wiki/Eni_Spa> prop:Subsidiary+ ?subsidiary .
          {select * where { {?title prop:Ownercompany ?subsidiary} UNION {?title prop:Operator ?subsidiary} } }
     }
}

[cbdavis]

I'm able to avoid the memory leak when writing the query in a much more sane (SPARQL 1.1) way:

select sum(?out) as ?total_MWh where {
     ?title rdf:type cat:Powerplant .
     ?title prop:Annual_Energyoutput_MWh ?out .
     <http://enipedia.tudelft.nl/wiki/Eni_Spa> prop:Subsidiary* ?company .
     ?title prop:Ownercompany|prop:Operator ?company .
}

[HughWilliams]

@cbdavis: What Virtuoso build are you using ( virtuoso-t -? ) ? There have been a number of memory leak fixes in the latest git develop/7 branch I would suggest u test with if not already doing so ...

[cbdavis]

I just installed from the develop/7 branch and am still having a memory leak with that query. Running virtuoso-t -? reports:

Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.2-rc1.3215-pthreads as of Dec  1 2015
Compiled for Linux (x86_64-unknown-linux-gnu)

If I look at the SPARQL endpoint page, then I get:

Virtuoso version 07.20.3215 on Linux (x86_64-unknown-linux-gnu), Single Server Edition 

The server is running Ubuntu 12.04.5 LTS. I've just checked this on another computer running Ubuntu 15.10 with a fresh install of virtuoso from the develop/7 branch and using the default virtuoso.ini. With an out of the box setup I don't get the memory leak.

I am however, able to reproduce the memory leak if I comment out the following three lines on the default virtuoso.ini. This is for the fresh install on Ubuntu 15.10.

AdjustVectorSize         = 0
ThreadsPerQuery          = 4
AsyncQueueMaxThreads     = 10

These lines were also missing in the virtuoso.inion my Ubuntu 12.04 server, and probably weren't in there since I've been using the same ini file for years.

[HughWilliams]

@cbdavis: Are you able to provide a sample dataset for local recreation ?

[cbdavis]

I was able to reproduce the issue with the rdf dump here (195 MB). The prefixes needed to run the example SPARQL query above can be found here.

[HughWilliams]

@cbdavis: I have setup the dataset locally and It is the commenting out of the "AdjustVectorSize" INI file param resulting in it defaulting to 1 that it causing the excessive memory consumption, resulting in an error of the form:

*** Error 42000: VD [Virtuoso Server]TN...: Exceeded 1000000000 bytes in transitive temp memory.  use t_distinct, t_max or more T_MAX_memory options to limit the search or increase the pool

Do u get a similar error ?

Anyway we have something to look into ...

[cbdavis]

@HughWilliams: the short answer is that I'm not seeing the error in virtuoso.log.

The longer answer is that on my Ubuntu 15.10 machine with a fresh virtuoso install, once I see the memory rapidly increasing I quickly kill the process otherwise all memory (16 GB) will be consumed with the default ini settings (minus those three lines). On my Ubuntu 12.04 server I'm not seeing any errors that look like that or any errors in general that seem to coincide with the memory leak.

[HughWilliams]

@cbdavis: Thats true actually as the dev server I was testing on has 192GB RAM and about 60GB was free during my testing, so I probably hit the Virtuoso "T_MAX_memory" limit before the system memory was exhausted ...

[openlink]

We committed a fix to develop/7 that should resolve this excessive use of memory on some queries. Please build from latest tree and retry.

[HughWilliams]

@cbdavis: Note the fix allows "AdjustVectorSize = 1" to be used and the query returns in about the same time sub-second time as when set to "0" ...

[cbdavis]

@HughWilliams the query performance is fine now when both uncommenting and commenting the three lines mentioned previously.

There are some strange things happening with the aggregation when I try the two queries mentioned above. I'm assuming that they're (roughly) equivalent queries since if I change the select statement from select sum(?out) as ?total_MWh where to select * where, I get a table with the same plants and numbers although with the rows returned in a different order. When I copy/paste these results into a spreadsheet and sum ?out from both queries, I get 2.37832e+07 which is the same as is returned by the second query.

The first query results in double counting (4.75458e+07) when the AdjustVectorSize, ThreadsPerQuery and AsyncQueueMaxThreads are uncommented, and triple counting (7.13084e+07) when these are commented out.

[HughWilliams]

@cbdavis: We are looking into this ...