openlink / virtuoso-opensource

Virtuoso is a high-performance and scalable Multi-Model RDBMS, Data Integration Middleware, Linked Data Deployment, and HTTP Application Server Platform
https://vos.openlinksw.com
Other
853 stars 211 forks source link

Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0 #571

Open jordipala opened 8 years ago

jordipala commented 8 years ago

The error occurs with different data and at different moments of the execution. After that, the data is corrupted and I must restore from and old backup and continue from that point.

I'm using dbpedia-live-mirror and getting the following error once or twice a day.

06:28:41,733 ERROR ChangesetExecutor: Cannot ADD triple: <http://dbpedia.org/property/autor> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/1999/02/22-rdf-syntax-ns#Property> . 06:28:41,734 WARN ChangesetExecutor: Error in query execution: org.dbpedia.extraction.live.mirror.sparul.SPARULException: org.dbpedia.extraction.live.mirror.sparul.SPARULException: virtuoso.jdbc4.VirtuosoException: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0 at org.dbpedia.extraction.live.mirror.sparul.SPARULVosExecutor.execSQLWrapper(SPARULVosExecutor.java:74) at org.dbpedia.extraction.live.mirror.sparul.SPARULVosExecutor.executeSPARUL(SPARULVosExecutor.java:29) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeSparulWrapper(ChangesetExecutor.java:140) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:96) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:117) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:117) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:117) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:117) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:117) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.executeAction(ChangesetExecutor.java:117) at org.dbpedia.extraction.live.mirror.changesets.ChangesetExecutor.applyChangeset(ChangesetExecutor.java:54) at org.dbpedia.extraction.live.mirror.LiveSync.main(LiveSync.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294) at java.lang.Thread.run(Thread.java:745) Caused by: org.dbpedia.extraction.live.mirror.sparul.SPARULException: virtuoso.jdbc4.VirtuosoException: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0 at org.dbpedia.extraction.live.mirror.sparul.SPARULVosExecutor.execSQL(SPARULVosExecutor.java:88) at org.dbpedia.extraction.live.mirror.sparul.SPARULVosExecutor.execSQLWrapper(SPARULVosExecutor.java:40) ... 17 more Caused by: virtuoso.jdbc4.VirtuosoException: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0 at virtuoso.jdbc4.VirtuosoResultSet.process_result(Unknown Source) at virtuoso.jdbc4.VirtuosoResultSet.<init>(Unknown Source) at virtuoso.jdbc4.VirtuosoStatement.sendQuery(Unknown Source) at virtuoso.jdbc4.VirtuosoStatement.executeQuery(Unknown Source) at com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464) at org.dbpedia.extraction.live.mirror.sparul.SPARULVosExecutor.execSQL(SPARULVosExecutor.java:86) ... 18 more

HughWilliams commented 8 years ago

@jordipala: I assume you are running the latest Virtuoso 07.20.3217 build (virtuoso-t -?)?

The following steps should be performed to enable transaction logging and special column-store debug logging of database activity that can be replayed step-by-step to determine the cause of the error:

  1. Ensure a full online backup or copy of a known good database that can be provided is in place.

  2. Enable Virtuoso Checkpoint AuditTrail logging with the CheckpointAuditTrail = 1 param in the [Parameters] section of the INI file as detailed here.

  3. Enable column store debug logging by adding the following params to the [Flags] section of the INI file —

    [Flags]   
    dbf_col_ins_dbg_log = 1  
    enable_ce_ins_check = 2  

    — and restart the server, to enable special debug transaction logs of the insert/delete operations logged in the order of occurrence to be produced in debug transaction log files of the name —

    virtuoso.debug.trx.<timestamp>
  4. Then, at regular intervals, make online backups, and check the database in consistent with the following commands/queries:

    backup '/dev/null'; 
    
    select count (*) from rdf_quad a table option (index rdf_quad) where not exists (select 1 from rdf_quad b table option (loop, index rdf_quad_pogs) where a.g = b.g and a.p = b.p and a.o = b.o and a.s = b.s); 
  5. Once it is verified that a known-consistent database has become inconsistent or is giving the error, you can replay the debug logs against a known-good backup and verify they produce the problem. Then the last known good backup plus column store debug logs can be made available for development to replay manually to find where the problem is occurring and fix.

jordipala commented 8 years ago

I confirm I'm using the latest 07.20.3217

hroptatyr commented 3 years ago

I can confirm it's still an issue in

14:40:42 OpenLink Virtuoso Universal Server
14:40:42 Version 07.20.3233-pthreads for Linux as of Jun 22 2021
14:40:42 uses OpenSSL 1.0.2u  20 Dec 2019
14:40:42 uses parts of PCRE, Html Tidy
14:40:43 Database version 3126

and I do have a reproducible set of database and transactions that trigger this bug.

HughWilliams commented 3 years ago

@hroptatyr Are you able to provide the reproducible test case such that it can be setup in-house to debug and fix ?

TallTed commented 3 years ago

@hroptatyr -- Please provide the first stanza of command-line output of virtuoso-t -?, as this will confirm the git_head of your build (which the log snippet you provided does not).

Alternatively, you can use the SPARQL query found here to query the running instance, to get the same detail.

hroptatyr commented 3 years ago

@TallTed

Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.6.3233-pthreads as of Jun 22 2021 (111d17e5b)
Compiled for Linux (x86_64-generic_glibc25-linux-gnu)
Copyright (C) 1998-2021 OpenLink Software

@HughWilliams https://yadi.sk/d/wVqdUunHyv5xPg

HughWilliams commented 3 years ago

@hroptatyr: Is the virtuoso.db file provided a copy of the original database before the special debug transaction logging was enabled? As I would have expected this query —

SQL> select count (*) from rdf_quad a table option (index rdf_quad) where not exists (select 1 from rdf_quad b table option (loop, index rdf_quad_pogs) where a.g = b.g and a.p = b.p and a.o = b.o and a.s = b.s);
count
INTEGER
_______________________________________________________________________________

4

1 Rows. -- 16151 msec.
SQL>

— to return 0 rather than 4, which indicates the rdf_quad_pogs index has missing values compared to the rdf_quad primary key index.

hroptatyr commented 3 years ago

You're right, my bad. Attached a proper version, double checked this time.

https://yadi.sk/d/Vks4seAia9ZzEQ

hroptatyr commented 2 years ago

An update. I ran into the issue again, this time on an empty database. rdf_loader_run on evtx.ttl.gz will reproduce the issue.

OpenLink Virtuoso Universal Server (Enterprise Edition)
Version 08.03.3322-pthreads as of Jun  3 2021 (6ceb94f1f5)
Compiled for Linux (x86_64-generic-linux-glibc25)
Hosted Runtime Environments:  VDB
Copyright (C) 1998-2021 OpenLink Software

Something's afoot with the turtle reader, I converted the file to ntriples (evty.nt.gz), and loading that one works just fine. Also, removing a few statements in the evtx.ttl.gz makes the error disappear so it might be to do with some size or count limit somewhere.

evtx.ttl.gz evty.nt.gz

TallTed commented 2 years ago

@pkleef @openlink @HughWilliams -- Please take a look at this.

HughWilliams commented 2 years ago

The evtx.ttl.gz dataset loads fine for me ...

Binary:

ubuntu@ip-172-30-0-77:~$ /home/ubuntu/v83/bin/virtuoso-iodbc-t -?
OpenLink Virtuoso Universal Server (Enterprise Edition)
Version 08.03.3322-pthreads as of Jun  3 2021 (6ceb94f1f5)
Compiled for Linux (x86_64-generic-linux-glibc25)
Hosted Runtime Environments:  VDB
Copyright (C) 1998-2021 OpenLink Software

Bulk load:

SQL> ld_dir ('/home/ubuntu/v83/database', 'evtx.ttl.gz', 'urn:evtx.ttl.gz');

Done. -- 2 msec.
SQL> select * from load_list where ll_state=0;
ll_file                                                                           ll_graph                                                                          ll_state    ll_started           ll_done              ll_host     ll_work_time  ll_error                                                                          LL_OPTIONS
VARCHAR NOT NULL                                                                  VARCHAR                                                                           INTEGER     TIMESTAMP            TIMESTAMP            INTEGER     INTEGER     VARCHAR                                                                           LONG VARCHAR
_______________________________________________________________________________

/home/ubuntu/v83/database/evtx.ttl.gz                                             urn:evtx.ttl.gz                                                                   0           NULL                 NULL                 NULL        NULL        NULL                                                                              NULL

1 Rows. -- 0 msec.
SQL> rdf_loader_run();

Done. -- 5297 msec.
SQL> select * from load_list where ll_state=0;
ll_file                                                                           ll_graph                                                                          ll_state    ll_started           ll_done              ll_host     ll_work_time  ll_error                                                                          LL_OPTIONS
VARCHAR NOT NULL                                                                  VARCHAR                                                                           INTEGER     TIMESTAMP            TIMESTAMP            INTEGER     INTEGER     VARCHAR                                                                           LONG VARCHAR
_______________________________________________________________________________

0 Rows. -- 1 msec.
SQL> select * from DB.DBA.LOAD_LIST where ll_error IS NOT NULL;
ll_file                                                                           ll_graph                                                                          ll_state    ll_started           ll_done              ll_host     ll_work_time  ll_error                                                                          LL_OPTIONS
VARCHAR NOT NULL                                                                  VARCHAR                                                                           INTEGER     TIMESTAMP            TIMESTAMP            INTEGER     INTEGER     VARCHAR                                                                           LONG VARCHAR
_______________________________________________________________________________

0 Rows. -- 0 msec.
SQL> checkpoint;

Done. -- 917 msec.
SQL> sparql select count(*) from <urn:evtx.ttl.gz> where {?s ?p ?o};
callret-0
INTEGER
_______________________________________________________________________________

477809

1 Rows. -- 7 msec.
SQL>

Virtuoso log:

23:27:23 PL LOG: Loader started
23:27:28 PL LOG: No more files to load. Loader has finished,
23:28:17 Checkpoint started
23:28:18 Checkpoint finished, log reused

Thus what are the specific steps you are performing and where is the error being seen ?

Might your virtuoso.ini config file have any special settings from the default in place ?

What Linux distro and version are you using ?

hroptatyr commented 2 years ago

Thanks for testing.

It's a openSuSE 15.2

$ /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.26 (git 1c9a5c270d8b), by Roland McGrath et al.
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 7.5.0.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://bugs.opensuse.org>.

The only line differing from the default virtuoso.ini is:

[Flags]
enable_ce_ins_check = 2

which I was told to put there.

hroptatyr commented 2 years ago

Sorry, found another diff in the virtuoso.ini:

;VectorSize  = 1000

when using this setting, even if it is VectorSize = 0 or VectorSize = 1 it is working, i.e. the error does not show up.

HughWilliams commented 2 years ago

You still have not provided the steps being performed to reproduce the problem in your environment, so we can see exactly where the error is occurring when bulk loading the dataset?

With ;VectorSize = 1000 (i.e., commented out in the INI file), the default value used by Virtuoso is 10000, as per the documentation. Commenting out the setting in my INI file and restarting Virtuoso the dataset still loads successfully for me.

The enable_ce_ins_check = 2 setting in the INI file was suggested previously (along with the dbf_col_ins_dbg_log = 1 Flag) to enable a debug transaction log to be created for debugging, and is not needed for normal use. Thus, as you indicate being able to recreate the error by simply loading the evtx.ttl.gz dataset into an empty database, those INI file params do not need to be set.

Can you provide a copy of the virtuoso.ini file being used, for our review and use to try and force recreation?

hroptatyr commented 2 years ago

Sure, I'm doing my experiments in /var/tmp/vdbe/. Here's the steps

$ rm vi*; /opt/virtuoso-enterprise/bin/virtuoso +foreground +debug -c .w.ini

in a second shell:

$ isql 1112 dba dba PROMPT=OFF CSV=ON load-evts.sql
OpenLink Virtuoso Interactive SQL (Virtuoso)
Version 07.20.3229 as of Nov 27 2018
Type HELP; for help and EXIT; to exit.
Connected to OpenLink Virtuoso
Driver: 07.20.3229 OpenLink Virtuoso ODBC Driver

Done. -- 1 msec.

Done. -- 1 msec.

Done. -- 0 msec.

Done. -- 1 msec.

Done. -- 1955 msec.

Done. -- 128 msec.
ll_file;ll_graph;ll_state;ll_started;ll_done;ll_host;ll_work_time;ll_error
/var/tmp/vdbe/evtx.ttl.gz;http://data.ga-group.nl/evts/;2;2022.1.9 5:8:45.691553000;2022.1.9 5:8:47.645320000;0;;XXXXX COL..: Insert stopped because out of seg data here or elsewhere host 1 key RDF_QUAD slice 0                                                                                                                                                                                              
1 Rows. -- 0 msec.
G;S;P;O

0 Rows. -- 229 msec.

Done. -- 66 msec.

Attached the files I'm using. Had to gzip them because github wouldn't let me upload them otherwise. load-evts.sql.gz .w.ini.gz

HughWilliams commented 2 years ago

You omitted the fact that LiteMode=1 is set in your INI file which is a significant difference, as in LiteMode Virtuoso runs with minimal resource/memory foot print.

And the setting of LiteMode= in conjunction with the VectorSize INI file param being commented out with the default of 10000 then being used results in the error:

ubuntu@ip-172-30-0-77:~/v83/database/git571$ isql 1112 dba dba ECHO=ON VERBOSE=ON PROMPT=OFF CSV=ON load-evts.sql
OpenLink Virtuoso Interactive SQL (Virtuoso)
Version 08.03.3322 as of Nov 12 2021
Type HELP; for help and EXIT; to exit.
Connected to OpenLink Virtuoso
Driver: 08.03.3322 OpenLink Virtuoso ODBC Driver

-- Line 1: log_enable(3,1)

Done. -- 0 msec.

-- Line 2: SPARQL CREATE SILENT GRAPH <http://data.ga-group.nl/evts/>

Done. -- 2 msec.

-- Line 4: DELETE FROM DB.DBA.LOAD_LIST WHERE ll_file = '/home/ubuntu/v83/database/evtx.ttl.gz'

Done. -- 0 msec.

-- Line 5: ld_add('/home/ubuntu/v83/database/evtx.ttl.gz', 'http://data.ga-group.nl/evts/')

Done. -- 1 msec.

-- Line 6: rdf_loader_run()

Done. -- 3360 msec.

-- Line 7: CHECKPOINT

Done. -- 162 msec.

-- Line 8: SELECT * FROM DB.DBA.LOAD_LIST WHERE ll_state < 2 OR ll_error IS NOT NULL
ll_file;ll_graph;ll_state;ll_started;ll_done;ll_host;ll_work_time;ll_error;ll_options
/home/ubuntu/v83/database/evtx.ttl.gz;http://data.ga-group.nl/evts/;2;2022.1.9 21:3:33.36830000;2022.1.9 21:3:36.395288000;0;;XXXXX COL..: Insert stopped because out of seg data here or elsewhere host 1 key RDF_QUAD slice 0;

1 Rows. -- 1 msec.

-- Line 9: select * from rdf_quad a table option (index rdf_quad) where not exists (select 1 from rdf_quad b table option (loop, index rdf_quad_pogs) where a.g = b.g and a.p = b.p and a.o = b.o and a.s = b.s)
G;S;P;O

0 Rows. -- 267 msec.

-- Line 10: CHECKPOINT

Done. -- 111 msec.
ubuntu@ip-172-30-0-77:~/v83/database/git571$

Have you tried uncommenting the ;VectorSize=1000 setting in the INI file such that the value of 1000 is used as per the default INI file we provide, with that setting the dataset loads without the error ?

Will still have development look into why the with VectorSize commented out the error occurs, as this should not be the case.

I presume you specifically set LiteMode=1 to reduce resource/memory consumption for your use case, as it is not set by default ?

hroptatyr commented 2 years ago

Oh, it's been a while that I thoroughly studied ini file parameters, my bad. I was under the impression that LiteMode is a toggle for the sparql endpoint. I specifically set it for the testing environment to exclude another source of potential mishap.

You're right indeed that this minimal file doesn't show the error in the default setup but a much larger file (~40 Mtriples) of the same structure does.

HughWilliams commented 2 years ago

As indicated in the LiteMode link, it does a lot more than toggle/disable the /sparql endpoint, which is more a side effect than a specific use for LiteMode.

By "default setup", I presume you mean with a default INI file, i.e., without LiteMode=1 set, etc?

If the larger file (~40 Mtriples) is converted to from Turtle to N-Triples, does it then load, as you found with the indicated evty.nt.gz dataset?

Development are looking into this issue and will report back with their findings ...

hroptatyr commented 2 years ago

Yes sorry, our jargons are not in sync. Default setup is virtuoso.ini as can be downloaded from this repository here, for instance.

The ntriples version does indeed load (without error), BUT I noticed 2 missing entries in the rdf_quad_pogs index.

Thanks for looking into this.

HughWilliams commented 2 years ago

Are you able to provide the dataset with the missing entries in the rdf_quad_pogs index, so we can attempt to recreate in-house?

Also, what is the output of the rdf_quad_pogs index query, and are any errors reported in the virtuoso.log file during the load?

hroptatyr commented 2 years ago

It looks like we're chasing a Heisenbug. I've been using the file I marked as working, well apart from two missing index entries, and to my surprise it shows the out-of-seg-data error today on a pristine default.ini.

Worse, I was able to cut it to about 4% (uniform sample of lines) and (for today at least) the error keeps showing. File's attached.

Can you confirm? Is this of interest in the first place, or should I be chasing the "almost working" variant (the one that shows missing index entries)?

If it helps the missing entires are all date or datetime literals.

evts.nt.gz

HughWilliams commented 2 years ago

I see the same issue with the "evts.nt.gz" dataset loading into an empty database with default INI file:

ubuntu@ip-172-30-0-77:~/v83/database/git571$ /opt/virtuoso/bin/isql 1111 dba dba ECHO=ON VERBOSE=ON PROMPT=OFF CSV=ON load-evts.sql
OpenLink Virtuoso Interactive SQL (Virtuoso)
Version 08.03.3322 as of Nov 12 2021
Type HELP; for help and EXIT; to exit.
Connected to OpenLink Virtuoso
Driver: 08.03.3322 OpenLink Virtuoso ODBC Driver

-- Line 1: log_enable(3,1)

Done. -- 0 msec.

-- Line 2: SPARQL CREATE SILENT GRAPH <http://data.ga-group.nl/evts/>

Done. -- 2 msec.

-- Line 4: DELETE FROM DB.DBA.LOAD_LIST WHERE ll_file = '/home/ubuntu/v83/database/evts.nt.gz'

Done. -- 0 msec.

-- Line 5: ld_add('/home/ubuntu/v83/database/evts.nt.gz', 'http://data.ga-group.nl/evts/')

Done. -- 0 msec.

-- Line 6: rdf_loader_run()

Done. -- 11878 msec.

-- Line 7: CHECKPOINT

Done. -- 739 msec.

-- Line 8: SELECT * FROM DB.DBA.LOAD_LIST WHERE ll_state < 2 OR ll_error IS NOT NULL
ll_file;ll_graph;ll_state;ll_started;ll_done;ll_host;ll_work_time;ll_error;ll_options
/home/ubuntu/v83/database/evts.nt.gz;http://data.ga-group.nl/evts/;2;2022.1.12 10:14:29.432404000;2022.1.12 10:14:41.307767000;0;;XXXXX COL..: Insert stopped because out of seg data here or elsewhere host 1 key RDF_QUAD slice 0;

1 Rows. -- 0 msec.

-- Line 9: select * from rdf_quad a table option (index rdf_quad) where not exists (select 1 from rdf_quad b table option (loop, index rdf_quad_pogs) where a.g = b.g and a.p = b.p and a.o = b.o and a.s = b.s)
G;S;P;O

0 Rows. -- 603 msec.

-- Line 10: CHECKPOINT

Done. -- 46 msec.
ubuntu@ip-172-30-0-77:~/v83/database/git571$

We should also pursue the missing index entries with the complete dataset that being what you are actually seeking to load. Thus if you can provide the dataset for local recreation that would be ideal ?

holycrab13 commented 2 years ago

Hello, I am experiencing the same error. I created a debug repository that can hopefully reproduce the bug (tested on 2 machines) using the vanilla openlink/virtuoso-opensource-7 docker image.

https://github.com/dbpedia/databus-transfer/tree/insert-stopped-debug

This is the log of the virtuoso after trying to insert 2 files after running

SQL > trace_on('transact', 'errors', 'exec') ;

The second file causes the error but only if it is inserted second.

virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:4 Begin transact 0x7f2d9c028d80
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:3 Begin transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:5 Begin transact 0x7f2d780037b0
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7319
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:4 Commit transact 0x7f2d9c028d80 0 7318
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:5 Commit transact 0x7f2d780037b0 0 7320
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:5 Restart transact 0x7f2d780037b0
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:5 Begin transact 0x7f2d780037b0
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:4 Restart transact 0x7f2d9c028d80
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:3 Begin transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:4 Begin transact 0x7f2d9c028d80
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:6 Begin transact 0x7f2d6c00d2f0
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:7 Begin transact 0x7f2d9c033400
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:6 Commit transact 0x7f2d6c00d2f0 0 7324
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:6 Restart transact 0x7f2d6c00d2f0
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:6 Begin transact 0x7f2d6c00d2f0
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:7 Commit transact 0x7f2d9c033400 0 7325
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:7 Restart transact 0x7f2d9c033400
virtuoso_1  | 08:35:58 LTRS_0 0 172.30.0.2 1111:7 Begin transact 0x7f2d9c033400
virtuoso_1  | 08:35:58 EXEC_1 0 172.30.0.2 1111:3 ps00 Exec 1 time(s) sparql
virtuoso_1  | CLEAR SILENT GRAPH <http://localhost:3002/g/test/mappings-geo-coordinates-mappingbased-2018.09.12-dataid.jsonld>
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7322
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:58 EXEC_1 0 172.30.0.2 1111:3 ps01 Exec 1 time(s) sparql
virtuoso_1  | DROP SILENT GRAPH <http://localhost:3002/g/test/mappings-geo-coordinates-mappingbased-2018.09.12-dataid.jsonld>
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7328
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:58 EXEC_1 0 172.30.0.2 1111:3 ps02 Exec 1 time(s) sparql
virtuoso_1  | INSERT IN GRAPH <http://localhost:3002/g/test/mappings-geo-coordinates-mappingbased-2018.09.12-dataid.jsonld> {
virtuoso_1  | <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" .
virtuoso_1  | <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sorted> "true"^^<http
virtuoso_1  | 08:35:58 ERRS_0 01V01 QW004 Incompatible types XML_ENTITY (230) and VARCHAR (182) in := for callretSimpleCASE and blob_to_string
virtuoso_1  | 08:35:58 ERRS_0 01V01 QW004 Incompatible types XML_ENTITY (230) and VARCHAR (182) in := for callretSimpleCASE and .RO_VAL
virtuoso_1  | 08:35:58 ERRS_0 01V01 QW004 Incompatible types XML_ENTITY (230) and VARCHAR (182) in := for callretSimpleCASE and blob_to_string
virtuoso_1  | 08:35:58 ERRS_0 01V01 QW004 Incompatible types XML_ENTITY (230) and VARCHAR (182) in := for callretSimpleCASE and .RO_VAL
virtuoso_1  | 08:35:58 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7329
virtuoso_1  | 08:35:58 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:59 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7330
virtuoso_1  | 08:35:59 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:59 EXEC_1 0 172.30.0.2 1111:3 ps03 Exec 1 time(s) sparql
virtuoso_1  | CLEAR SILENT GRAPH <http://localhost:3002/g/test/mappings-mappingbased-literals-2018.12.01-dataid.jsonld>
virtuoso_1  | 08:35:59 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7331
virtuoso_1  | 08:35:59 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:59 EXEC_1 0 172.30.0.2 1111:3 ps04 Exec 1 time(s) sparql
virtuoso_1  | DROP SILENT GRAPH <http://localhost:3002/g/test/mappings-mappingbased-literals-2018.12.01-dataid.jsonld>
virtuoso_1  | 08:35:59 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7332
virtuoso_1  | 08:35:59 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:59 EXEC_1 0 172.30.0.2 1111:3 ps05 Exec 1 time(s) sparql
virtuoso_1  | INSERT IN GRAPH <http://localhost:3002/g/test/mappings-mappingbased-literals-2018.12.01-dataid.jsonld> {
virtuoso_1  | <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sl.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=sl.ttl.bz2> .
virtuoso_1  | <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sl.ttl.bz2> <http
virtuoso_1  | 08:35:59 ERRS_0 XXXXX COL.. Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 08:35:59 LTRS_1 0 172.30.0.2 1111:3 Rollback transact 0x7f2d6c0037b0 0 7333
virtuoso_1  | 08:35:59 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:59 EXEC_1 0 172.30.0.2 1111:3 ps06 Exec 1 time(s) SELECT name_part(\KEY_TABLE,0) AS \TABLE_CAT VARCHAR(128),name_part(\KEY_TABLE,1) AS \TABLE_SCHEM VARCHAR(128),name_part(\KEY_TABLE,2) AS \TABLE_NAME VARCHAR(128),table_type(\KEY_TABLE)  AS \TABLE_TYPE VARCHAR(128),NULL AS \REMARKS VARCHAR(254) ,NULL AS \TYPE_CAT  VARCHAR(128) ,NULL AS \TYPE_SCHEM VARCHAR(128) ,NULL AS \TYPE_NAME VARCHAR(128) ,NULL AS \SELF_REFERENCING_COL_NAME VARCHAR(128) ,NULL AS \REF_GENERATION VARCHAR(128) FROM DB.DBA.SYS_KEYS WHERE __any_grants(\KEY_TABLE) AND UPPER(name_p
virtuoso_1  | 08:35:59 LTRS_1 0 172.30.0.2 1111:3 Commit transact 0x7f2d6c0037b0 0 7334
virtuoso_1  | 08:35:59 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0
virtuoso_1  | 08:35:59 LTRS_1 0 172.30.0.2 1111:3 Rollback transact 0x7f2d6c0037b0 0 7335
virtuoso_1  | 08:35:59 LTRS_2 0 172.30.0.2 1111:3 Restart transact 0x7f2d6c0037b0

There is no error when only inserting the second file There is also no error when inserting the second file first and then inserting both

The setup is using a docker image called dbpedia/gstore which is taking jsonld files and makes SPARQL updates The input files are in the debug-in folder.

I will set up an example without the dbpedia/gstore asap

holycrab13 commented 2 years ago

In some rare cases, the database was corrputed after and failed to start with either

virtuoso_1  | 07:36:30 Roll forward started
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RDF_LANGUAGE on table DB.DBA.RDF_LANGUAGE. Transaction killed.
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RDF_LANGUAGE on table DB.DBA.RDF_LANGUAGE. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RDF_LANGUAGE on table DB.DBA.RDF_LANGUAGE. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0

virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0

....

virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: 23000 : SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 In roll forward batch commit failed code 5
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 Rfwd error: 23000: SR175: Uniqueness violation : Violating unique index RO_START on table DB.DBA.RO_START. Transaction killed.
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
virtuoso_1  | 07:36:30 Rfwd error: XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 SQL Error: XXXXX : COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_GS slice 0
virtuoso_1  | 07:36:30 Roll forward complete
virtuoso_1  | 07:36:30 Error executing a server init statement : XXXXX: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_SP slice 0 --  WS.WS.SYS_DAV_INIT_RDF ()

OR

virtuoso_1  | 07:41:13 Compiler unit is timed at 0.000103 msec
virtuoso_1  | 07:41:13 Reference to page with free remap dp = 6917, remap = 6917
virtuoso_1  | 07:41:13 Reference to page with free remap dp = 6917, remap = 6917
virtuoso_1  | 07:41:13 Reference to page with free remap dp = 6917, remap = 6917
virtuoso_1  | 07:41:13 Reference to page with free remap dp = 5378, remap = 5378
virtuoso_1  | 07:41:13 Broken index RDF_QUAD
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x917aca]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x917b55]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x4a2ec0]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x4a3619]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x48182b]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x541891]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x541b4b]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x547991]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x548312]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x548c57]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x548dbc]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x59cd38]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x5b0c3a]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x5b14fe]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x5b62d7]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x5b6503]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x5b68ec]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x594382]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x596b61]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x693fe7]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x6946ec]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x679e9d]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x61adde]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t() [0x45a164]
virtuoso_1  | 07:41:13 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f03ea3a3bf7]
virtuoso_1  | 07:41:13 /opt/virtuoso-opensource/bin/virtuoso-t(realloc+0x351) [0x454c29]
virtuoso_1  | 07:41:13 GPF: colsearch.c:818 ref to deld col page
virtuoso_1  | GPF: colsearch.c:818 ref to deld col page
HughWilliams commented 2 years ago

We are looking into this issue ...

manonthegithub commented 2 years ago

@HughWilliams @holycrab13 Hey guys here is minimal repro snippet in java JDBC Driver used (http://download3.openlinksw.com/uda/virtuoso/jdbc/virtjdbc4.jar) virtuoso used (docker pull openlink/virtuoso-opensource-7):

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Properties;

public class Repro {

    public static void main(String[] args) throws Exception {
        Connection conn = getConnection("localhost", 1111, "dba", "your password");
        String filePa = "<path to file with the query below>";
        Path req = Paths.get(filePa);
        byte[] queryBs = Files.readAllBytes(req);
        String q = new String(queryBs, "UTF-8");
        PreparedStatement ps = conn.prepareStatement("sparql\n" + q);
        int re = ps.executeUpdate();
        System.out.println(re);
    }

    public static Connection getConnection(String host, Integer port, String login, String pass) throws SQLException {
        Properties connectionProps = new Properties();
        connectionProps.put("user", login);
        connectionProps.put("password", pass);
        Connection conn = DriverManager.getConnection(
                "jdbc:virtuoso://" + host + ":" + port + "/charset=UTF-8",
                connectionProps);
        return conn;
    }

}

Here is the query:

INSERT IN GRAPH <http://localhost:3002/g/test/mappings-mappingbased-literals-2018.12.01-dataid.jsonld> {
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/core#preview> "<http://sr.dbpedia.org/resource/!!!> <http://xmlns.com/foaf/0.1/name> \"!!!\"@sr .\n<http://sr.dbpedia.org/resource/!!!__Album__1> <http://dbpedia.org/ontology/year> \"2007\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://sr.dbpedia.org/resource/!!!__Album__1> <http://xmlns.com/foaf/0.1/name> \"-{Myth Takes}-\"@sr .\n<http://sr.dbpedia.org/resource/$O$> <http://dbpedia.org/ontology/runtime> \"4083.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://sr.dbpedia.org/resource/$O$> <http://dbpedia.org/ontology/year> \"2009\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://sr.dbpedia.org/resource/$O$> <http://xmlns.com/foaf/0.1/name> \"$O$\"@sr .\n<http://sr.dbpedia.org/resource/$O$__Album__2> <http://dbpedia.org/ontology/year> \"2010\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://sr.dbpedia.org/resource/$O$__Album__2> <http://xmlns.com/foaf/0.1/name> \"5\"@sr .\n<http://sr.dbpedia.org/resource/'64_-_'95> <http://dbpedia.org/ontology/runtime> \"3180.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://sr.dbpedia.org/resource/'64_-_'95> <http://dbpedia.org/ontology/year> \"2005\"^^<http://www.w3.org/2001/XMLSchema#gYear> ." .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://www.w3.org/ns/dcat#mediaType> <http://dataid.dbpedia.org/ns/mt#ApplicationNTriples> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/core#formatExtension> "ttl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/core#file> <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=sr.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=sr.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/cv#lang> "sr" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sha256sum> "3f99c63424d0b6b1bd659573236f6b5c9b8f00c4e06ec66ccff8b99d8cef0259" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://dataid.dbpedia.org/ns/core#nonEmptyLines> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://purl.org/dc/terms/license> <http://purl.oclc.org/NET/rdflicense/cc-by3.0> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=sr.ttl.bz2> <http://purl.org/dc/terms/issued> "2018-12-01T00:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=gl.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://www.w3.org/ns/dcat#mediaType> <http://dataid.dbpedia.org/ns/mt#ApplicationNTriples> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sorted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://www.w3.org/ns/dcat#byteSize> "2590477"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://purl.org/dc/terms/title> "Literals extracted with mappings"@en .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#compression> "bzip2" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://purl.org/dc/terms/issued> "2018-12-01T00:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://purl.org/dc/terms/hasVersion> "2018.12.01" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#preview> "<http://gl.dbpedia.org/resource/!!!> <http://dbpedia.org/ontology/activeYearsStartYear> \"1996\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://gl.dbpedia.org/resource/!!!> <http://xmlns.com/foaf/0.1/name> \"!!!\"@gl .\n<http://gl.dbpedia.org/resource/!!!> <http://xmlns.com/foaf/0.1/name> \"\"@gl .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"2727.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"304.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"315.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"348.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"368.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"414.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://gl.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"446.0\"^^<http://www.w3.org/2001/XMLSchema#double> ." .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dataid.dbpedia.org/ns/core#Part> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sha256sum> "096779e28af73e8ee9f1fb5d5383d045bc1e18a8dbbdcd093b47cdff7b85af0d" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#formatExtension> "ttl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/cv#lang> "gl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://purl.org/dc/terms/license> <http://purl.oclc.org/NET/rdflicense/cc-by3.0> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://purl.org/dc/terms/modified> "2019-01-11T01:01:01Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#file> <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=gl.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=gl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#nonEmptyLines> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sorted> "true"^^<http://www.w3.org/2001/XMLSchema#boolean> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#compression> "bzip2" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#nonEmptyLines> "331.0"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://www.w3.org/ns/dcat#byteSize> "4216"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/cv#lang> "hy" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://purl.org/dc/terms/license> <http://purl.oclc.org/NET/rdflicense/cc-by3.0> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://purl.org/dc/terms/title> "Literals extracted with mappings"@en .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://purl.org/dc/terms/issued> "2018-12-01T00:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#file> <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=hy.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "51933.0"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sha256sum> "03dd0a618b90e0ab204306fa8d55e1fccfcf29e543541611b2781d48525d5911" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#formatExtension> "ttl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=hy.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://dataid.dbpedia.org/ns/core#preview> "<http://hy.dbpedia.org/resource/\u0531\u0562\u0580\u0561\u0570\u0561\u0574_\u0536\u0561\u0574\u056B\u0576\u0575\u0561\u0576> <http://xmlns.com/foaf/0.1/name> \"\u0531\u0562\u0580\u0561\u0570\u0561\u0574 \u0536\u0561\u0574\u056B\u0576\u0575\u0561\u0576\"@hy .\n<http://hy.dbpedia.org/resource/\u0531\u0563\u0561\u0569\u0561_\u0554\u0580\u056B\u057D\u057F\u056B> <http://dbpedia.org/ontology/activeYearsEndYear> \"1976\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u0563\u0561\u0569\u0561_\u0554\u0580\u056B\u057D\u057F\u056B> <http://dbpedia.org/ontology/activeYearsStartYear> \"1920\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u0564\u0578\u0576\u056B\u057D_(\u0562\u0561\u0576\u0561\u057D\u057F\u0565\u0572\u056E)> <http://dbpedia.org/ontology/activeYearsEndYear> \"0020\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u0564\u0578\u0576\u056B\u057D_(\u0562\u0561\u0576\u0561\u057D\u057F\u0565\u0572\u056E)> <http://dbpedia.org/ontology/activeYearsStartYear> \"0020\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u0564\u0578\u0576\u056B\u057D_(\u0562\u0561\u0576\u0561\u057D\u057F\u0565\u0572\u056E)> <http://dbpedia.org/ontology/pseudonym> \"\u0531\u0564\u0578\u0576\u056B\u057D\"@hy .\n<http://hy.dbpedia.org/resource/\u0531\u056C\u0562\u0565\u0580_\u053F\u0561\u0574\u0575\u0578\u0582> <http://dbpedia.org/ontology/activeYearsEndYear> \"1960\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u056C\u0562\u0565\u0580_\u053F\u0561\u0574\u0575\u0578\u0582> <http://dbpedia.org/ontology/activeYearsStartYear> \"1934\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u056C\u0564\u0561_\u0544\u0565\u0580\u056B\u0576\u056B> <http://dbpedia.org/ontology/activeYearsEndYear> \"2008\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://hy.dbpedia.org/resource/\u0531\u056C\u0564\u0561_\u0544\u0565\u0580\u056B\u0576\u056B> <http://dbpedia.org/ontology/activeYearsStartYear> \"1953\"^^<http://www.w3.org/2001/XMLSchema#gYear> ." .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://www.w3.org/ns/dcat#mediaType> <http://dataid.dbpedia.org/ns/mt#ApplicationNTriples> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://purl.org/dc/terms/hasVersion> "2018.12.01" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dataid.dbpedia.org/ns/core#Part> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=hy.ttl.bz2> <http://purl.org/dc/terms/modified> "2019-01-11T01:01:06Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sha256sum> "dc74352d57e6fd9118e5148261b31d961cb47bb9621da4513e021e6f865f8f6b" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#formatExtension> "ttl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#compression> "bzip2" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://www.w3.org/ns/dcat#mediaType> <http://dataid.dbpedia.org/ns/mt#ApplicationNTriples> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/cv#lang> "es" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#file> <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=es.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://purl.org/dc/terms/issued> "2018-12-01T00:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://purl.org/dc/terms/license> <http://purl.oclc.org/NET/rdflicense/cc-by3.0> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://www.w3.org/ns/dcat#byteSize> "27794328"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dataid.dbpedia.org/ns/core#Part> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://purl.org/dc/terms/title> "Literals extracted with mappings"@en .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://purl.org/dc/terms/modified> "2019-01-11T00:59:47Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#nonEmptyLines> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sorted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=es.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://dataid.dbpedia.org/ns/core#preview> "<http://es.dbpedia.org/resource/!!!> <http://dbpedia.org/ontology/activeYearsStartYear> \"1995\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://es.dbpedia.org/resource/!!!> <http://dbpedia.org/ontology/background> \"grupo_o_banda\" .\n<http://es.dbpedia.org/resource/!!!> <http://dbpedia.org/ontology/status> \"Activo\" .\n<http://es.dbpedia.org/resource/!!!> <http://xmlns.com/foaf/0.1/name> \"!!!\"@es .\n<http://es.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/review> \"http://web.archive.org/web/http://pitchforkmedia.com/article/record_review/16471/_hk_Chk_Chk_\"^^<http://www.w3.org/2001/XMLSchema#anyURI> .\n<http://es.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/review> \"http://www.allmusic.com/album/r540530\"^^<http://www.w3.org/2001/XMLSchema#anyURI> .\n<http://es.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"0.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://es.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"2727.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://es.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"304.0\"^^<http://www.w3.org/2001/XMLSchema#double> .\n<http://es.dbpedia.org/resource/!!!_(\u00E1lbum)> <http://dbpedia.org/ontology/runtime> \"315.0\"^^<http://www.w3.org/2001/XMLSchema#double> ." .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=es.ttl.bz2> <http://purl.org/dc/terms/hasVersion> "2018.12.01" .
<http://localhost:3000/janfo/mappings/mappingbased-literals> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dataid.dbpedia.org/ns/core#Artifact> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://www.w3.org/ns/dcat#byteSize> "14262606"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://purl.org/dc/terms/hasVersion> "2018.12.01" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#nonEmptyLines> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#file> <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=ja.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/cv#lang> "ja" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://purl.org/dc/terms/title> "Literals extracted with mappings"@en .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://purl.org/dc/terms/issued> "2018-12-01T00:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://www.w3.org/ns/dcat#mediaType> <http://dataid.dbpedia.org/ns/mt#ApplicationNTriples> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#preview> "<http://ja.dbpedia.org/resource/!!!> <http://dbpedia.org/ontology/activeYearsStartYear> \"1995\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://ja.dbpedia.org/resource/!!!> <http://dbpedia.org/ontology/background> \"band\" .\n<http://ja.dbpedia.org/resource/!!!> <http://xmlns.com/foaf/0.1/name> \"!!!\"@ja .\n<http://ja.dbpedia.org/resource/!!!Fuck_You!!!> <http://dbpedia.org/ontology/title> \"!!!Fuck You!!!\"@ja .\n<http://ja.dbpedia.org/resource/!!!Fuck_You!!!> <http://dbpedia.org/ontology/title> \"\"@ja .\n<http://ja.dbpedia.org/resource/!LOUD!> <http://dbpedia.org/ontology/title> \"!LOUD!\"@ja .\n<http://ja.dbpedia.org/resource/!SHOUT!> <http://dbpedia.org/ontology/title> \"!SHOUT!\"@ja .\n<http://ja.dbpedia.org/resource/!_-attention-> <http://dbpedia.org/ontology/title> \"! -attention-\"@ja .\n<http://ja.dbpedia.org/resource/!wagero!> <http://dbpedia.org/ontology/activeYearsEndYear> \"2009\"^^<http://www.w3.org/2001/XMLSchema#gYear> .\n<http://ja.dbpedia.org/resource/!wagero!> <http://dbpedia.org/ontology/activeYearsStartYear> \"1998\"^^<http://www.w3.org/2001/XMLSchema#gYear> ." .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=ja.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sorted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dataid.dbpedia.org/ns/core#Part> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#compression> "bzip2" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://purl.org/dc/terms/modified> "2019-01-11T01:01:59Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://purl.org/dc/terms/license> <http://purl.oclc.org/NET/rdflicense/cc-by3.0> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sha256sum> "bfcdb21f91d6957345b42225a736ed209881fe962f34f690a929eb1cd960b6f9" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ja.ttl.bz2> <http://dataid.dbpedia.org/ns/core#formatExtension> "ttl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/cv#lang> "ga" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/core#formatExtension> "ttl" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://purl.org/dc/terms/hasVersion> "2018.12.01" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://purl.org/dc/terms/issued> "2018-12-01T00:00:00Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/core#preview> "[binary data - no preview]" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/core#file> <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01/mappingbased-literals_lang=ga.ttl.bz2> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/core#compression> "bzip2" .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://purl.org/dc/terms/title> "Literals extracted with mappings"@en .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://www.w3.org/ns/dcat#byteSize> "273212"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ga.ttl.bz2> <http://dataid.dbpedia.org/ns/core#nonEmptyLines> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> .
}

output after running:

Exception in thread "main" virtuoso.jdbc4.VirtuosoException: COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_OP slice 0
    at virtuoso.jdbc4.VirtuosoResultSet.process_result(VirtuosoResultSet.java:363)
    at virtuoso.jdbc4.VirtuosoResultSet.getMoreResults(VirtuosoResultSet.java:85)
    at virtuoso.jdbc4.VirtuosoPreparedStatement.sendQuery(VirtuosoPreparedStatement.java:88)
    at virtuoso.jdbc4.VirtuosoPreparedStatement.executeUpdate(VirtuosoPreparedStatement.java:119)
    at org.dbpedia.databus.Repro.main(Repro.java:23)
HughWilliams commented 2 years ago

Using the latest Virtuoso open source docker image from docker pull openlink/virtuoso-opensource-7, I am able to run your test application successfully inserting 100 triples:

De-iMac:~ hwilliams$ javac -cp "./Downloads/virtjdbc4.jar:." Repro.java 
De-iMac:~ hwilliams$ java -cp "./Downloads/virtjdbc4.jar:." Repro
100
De-iMac:~ hwilliams$ ~/isql 1111
OpenLink Virtuoso Interactive SQL (Virtuoso)
Type HELP; for help and EXIT; to exit.
Connected to OpenLink Virtuoso
SQL> sparql select count(*) from <http://localhost:3002/g/test/mappings-mappingbased-literals-2018.12.01-dataid.jsonld> where {?s ?p ?o};
callret-0
INTEGER
_______________________________________________________________________________

100

1 Rows. -- 3 msec.
SQL>

Note: this is starting with an empty database, i.e., new docker instance. Is this where you are starting from, or is yours an existing database with other RDF data already loaded?

holycrab13 commented 2 years ago

The example I posted earlier was tested on two different machines with a clean docker container and the bug occured in both cases.

I have tested the example posted by @manonthegithub and cannot reproduce it with that either. I think the reason why he was able to reproduce it with the posted INSERT statement was that his virtuoso was already in the faulty state from running my repro example.

holycrab13 commented 2 years ago

Here is an example that is reproducing the issue for me on a clean virtuoso: https://github.com/dbpedia/databus-transfer/tree/insert-stopped-debug

JDBC Driver used (http://download3.openlinksw.com/uda/virtuoso/jdbc/virtjdbc4.jar) Virtuoso image used (docker pull openlink/virtuoso-opensource-7)

with

docker run --name test_virtuoso --interactive --tty --env DBA_PASSWORD=password --publish 1111:1111 --publish 8890:8890 openlink/virtuoso-opensource-7:latest

and then

javac -cp "./Downloads/virtjdbc4.jar:." Repro.java 
java -cp "./Downloads/virtjdbc4.jar:." Repro

It triggers when doing 2 inserts sequentially. I pruned both inserts down step by step. I haven't found the sweet spot yet, but at some point the issue stops happening.

Inserting the same data into one graph works without problems Running only one insert works

manonthegithub commented 2 years ago

@HughWilliams so @holycrab13's example worked for me (reproduced). Please try that one :). I used non empty database as you supposed.

pkleef commented 2 years ago

I have recreated the issue internally and discussed the matter within development and we will have a closer look to see what is going on internally. I will keep you posted.

holycrab13 commented 2 years ago

Any updates on this?

TallTed commented 2 years ago

@manonthegithub @holycrab13 -- This is still on Development's task list. /cc @pkleef

As an aside -- please be aware that your notes —

JDBC Driver used (http://download3.openlinksw.com/uda/virtuoso/jdbc/virtjdbc4.jar) Virtuoso image used (docker pull openlink/virtuoso-opensource-7)

— do not provide static information, as the driver and image retrieved from those locations will vary over time. In future, please instead provide your local output of java -jar virtjdbc4.jar for the JDBC driver, and virtuoso-t -? for Open Source Edition or virtuoso-iodbc-t -? for Enterprise Edition — as these will accurately reflect what you are running even if we've updated the files at the locations you listed above.

kurzum commented 2 years ago

ok, I also reproduced the bug now. I made a test set for bash:

isql-vt 1111 dba dba VERBOSE=ON i1.sparql.txt > 1.1.txt 2>1.2.txt
isql-vt 1111 dba dba VERBOSE=ON i2.sparql.txt > 2.1.txt 2>2.2.txt

i1.sparql.txt i2.sparql.txt

I am using:

10:16:29 Version 07.20.3230-pthreads for Linux as of Feb  4 2020
(docker)
and 

Driver: 06.01.3127 OpenLink Virtuoso ODBC Driver
OpenLink Interactive SQL (Virtuoso), version 0.9849b.

I split the triples and ran them individually:

while read p; do
  echo "----------------"
  echo "$p"
  isql-vt 1111 dba password VERBOSE=ON exec="sparql INSERT IN GRAPH <http://localhost:3002/g/test/mappings-geo-coordinates-mappingbased-2018.09.12-dataid.jsonld> { $p } ;"
  echo "----------------"
done <triples.txt

seems like it is definitely the "preview" triples. When run individually they throw syntax errors: ri2.txt

Some explanation: preview is a property that provides the first 5 lines of an RDF file, so it is RDF encoded as RDF Literal. The problematic triples are in this file: triples_preview.txt

Then I split the triples of i2 into no preview (i3) and only preview (i4): i3.sparql.txt i4.sparql.txt

Then I tested it again:

# no preview triples are loaded first. This seems to initiate the DB properly and sets up the graph. Then loading i1 and i2 still throw an error "-- More than 0 parameters, ignoring all the rest of the statement #line 1 "i2.sparql.txt"" but they do not corrupt the store any more.
isql-vt 1111 dba password VERBOSE=ON i3.sparql.txt > 3.1.txt 2>3.2.txt
isql-vt 1111 dba password VERBOSE=ON i1.sparql.txt > 1.1.txt 2>1.2.txt
isql-vt 1111 dba password VERBOSE=ON i2.sparql.txt > 2.1.txt 2>2.2.txt
# running i1 or i2 first which contain the preview property mess up the store:
isql-vt 1111 dba password VERBOSE=ON i1.sparql.txt > 1.1.txt 2>1.2.txt
isql-vt 1111 dba password VERBOSE=ON i2.sparql.txt > 2.1.txt 2>2.2.txt
isql-vt 1111 dba password VERBOSE=ON i3.sparql.txt > 3.1.txt 2>3.2.txt

Fazit: Overall this seems to be an encoding thing. ODBC/JDBC have certain control and macro characters like $. The preview triple was originally created by me in the old maven upload client. back then I already had trouble with creating these as it is -- until now -- unclear to me, what I needed to escape/encode exactly when putting RDF in RDF as a Literal. This gets potentiated by the different available syntaxes (ntriples, ttl, rdf/xml) plus also the they have to go into SPARQL which is yet another syntax and I am not sure, if SPARQL INSERT is exactly like turtle or has different details.

@TallTed @pkleef maybe that helps you to identify this issue and prevent the DB crash. This was quite a serious bug for us and it almost caused us to deliver the DBpedia Databus 2.0 with Fuseki instead of Virtuoso. We are removing the preview triples for now.

TallTed commented 2 years ago

@kurzum — I notice that you're using a Release 6 iSQL, which is statically linked to the 06.01.3127 OpenLink Virtuoso ODBC Driver, though you're working with Release 7 Virtuoso. This is unlikely to be the cause of the described crash, and should generally work, but may lead to other issues over time, and certainly doesn't include enhancements that have been made in more recent components. We always recommend using client components that match at least the major Release version, if not the least minor version, of the target server instance.

Please also note that 07.20.3230-pthreads for Linux as of Feb 4 2020 (as written to the Virtuoso log file) does not include the git_head value, which is seen in the parentheses of the virtuoso-t -? output (as seen above). The git_head value pins down the exact last-commit of the codebase that resulted in a given binary, while the 07.20.3230 may include dozens or even hundreds of commits between 07.20.3229 and 07.20.3231, and the as of Feb 4 2020 just indicates the date on which the binary was built — which could be months or even years after the git_head of that codebase.

kurzum commented 2 years ago

@TallTed I wouldn't use the old isql client anywhere in production really and also not mix versions. I just wanted to dig into this issue here, so I just ran apt-get install virtuoso-opensource quickly to get the client for bash scripting.

I am unsure about the commit. I ran

docker run --name test_virtuoso --interactive --tty --env DBA_PASSWORD=password --publish 1111:1111 --publish 8890:8890 openlink/virtuoso-opensource-7:latest

and I have the code for the image:

docker images | grep virtuoso-ope
openlink/virtuoso-opensource-7    latest                                  010c4d384d1e   2 years ago     155MB

Can't get the commit hash at the moment. I hope that is enough.

TallTed commented 2 years ago

@kurzum — I think that's enough detail for this issue. For future reference, as we document in the docker hub details —

To check the version of the Virtuoso binary, you can use the following command:

$ docker run openlink/virtuoso-opensource-7 version

This Docker image is using the following version of Virtuoso:
Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.6-rc1.3230-pthreads as of Nov  2 2018 (4d226f4)
Compiled for Linux (x86_64-generic_glibc25-linux-gnu)
Copyright (C) 1998-2018 OpenLink Software

— and the git_head is revealed in this example as 4d226f4.

You can also use the SPARQL query documented here to get these version details from a running instance

pkleef commented 2 years ago

@kurzum

I checked docker hub and found the image you are using:

$ docker images | grep virtuoso-open
openlink/virtuoso-opensource-7       7.2.6-r3-g1b16668    010c4d384d1e   2 years ago     155MB

$ docker run -i -t openlink/virtuoso-opensource-7:7.2.6-r3-g1b16668 version

This docker image is using the following version of Virtuoso:

Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.6-rc1.3230-pthreads as of Feb  4 2020 (a11a8e37b)
Compiled for Linux (x86_64-generic_glibc25-linux-gnu)
Copyright (C) 1998-2020 OpenLink Software

Note that docker does not automatically check for newer versions, unless you perform another pull command:

$ docker pull openlink/virtuoso-opensource-7:latest

This will give you the latest stable version of VOS 7.2.6 from June 22, 2021.

We are currently finalizing the 7.2.7 release which will be ready soon, and will also be released as a new docker image.

kurzum commented 2 years ago

Thanks for the hints, I am still a docker newbie, just using it & never build something serious with it. a11a8e37b is the right version. Jan and Kirill might have used others as well.

kurzum commented 2 years ago

@TallTed @pkleef I did some more digging and produced a smaller set of triples.

setup

isql-vt as before Driver: 06.01.3127 OpenLink Virtuoso ODBC Driver. OpenLink Interactive SQL (Virtuoso), version 0.9849b. I updated docker image, it is now:

docker run openlink/virtuoso-opensource-7 version

This Docker image is using the following version of Virtuoso:

Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.6.3233-pthreads as of Jun 22 2021 (111d17e5b)
Compiled for Linux (x86_64-generic_glibc25-linux-gnu)
Copyright (C) 1998-2021 OpenLink Software

I am restarting the docker image with:

docker stop test_virtuoso
docker rm test_virtuoso
docker run --name test_virtuoso --interactive --tty --env DBA_PASSWORD=password --publish 1111:1111 --publish 8890:8890 openlink/virtuoso-opensource-7:latest

Issue 1: -- More than 0 parameters, ignoring all the rest of the statement #line 1

more.sparql.txt triples.nt.txt

Summary: the triples were exported from the virtuoso at databus.dbpedia.org/sparql . So we loaded them once. The error is -- More than 0 parameters, ignoring all the rest of the statement sparql INSERT IN GRAPH <http://localhost:3002/dataid.jsonld> { I ran rapper over the triples.nt.txt and the syntax is ok. This error also doesn't crash the store. It just returns the error message. The problem seems to only occur using ODBC/JDBC or isql-vt -> unclear whether it is related

issue 2

I removed triples via trial and error from the above set. The error is still quite elusive as it seems to occur only in some occasions:

  1. only ODBC/JDBC/ISQL, no bulk loading
  2. only occurs after a previous insert on same graph or another

I narrowed it down to these two statements, however, if I remove more triples from the first or last it suddenly works. use these statements to reproduce:

 sparql INSERT IN GRAPH <http://localhost:3002/dataid.jsonld> { <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://purl.org/dc/terms/conformsTo> "http://dataid.dbpedia.org/ns/core#" . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sorted> "true"^^<http://www.w3.org/2001/XMLSchema#boolean> . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#sha256sum> "6890e47989091b58037e3fc31d23e03467f83259182fb84c03e4645c77ce6b47" . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://www.w3.org/ns/dcat#mediaType> <http://dataid.dbpedia.org/ns/mt#ApplicationNTriples> . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://www.w3.org/ns/dcat#downloadURL> <https://downloads.dbpedia.org/repo/lts/mappings/geo-coordinates-mappingbased/2018.09.12/geo-coordinates-mappingbased_lang=nl.ttl.bz2> . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://purl.org/dc/terms/modified> "2019-02-06T06:53:34Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "4405176"^^<http://www.w3.org/2001/XMLSchema#decimal> . <http://localhost:3000/janfo/mappings/geo-coordinates-mappingbased/2018.09.12#geo-coordinates-mappingbased_lang=nl.ttl.bz2> <http://dataid.dbpedia.org/ns/core#compression> "bzip2" . } ; 
sparql INSERT IN GRAPH <http://localhost:3002/dataid.jsonld> { <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=pt.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> . <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ro.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "854549.0"^^<http://www.w3.org/2001/XMLSchema#decimal> .  } ;

Load command:

isql-vt 1111 dba password VERBOSE=ON i3.sparql.txt > 3.1.txt 2>3.2.txt

Log:

Connected to OpenLink Virtuoso
Driver: 06.01.3127 OpenLink Virtuoso ODBC Driver

Done. -- 6 msec.

*** Error XXXXX: [Virtuoso Driver][Virtuoso Server]COL..: Insert stopped because out of seg data here or elsewhere host 0 key RDF_QUAD_POGS slice 0
at line 2 of Command-Line-Load i3.sparql.txt:
sparql INSERT IN GRAPH <http://localhost:3002/dataid.jsonld> { <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=pt.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "NaN"^^<http://www.w3.org/2001/XMLSchema#double> . <http://localhost:3000/janfo/mappings/mappingbased-literals/2018.12.01#mappingbased-literals_lang=ro.ttl.bz2> <http://dataid.dbpedia.org/ns/core#uncompressedByteSize> "854549.0"^^<http://www.w3.org/2001/XMLSchema#decimal> .  } 

A note: #uncompressedByteSize sometimes uses xsd:decimal and sometimes xsd:double this is the only irregularity I see at the moment.

kurzum commented 2 years ago

Regarding the more than 0 params

seems like this was additionally introduced by me, testing with isql-vt. pkleef send me these lines:

set macro_substitution off;
set ignore_params on;

which makes this go away

RDF_QUAD_POGS slice 0

seems fixed in

docker pull pkleef/virtuoso-opensource-7:latest
# Version 7.2.7-dev.3233-pthreads as of May 11 2022 (f80f792a1)
docker stop test_virtuoso
docker rm test_virtuoso
docker run --name test_virtuoso --interactive --tty --env DBA_PASSWORD=password --publish 1111:1111 --publish 8890:8890 pkleef/virtuoso-opensource-7:latest

So I can verify the fix (done with the test data above).

holycrab13 commented 2 years ago

the image pkleef/virtuoso-opensource-7:latest resolved the bug for me

pkleef commented 2 years ago

@holycrab13 That is good to hear. We are one step closer to a new stable release.

excursus commented 1 year ago

Any updates on this? I'm still experiencing this bug in openlink/virtuoso-opensource-7:7.2.8-r14-geef1fc5-ubuntu.

TallTed commented 1 year ago

@excursus — Would you please provide the version details obtained from virtuoso-t -? or this SPARQL query? It is particularly important that we know the git_head value of your Virtuoso binary.

excursus commented 1 year ago

Sure, virtuoso-t -? returns this:

Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.8.3235-pthreads as of Oct 19 2022 (64e6ecd39)
Compiled for Linux (aarch64-ubuntu_bionic-linux-gnu)
Copyright (C) 1998-2022 OpenLink Software