Strange behavior with graph listing

ghost commented 10 years ago

Hi,

I have a strange behavior from Virtuoso 7.1. When I add a new graph some other graphs disapears. For example, at the beginning I have these graphs :

http://www.openlinksw.com/schemas/virtrdf#
http://localhost:8890/sparql
http://www.w3.org/2002/07/owl#
b3sifp
http://www.openlinksw.com/schemas/oplweb#
virtrdf-label
facets

Now I add a new graph with these two commands in isql :

ld_dir('/my/path/', 'file.nt', 'http://my.new.graph');
rdf_loader_run();

And now I have only these graphs displayed :

http://www.openlinksw.com/schemas/virtrdf#
http://localhost:8890/sparql
http://www.w3.org/2002/07/owl#
http://my.new.graph

Once again, if I add another graph like "http://my.new.graph.two" I have the same list than above. The new one doesn't appears.

Doesn't matter if I list the graphs either with a SPARQL query, through the graph tab on the conductor or with the procedure "DB.DBA.SPARQL_SELECT_KNOWN_GRAPHS();" I have always the same false results.

I don't think it's a normal behavior. Anyone has an idea of what's happened ?

Thanks in advance.

Best.

HughWilliams commented 10 years ago

What do the following queries return as the list of available graphs ?

sparql select ?g where {GRAPH ?g{?s a ?t}};

SPARQL SELECT ?g WHERE { GRAPH ?g { ?s ?p ?o } };

ghost commented 10 years ago

For the first one :

957963 Rows. -- 47310 msec.

And as I could see among the result it's the same graphs than with the "DB.DBA.SPARQL_SELECT_KNOWN_GRAPHS();" command who are displayed.

For the second query :

15530703 Rows. -- 823162 msec.

Exactly the same result than the previous query.

ghost commented 10 years ago

Last thing, I rerun the command "DB.DBA.SPARQL_SELECT_KNOWN_GRAPHS();" and I have a really unexpted answer time :

7 Rows. -- 23539264 msec.

My virtuoso tooks around 6 hours just to give me an answer. And I have around 12 graphs loaded which are pretty little, just one graph at 3 millions triples and the others are around 100K.

HughWilliams commented 10 years ago

Those query times are excessive, how many triples are in the database as this can influence the time taken ? On my server with 20 graphs they are returned in msecs:

20 Rows. -- 70 msec.

HughWilliams commented 10 years ago

Those query times are excessive, how many triples are in the database as this can influence the time taken ? On my server with 20 graphs they are returned in msecs:

20 Rows. -- 70 msec.

ghost commented 10 years ago

I have something like only 15 millions triples in this database.

HughWilliams commented 10 years ago

What is the virtuoso of 7.1 build:

virtuoso-t -?

ghost commented 10 years ago

I had a new bug certainly related with this one, the exactly same graph is listed twice in the interface. I put a screenshot in attachment. The concerned graph is http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo. I loaded the rdf_mapper package.

virtuoso

HughWilliams commented 10 years ago

Julian, Is that same graph duplicated when running either of the following queries ?

DB.DBA.SPARQL_SELECT_KNOWN_GRAPHS();

sparql select distinct ?g where {GRAPH ?g{?s a ?t}};

SPARQL SELECT DISTINCT ?g WHERE { GRAPH ?g { ?s ?p ?o } };

ghost commented 10 years ago

Yes, it appears twice for each of these queries/commands.

ghost commented 10 years ago

Sorry I forgot to tell you that these commands don't give me the same result :

SQL> DB.DBA.SPARQL_SELECT_KNOWN_GRAPHS();
GRAPH_IRI
VARCHAR
_______________________________________________________________________________

http://www.openlinksw.com/schemas/virtrdf#
http://localhost:8890/sparql
http://localhost:8890/DAV/
http://www.w3.org/2002/07/owl#
http://www.w3.org/2002/07/owl
b3sifp
http://bdc.orange.com/graph/prod/wiki
http://bdc.orange.com/graph/prod/webedia
http://bdc.orange.com/graph/prod/deezer
http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo
http://bdc.orange.com/graph/ProcessConfiguration/SSearchPeople
http://bdc.orange.com/graph/ProcessConfiguration/SSearchSport
http://bdc.orange.com/graph/ProcessConfiguration/SSearchTele
http://www.openlinksw.com/schemas/RDF_Mapper_Ontology/1.0/
http://purl.org/ontology/bibo/
http://www.openlinksw.com/schemas/googleplus
http://www.w3.org/1999/02/22-rdf-syntax-ns#
http://www.w3.org/2000/01/rdf-schema#
http://www.w3.org/2004/02/skos/core
http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo

http://bdc.orange.com/graph/vodpc
http://purl.org/dc/terms/
http://xmlns.com/foaf/0.1/

23 Rows. -- 170 msec.

SQL> sparql select distinct ?g where {GRAPH ?g{?s a ?t}};
g
LONG VARCHAR
_______________________________________________________________________________

http://www.w3.org/1999/02/22-rdf-syntax-ns#
http://www.openlinksw.com/schemas/virtrdf#
local:/sparql
http://purl.org/dc/terms/
http://purl.org/ontology/bibo/
http://www.w3.org/2000/01/rdf-schema#
http://www.w3.org/2002/07/owl#
http://www.w3.org/2002/07/owl
http://xmlns.com/foaf/0.1/
http://bdc.orange.com/graph/ProcessConfiguration/SSearchTele
http://bdc.orange.com/graph/prod/wiki
http://bdc.orange.com/graph/prod/webedia
http://bdc.orange.com/graph/ProcessConfiguration/SSearchPeople
http://purl.org/dc/elements/1.1/
http://purl.org/goodrelations/v1
b3sifp
http://www.w3.org/2004/02/skos/core
http://www.openlinksw.com/schemas/oplweb#
http://rdfs.org/sioc/ns#
http://purl.org/ontology/mo/
http://www.openlinksw.com/schema/attribution#
virtrdf-label
http://www.w3.org/2007/05/powder-s#
http://www.openlinksw.com/schemas/zillow
http://www.openlinksw.com/schemas/googleplus
http://www.openlinksw.com/schemas/twitter
http://www.openlinksw.com/schemas/linkedin
http://www.openlinksw.com/schemas/opengraph
http://www.openlinksw.com/schemas/oplbase
http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo
http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo

http://bdc.francetelecom.fr/graph/repository
http://bdc.orange.com/ontology/media/deezer/
http://bdc.orange.com/graph/ProcessConfiguration/SSearchSport
http://bdc.orange.com/graph/prod/deezer
http://testSILK
http://bdc.orange.com/graph/vodpc
http://www.openlinksw.com/schemas/RDF_Mapper_Ontology/1.0/
http://bblfish.net/work/atom-owl/2006-06-06/
http://open.vocab.org/terms
http://www.openlinksw.com/schemas/amazon
http://www.openlinksw.com/schemas/ebay
http://www.openlinksw.com/schemas/angel#
http://www.openlinksw.com/schemas/dbpedia-spotlight#
http://www.openlinksw.com/schemas/bestbuy
http://www.openlinksw.com/schemas/cert#
http://www.openlinksw.com/schemas/cnet
http://www.openlinksw.com/schemas/crunchbase
http://www.openlinksw.com/schemas/cv
http://www.openlinksw.com/schemas/evri
http://www.openlinksw.com/schemas/google-base
http://www.openlinksw.com/schemas/klout
http://www.openlinksw.com/schemas/money
http://www.openlinksw.com/schemas/nyt
http://www.openlinksw.com/schemas/stackoverflow#
http://www.openlinksw.com/schemas/tesco
http://www.openlinksw.com/schemas/wolframalpha#
http://scot-project.org/scot/ns#
http://bdc.orange.com/ref/prod/ShortcutBdCPeople

59 Rows. -- 84 msec.

SPARQL SELECT DISTINCT ?g WHERE { GRAPH ?g { ?s ?p ?o } };
g
LONG VARCHAR
_______________________________________________________________________________

http://www.openlinksw.com/schemas/virtrdf#
http://localhost:8890/sparql
http://localhost:8890/DAV/
http://www.w3.org/2002/07/owl#
http://www.w3.org/2002/07/owl
b3sifp
http://bdc.orange.com/graph/prod/wiki
http://bdc.orange.com/graph/prod/webedia
http://bdc.orange.com/graph/prod/deezer
http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo
http://bdc.orange.com/graph/ProcessConfiguration/SSearchPeople
http://bdc.orange.com/graph/ProcessConfiguration/SSearchSport
http://bdc.orange.com/graph/ProcessConfiguration/SSearchTele
http://www.openlinksw.com/schemas/RDF_Mapper_Ontology/1.0/
http://purl.org/ontology/bibo/
http://www.openlinksw.com/schemas/googleplus
http://www.w3.org/1999/02/22-rdf-syntax-ns#
http://www.w3.org/2000/01/rdf-schema#
http://www.w3.org/2004/02/skos/core
http://bdc.orange.com/graph/ProcessConfiguration/SSearchGeo

http://bdc.orange.com/graph/vodpc
http://purl.org/dc/terms/
http://xmlns.com/foaf/0.1/

23 Rows. -- 104 msec.

ghost commented 10 years ago

Any news about this issue ? If it can help I can give you a ssh connection to a machine with the same problem of missing graphs.

HughWilliams commented 10 years ago

Not solution yet ... ssh access would be good or if the database is not too large and a copy of the DB could be provided then would be ideal ...

ghost commented 10 years ago

I think I found something about this bug :

First I recompiled with the latest version of develop/7 git branch and installed this new version ;
Next I loaded dbpedia, fct and rdb2rdf packages and changed the dba password ;
Next I loaded three graphs :
- http://fr.dbpedia.org => 126449865 triples ;
- http://www.wikidata.dbpedia.org => 25584510 triples ;
- http://fr.wiktionary.dbpedia.org => 20616888 triples ;

With these commands : ld_dir('path', '*.nt.gz', 'graph_name'); and rdf_loader_run();.

Then after each loaded graph I displayed the graphs list from the conductor and each of these three display took around 20-30 minutes, and at the end I had this graphs list :

http://www.openlinksw.com/schemas/virtrdf#
http://data.lirmm.fr/sparql
http://data.lirmm.fr/DAV/
http://www.w3.org/2002/07/owl#
http://dbpedia.org/schema/property_rules#
dbprdf-label
b3sonto
b3sifp
urn:rules.skos
http://www.openlinksw.com/schemas/oplweb#
virtrdf-label
facets
http://www.w3.org/ns/r2rml#OVL
http://fr.dbpedia.org
http://www.wikidata.dbpedia.org
http://fr.wiktionary.dbpedia.org

Until there no problem but the time to give me this list because all the graphs was displayed. So I decided to restart Virtuoso to see if the list can be loaded faster after a restart, and yes the graphs list took few seconds to be displayed but unfortunately some graphs disapeared from the new graphs list, like you can see :

http://www.openlinksw.com/schemas/virtrdf#
http://data.lirmm.fr:8890/sparql
http://data.lirmm.fr:8890/DAV/
http://www.w3.org/2002/07/owl#
http://fr.dbpedia.org
b3sonto
virtrdf-label
facets
http://www.wikidata.dbpedia.org
http://fr.wiktionary.dbpedia.org

So I think with these details you can reproduce the bug easily as the bug come from the restart operation. I created you a ssh access and conductor access, I send you everything by email. The data are available there :

french dbpedia => http://data.lirmm.fr/download/dbpedia/ wikidata => http://data.lirmm.fr/download/wikidata/ french wiktionary => http://data.lirmm.fr/download/wiktionary/

I think I told you every details.

HughWilliams commented 10 years ago

I don't know what the "dba" pwd is for you instance and thus cannot access the Virtuoso server.

I am downloading the datasets locally and will try to recreate locally following you steps above ...

HughWilliams commented 10 years ago

I followed the steps you provided against a local database and could not recreate the issue the graph counts were returned instantly and the same:

SQL> vad_install ('../vad/dbpedia_dav.vad', 0); SQL_STATE SQL_MESSAGE VARCHAR VARCHAR

00000 No errors detected 00000 Installation of "DBpedia" is complete. 00000 Now making a final checkpoint. 00000 Final checkpoint is made. 00000 SUCCESS

6 Rows. -- 1235 msec. SQL> vad_install ('../vad/fct_dav.vad', 0); SQL_STATE SQL_MESSAGE VARCHAR VARCHAR

00000 GUI is accesible via http://host:port/fct 00000 Post-installation guide is available from http://host:port/fct/post_install.html 00000 No errors detected 00000 Installation of "Faceted Browser" is complete. 00000 Now making a final checkpoint. 00000 Final checkpoint is made. 00000 SUCCESS

8 Rows. -- 2625 msec. SQL> vad_install ('../vad/rdb2rdf_dav.vad', 0); SQL_STATE SQL_MESSAGE VARCHAR VARCHAR

00000 No errors detected 00000 Installation of "RDB2RDF support" is complete. 00000 Now making a final checkpoint. 00000 Final checkpoint is made. 00000 SUCCESS

6 Rows. -- 1345 msec. SQL> ld_dir_all('.', '*.nt.gz', 'graph_name');

Done. -- 37 msec. SQL> select * from load_list; ll_file ll_graph ll_state ll_started ll_done ll_host ll_work_time ll_error VARCHAR NOT NULL VARCHAR INTEGER TIMESTAMP TIMESTAMP INTEGER INTEGER VARCHAR

./dbpedia/frwiki-20140226-article-categories.nt.gz ./dbpedia/frwiki-20140226-article-templates.nt.gz ./dbpedia/frwiki-20140226-category-labels.nt.gz ./dbpedia/frwiki-20140226-disambiguations.nt.gz ./dbpedia/frwiki-20140226-external-links.nt.gz ./dbpedia/frwiki-20140226-french-population.nt.gz ./dbpedia/frwiki-20140226-geo-coordinates.nt.gz ./dbpedia/frwiki-20140226-homepages.nt.gz ./dbpedia/frwiki-20140226-images.nt.gz ./dbpedia/frwiki-20140226-infobox-properties.nt.gz ./dbpedia/frwiki-20140226-infobox-property-definitions.nt.gz ./dbpedia/frwiki-20140226-infobox-test.nt.gz ./dbpedia/frwiki-20140226-instance-types.nt.gz ./dbpedia/frwiki-20140226-interlanguage-links.nt.gz ./dbpedia/frwiki-20140226-labels.nt.gz ./dbpedia/frwiki-20140226-mappingbased-properties.nt.gz ./dbpedia/frwiki-20140226-page-ids.nt.gz ./dbpedia/frwiki-20140226-page-links.nt.gz ./dbpedia/frwiki-20140226-pnd.nt.gz ./dbpedia/frwiki-20140226-redirects.nt.gz ./dbpedia/frwiki-20140226-revision-ids.nt.gz ./dbpedia/frwiki-20140226-revision-uris.nt.gz ./dbpedia/frwiki-20140226-skos-categories.nt.gz ./dbpedia/frwiki-20140226-specific-mappingbased ./dbpedia/frwiki-20140226-topical-concepts.nt.gz ./dbpedia/frwiki-20140226-wikipedia-links.nt.gz ./wikidata/wikidatawiki-20140226-wikidata-labels.nt.gz ./wikidata/wikidatawiki-20140226-wikidata-ll.nt.gz ./wikidata/wikidatawiki-20140226-wikidata-mapped.nt.gz ./wikidata/wikidatawiki-20140226-wikidata-names ./wikidata/wikidatawiki-20140226-wikidata-sameas.nt.gz ./wikidata/wikidatawiki-20140226-wikidata.nt.gz ./wiktionary/frwiktionary-20140314-wiktionary.d http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL -properties.nt.gz http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://fr.dbpedia.org 0 NULL NULL NULL NULL NULL http://www.wikidata.dbpedia.org 0 NULL NULL NULL NULL NULL http://www.wikidata.dbpedia.org 0 NULL NULL NULL NULL NULL http://www.wikidata.dbpedia.org 0 NULL NULL NULL NULL NULL pace-sameas.nt.gz http://www.wikidata.dbpedia.org 0 NULL NULL NULL NULL NULL http://www.wikidata.dbpedia.org 0 NULL NULL NULL NULL NULL http://www.wikidata.dbpedia.org 0 NULL NULL NULL NULL NULL bpedia.org.nt.gz http://fr.wiktionary.dbpedia.org 0 NULL NULL NULL NULL NULL

33 Rows. -- 3 msec. SQL> rdf_loader_run();

Done. -- 1559811 msec. SQL> sparql select distinct ?g where {GRAPH ?g{?s a ?t}}; g VARCHAR

http://www.openlinksw.com/schemas/virtrdf# http://localhost:8890/sparql http://fr.dbpedia.org http://fr.wiktionary.dbpedia.org b3sifp http://www.openlinksw.com/schemas/oplweb# virtrdf-label http://www.w3.org/ns/r2rml#OVL

8 Rows. -- 778 msec. SQL> SPARQL SELECT distinct ?g WHERE { GRAPH ?g { ?s ?p ?o } }; g VARCHAR

http://www.openlinksw.com/schemas/virtrdf# http://localhost:8890/sparql http://localhost:8890/DAV/ b3sifp http://www.openlinksw.com/schemas/oplweb# http://fr.dbpedia.org http://www.wikidata.dbpedia.org http://fr.wiktionary.dbpedia.org

8 Rows. -- 46 msec. SQL> DB.DBA.SPARQL_SELECT_KNOWN_GRAPHS(); GRAPH_IRI VARCHAR

http://www.openlinksw.com/schemas/virtrdf# http://localhost:8890/sparql http://localhost:8890/DAV/ b3sifp http://www.openlinksw.com/schemas/oplweb# http://fr.dbpedia.org http://www.wikidata.dbpedia.org http://fr.wiktionary.dbpedia.org

8 Rows. -- 28 msec. SQL> SQL> status(''); REPORT VARCHAR

OpenLink Virtuoso Server Version 07.10.3208-pthreads for Linux as of Mar 24 2014 Started on: 2014-03-26 23:13 GMT+1

So we need to be able to access your database instance ...