SPARQL delete where not working

[hansidm]

I have the following behaviour when dealing with DELETE WHERE clauses.

Test 1:

delete where { <http://localhost:9000/hadatac/kb/DP/00000001/000000005> ?x ?y }

When I have no triples to be deleted, I get status 0.

<response>
    <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst>
</response>

Test 2:

delete where { <http://localhost:9000/hadatac/kb/DP/00000001/000000005> ?x ?y }

When I do have triples to be deleted, I get the following error:

<response>
    <lst name="responseHeader">
        <int name="status">400</int>
        <int name="QTime">3</int>
    </lst>
    <lst name="error">
        <str name="msg">&lt;SOLRDF-00099&gt; : Invalid update request: delete where { &lt;http://localhost:9000/hadatac/kb/DP/00000001/000000005&gt; ?x ?y }
        </str>
        <int name="code">400</int>
    </lst>
</response>

Test 3

PREFIX vstoi: <http://hadatac.org/ont/vstoi#>
DELETE
{
    <http://localhost:9000/hadatac/kb/DP/00000001/000000001> ?p ?o .
}
WHERE {
    <http://localhost:9000/hadatac/kb/DP/00000001/000000001> a vstoi:Deployment .
}

I get the following (doesn't matter if I have triples do delete or not):

<response>
    <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst>
</response>

But when I do have triples to delete, they are not deleted and are still on the triple store. I can confirm that by using:

describe <http://localhost:9000/hadatac/kb/DP/00000001/000000001>

[hansidm]

Update

On Test 2, I noticed that the DELETE indeed deletes some triples, but not them all. It is crashing in the process of deleting and returning a Bad Request 400.

[agazzarini]

Hi @handsim I think reproduced the problem. I said "think" because I don't have your dataset. The problem seems in a wrong date parsing....do you have some parse error message in the log?

[hansidm]

Hi @agazzarini, here is the stacktrace that I am getting when trying Test 1:

org.apache.solr.common.SolrException: Invalid Date String:'2015-09-29T08' at org.apache.solr.schema.DateField.parseMath(DateField.java:184) at org.apache.solr.schema.TrieField.readableToIndexed(TrieField.java:402) at org.apache.solr.schema.TrieField.readableToIndexed(TrieField.java:379) at org.apache.solr.schema.TrieDateField.readableToIndexed(TrieDateField.java:132) at org.apache.solr.schema.FieldType.readableToIndexed(FieldType.java:398) at org.apache.solr.schema.FieldType.getFieldQuery(FieldType.java:750) at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:737) at org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:541) at org.apache.solr.parser.QueryParser.Term(QueryParser.java:299) at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185) at org.apache.solr.parser.QueryParser.Query(QueryParser.java:139) at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96) at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151) at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50) at org.apache.solr.search.QParser.getQuery(QParser.java:141) at org.apache.solr.update.DirectUpdateHandler2.getQuery(DirectUpdateHandler2.java:354) at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:384) at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:80) at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:931) at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1433) at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1226) at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121) at org.gazzax.labs.solrdf.graph.standalone.LocalGraph.performDelete(LocalGraph.java:214) at com.hp.hpl.jena.graph.impl.GraphBase.delete(GraphBase.java:225) at org.gazzax.labs.solrdf.graph.DatasetGraphSupertypeLayer.deleteFromDftGraph(DatasetGraphSupertypeLayer.java:87) at com.hp.hpl.jena.sparql.core.DatasetGraphTriplesQuads.delete(DatasetGraphTriplesQuads.java:56) at com.hp.hpl.jena.sparql.core.DatasetGraphTriplesQuads.delete(DatasetGraphTriplesQuads.java:40) at com.hp.hpl.jena.sparql.core.DatasetGraphWrapper.delete(DatasetGraphWrapper.java:92) at com.hp.hpl.jena.sparql.modify.UpdateEngineWorker.execDelete(UpdateEngineWorker.java:517) at com.hp.hpl.jena.sparql.modify.UpdateEngineWorker.visit(UpdateEngineWorker.java:371) at com.hp.hpl.jena.sparql.modify.request.UpdateDeleteWhere.visit(UpdateDeleteWhere.java:38) at com.hp.hpl.jena.sparql.modify.UpdateVisitorSink.send(UpdateVisitorSink.java:46) at com.hp.hpl.jena.sparql.modify.UpdateVisitorSink.send(UpdateVisitorSink.java:26) at com.hp.hpl.jena.sparql.modify.UsingUpdateSink.send(UsingUpdateSink.java:61) at com.hp.hpl.jena.sparql.modify.UsingUpdateSink.send(UsingUpdateSink.java:31) at com.hp.hpl.jena.sparql.lang.SPARQLParserBase.emitUpdate(SPARQLParserBase.java:194) at com.hp.hpl.jena.sparql.lang.sparql_11.SPARQLParser11.Update1(SPARQLParser11.java:1145) at com.hp.hpl.jena.sparql.lang.sparql_11.SPARQLParser11.Update(SPARQLParser11.java:1084) at com.hp.hpl.jena.sparql.lang.sparql_11.SPARQLParser11.UpdateUnit(SPARQLParser11.java:51) at com.hp.hpl.jena.sparql.lang.ParserSPARQL11Update._parse(ParserSPARQL11Update.java:60) at com.hp.hpl.jena.sparql.lang.ParserSPARQL11Update.parse$(ParserSPARQL11Update.java:46) at com.hp.hpl.jena.sparql.lang.UpdateParser.parse(UpdateParser.java:48) at com.hp.hpl.jena.update.UpdateAction.parseExecute(UpdateAction.java:438) at com.hp.hpl.jena.update.UpdateAction.parseExecute(UpdateAction.java:395) at com.hp.hpl.jena.update.UpdateAction.parseExecute(UpdateAction.java:371) at com.hp.hpl.jena.update.UpdateAction.parseExecute(UpdateAction.java:360) at org.gazzax.labs.solrdf.handler.update.Sparql11UpdateRdfDataLoader.execute(Sparql11UpdateRdfDataLoader.java:91) at org.gazzax.labs.solrdf.handler.update.Sparql11UpdateRdfDataLoader.load(Sparql11UpdateRdfDataLoader.java:75) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.gazzax.labs.solrdf.handler.search.handler.Sparql11SearchHandler.handleRequestBody(Sparql11SearchHandler.java:76) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)

And the triple that is not deleted (and is giving this error):

<http://localhost:9000/hadatac/kb/DP/00000001/000000001>
        <http://www.w3.org/ns/prov#startedAtTime>
                "2015-09-29T08:41:00.000Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> .

[agazzarini]

Argh, my bet was right :) ok that's a matter of date parsing. I will manage that as soon as possible and I will formalize the fix in a test case so we can have a regression, too.

[agazzarini]

Hi @hansidm, issue has been finally fixed ;)

I created 4 integration tests [1] for replicating the scenarios described above. Specifically all those tests use a dataset [2] which contains dates (as objects). Each of them executes a (delete) query with a different combination of variables within the same triple pattern:

?s ?p ?o 
?s <:date>  ?o    
<:xyz> ?p ?o .       
?s ?p "2010-10-10"^^xsd:date  

---

[1] https://goo.gl/kOQ4qn [2] https://goo.gl/GSSxfA

[hansidm]

Hi @agazzarini,

I can't find the use cases containing xsd:dateTime, only xsd:date. I have compiled the latest sources and the problem is still happening when trying to delete a triple with a xsd:dateTime object.

Also, I have tried inserting as xsd:data, but it won't let me insert a date and time under the type xsd:date.

Thanks.

[paulopinheiro1234]

Hi @agazzarini ,

The insert below may provide all you need to test the delete of an object with a date/time property. The insert works fine. The delete will generate a 400 bad request result. When we do a describe in the object after to bad request result, we see that some of the triples of the object were deleted, but not the triple with the data/time property.

INSERT DATA { http://localhost:9000/hadatac/kb/DP/00000001/000131313 a http://hadatac.org/ont/vstoi#Deployment ; http://hadatac.org/ont/hasneto#hasDetector http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#VaisalaTHERMOCAP-0003 ; http://hadatac.org/ont/hasneto#hasInstrument http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#WXT520-0003 ; http://hadatac.org/ont/vstoi#hasPlatform http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#JP_WX_CEDAR ; http://www.w3.org/ns/prov#startedAtTime "2015-10-06T05:28:00.000Z"^^http://www.w3.org/2001/XMLSchema#dateTime }

Thank you a lot for your help. Getting this problem fixed is one of our major concerns these days.

Cheers, Paulo

[agazzarini]

Hi Paulo, Henrique, many thanks I will check that immediately

Best, Andrea

2015-10-15 4:55 GMT+02:00 Paulo Pinheiro notifications@github.com:

Hi @agazzarini https://github.com/agazzarini ,

The insert below may provide all you need to test the delete of an object with a date/time property. The insert works fine. The delete will generate a 400 bad request result. When we do a describe in the object after to bad request result, we see that some of the triples of the object were deleted, but not the triple with the data/time property.

INSERT DATA { http://localhost:9000/hadatac/kb/DP/00000001/000131313 a http://hadatac.org/ont/vstoi#Deployment ; http://hadatac.org/ont/hasneto#hasDetector http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#VaisalaTHERMOCAP-0003 ; http://hadatac.org/ont/hasneto#hasInstrument http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#WXT520-0003 ; http://hadatac.org/ont/vstoi#hasPlatform http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#JP_WX_CEDAR ; http://www.w3.org/ns/prov#startedAtTime "2015-10-06T05:28:00.000Z"^^ http://www.w3.org/2001/XMLSchema#dateTime }

Thank you a lot for your help. Getting this problem fixed is one of our major concerns these days.

Cheers, Paulo

— Reply to this email directly or view it on GitHub https://github.com/agazzarini/SolRDF/issues/112#issuecomment-148266501.

[paulopinheiro1234]

Hi Andrea,

We are very thankful for all the work you have to done developing solrdf, which is core for our work on hadatac.

Regarding issue #112, if it would be possible for us to handle timestamps with date instead of datetime, we would do it. Unfortunately, date is not an option for us.

Many thanks, Paulo On Oct 15, 2015 3:26 AM, "Andrea Gazzarini" notifications@github.com wrote:

Hi Paulo, Henrique, many thanks I will check that immediately

Best, Andrea

2015-10-15 4:55 GMT+02:00 Paulo Pinheiro notifications@github.com:

Hi @agazzarini https://github.com/agazzarini ,

The insert below may provide all you need to test the delete of an object with a date/time property. The insert works fine. The delete will generate a 400 bad request result. When we do a describe in the object after to bad request result, we see that some of the triples of the object were deleted, but not the triple with the data/time property.

INSERT DATA { http://localhost:9000/hadatac/kb/DP/00000001/000131313 a http://hadatac.org/ont/vstoi#Deployment ; http://hadatac.org/ont/hasneto#hasDetector http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#VaisalaTHERMOCAP-0003 ; http://hadatac.org/ont/hasneto#hasInstrument http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#WXT520-0003 ; http://hadatac.org/ont/vstoi#hasPlatform http://jefferson.tw.rpi.edu/ontology/jp-kb.owl#JP_WX_CEDAR ; http://www.w3.org/ns/prov#startedAtTime "2015-10-06T05:28:00.000Z"^^ http://www.w3.org/2001/XMLSchema#dateTime }

Thank you a lot for your help. Getting this problem fixed is one of our major concerns these days.

Cheers, Paulo

— Reply to this email directly or view it on GitHub <https://github.com/agazzarini/SolRDF/issues/112#issuecomment-148266501 .

— Reply to this email directly or view it on GitHub https://github.com/agazzarini/SolRDF/issues/112#issuecomment-148304021.

[agazzarini]

Hi @paulopinheiro1234, I'm proud to hear that; about the #112: absolutely no problem at all, this is an issue and it must be solved. I don't like very much workarounds

Best, Andrea

[agazzarini]

Hi @paulopinheiro1234 @hansidm I just committed another patch. I added a method in the testcase above that is reproducing the scenario described by @paulopinheiro1234.

Let me know if you have other scenarios, I think everything should be fixed but date and datetime management is a complicated topic so some side-effectcould be still possible.

I'll leave open the issue for some time

Andrea

[paulopinheiro1234]

We tested on our side, and it appears to be working fine now. That you a lot, @agazzarini!