Open gfoo opened 4 years ago
@benjamingeer and @tobiasschweizer I shared with you on switchdrive the file export-LL-23.10.2019.trig.zip
(x-trig export of my graphdb repos)
To reproduce with gravsearch (being the user fantin.reichler@unil.ch
/ 9e1601a8
) :
PREFIX knora-api: <http://api.knora.org/ontology/knora-api/v2#>
PREFIX knora-simple-api: <http://api.knora.org/ontology/knora-api/simple/v2#>
PREFIX ll: <http://0.0.0.0:3333/ontology/0113/lumieres-lausanne/v2#>
CONSTRUCT {
?Collection knora-api:isMainResource true .
?Collection ll:collectionHasTitle ?collectionHasTitle .
?Collection ll:collectionHasUrlElement ?collectionHasUrlElement .
?Collection ll:collectionHasDescription ?collectionHasDescription .
?Collection ll:hasBibliographicNoticeElement ?hasBibliographicNoticeElement .
?Collection ll:hasTranscriptionElement ?hasTranscriptionElement .
?Collection ll:hasPersonElement ?hasPersonElement .
} WHERE {
?Collection a ll:Collection .
?Collection ll:collectionHasTitle ?collectionHasTitle .
?Collection ll:collectionHasUrlElement ?collectionHasUrlElement .
OPTIONAL {
?Collection ll:collectionHasDescription ?collectionHasDescription .
}
OPTIONAL {
?Collection ll:hasBibliographicNoticeElement ?hasBibliographicNoticeElement .
?hasBibliographicNoticeElement a ll:BibliographicNotice .
}
OPTIONAL {
?Collection ll:hasTranscriptionElement ?hasTranscriptionElement .
?hasTranscriptionElement a ll:Transcription .
}
OPTIONAL {
?Collection ll:hasPersonElement ?hasPersonElement .
?hasPersonElement a ll:Person .
}
}
ORDER BY ASC(?collectionHasTitle)
OFFSET 2
The related resource and its value reported by the error seem to be fine:
.../v2/resources/http%3A%2F%2Frdfh.ch%2F0113%2F9h2huqLgS4aG9d3xLgXcIQ
{
"@id": "http://rdfh.ch/0113/9h2huqLgS4aG9d3xLgXcIQ/values/-szOyfUkRnGYbF0UULjnUg",
"@type": "knora-api:LinkValue",
"knora-api:arkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/9h2huqLgS4aG9d3xLgXcIQ8/4TyOg7WDQFyuRQ2vYMUuBQo"
},
"knora-api:attachedToUser": {
"@id": "http://rdfh.ch/users/lumieres-lausanne-fantin_reichler"
},
"knora-api:hasPermissions": "CR http://rdfh.ch/groups/0113/lumieres-lausanne-administrator,http://rdfh.ch/groups/0113/lumieres-lausanne-director|D knora-admin:Creator|M http://rdfh.ch/groups/0113/lumieres-lausanne-phdstudent,http://rdfh.ch/groups/0113/lumieres-lausanne-researcher,http://rdfh.ch/groups/0113/lumieres-lausanne-student|V http://rdfh.ch/groups/0113/lumieres-lausanne-user,knora-admin:KnownUser,knora-admin:UnknownUser",
"knora-api:linkValueHasTarget": {
"@id": "http://rdfh.ch/0113/oDFFjxYWQ-iz6WBKwhtnCA",
"@type": "lumieres-lausanne:BibliographicNotice",
"knora-api:arkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/oDFFjxYWQ=iz6WBKwhtnCAY"
},
"knora-api:attachedToProject": {
"@id": "http://rdfh.ch/projects/0113"
},
"knora-api:attachedToUser": {
"@id": "http://rdfh.ch/users/lumieres-lausanne-fantin_reichler"
},
"knora-api:creationDate": {
"@type": "xsd:dateTimeStamp",
"@value": "2017-03-07T12:23:41Z"
},
"knora-api:hasPermissions": "CR http://rdfh.ch/groups/0113/lumieres-lausanne-administrator,http://rdfh.ch/groups/0113/lumieres-lausanne-director|D knora-admin:Creator|M http://rdfh.ch/groups/0113/lumieres-lausanne-phdstudent,http://rdfh.ch/groups/0113/lumieres-lausanne-researcher|V http://rdfh.ch/groups/0113/lumieres-lausanne-student,http://rdfh.ch/groups/0113/lumieres-lausanne-user,knora-admin:KnownUser,knora-admin:UnknownUser",
"knora-api:userHasPermission": "D",
"knora-api:versionArkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/oDFFjxYWQ=iz6WBKwhtnCAY.20170307T122341Z"
},
"rdfs:label": "Qu'est-ce qu'un médecin des Lumières? Portraits et discours croisés de quelques contemporains de Tissot"
},
"knora-api:userHasPermission": "D",
"knora-api:valueCreationDate": {
"@type": "xsd:dateTimeStamp",
"@value": "2019-10-22T15:30:12.399105Z"
},
"knora-api:valueHasUUID": "4TyOg7WDQFyuRQ2vYMUuBQ",
"knora-api:versionArkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/9h2huqLgS4aG9d3xLgXcIQ8/4TyOg7WDQFyuRQ2vYMUuBQo.20191022T153012399105Z"
}
},
Is the resource very large? If so, it could be that the SPARQL query result is truncated. @tobiasschweizer would you have time to look at this?
Is the resource very large?
If so, it could be that the SPARQL query result is truncated.
Yes, I think we had such problems before. But you removed the size limit, didn't you?
It's still there in the scripts in src/webapi/scripts
:
owlim:query-limit-results "50000";
@gfoo Could you just try to make the limit higher and see if this fixes the problem? You have to change it it the template file, e.g., webapi/scripts/graphdb-se-knora-prod-repository-config.ttl.tmpl
.
I thought it would be good to have both a time and size limit to protect agains overload.
Of course, it is hard so say what a reasonable limit is. Could we make some recommendations in terms of how many properties a resource should have at max.? I really do not want to implement paging for resource requests.
Relevant discussion with @mrivoal here:
https://github.com/dasch-swiss/knora-api/issues/1320#issuecomment-494300576
In the past I have spotted problems when GraphDB couldn't serialize a response because it was too large. This could also be problem with fewer properties that have large values like texts, see https://github.com/dhlab-basel/knora-large-texts.
In the past I have spotted problems when GraphDB couldn't serialize a response because it was too large. This could also be problem with fewer properties that have large values like texts
Yes, I think I've seen that too, but haven't had time to investigate it yet. I wonder if there's a configurable response size limit in GraphDB (in KB rather than in triples).
I wonder if there's a configurable response size limit in GraphDB (in KB rather than in triples).
Good question. I couldn't find anything like this on http://graphdb.ontotext.com/documentation/free/configuring-a-repository.html#configure-a-repository-programmatically
It would be nice to know how many bytes GraphDB is actually returning in this case. That could give us a clue.
(and how many statements)
It would be nice to know how many bytes GraphDB is actually returning in this case. That could give us a clue.
Haven't you already written some kind of logic that could be used for this in the code that handles the communication with the triplestore?
In HttpTriplestoreConnector.getSparqlHttpResponse
, you could print the length of the response string before returning it:
If it's an extended construct response, it will be parsed by ConstructResponseTurtleHandler
, which could count the statements in handleStatement
and print out the statement count in endRDF
:
@gfoo Could you just try to make the limit higher and see if this fixes the problem? You have to change it it the template file, e.g., webapi/scripts/graphdb-se-knora-prod-repository-config.ttl.tmpl.
yes it works with owlim:query-limit-results "100000"
@gfoo Good to know. The problem with owlim:query-limit-results
is that it just truncates the result if it's too big. It would be better to have a setting that returned an error if the result was too big.
yes it works with owlim:query-limit-results "100000"
So, where's your problem then? ;-)
Anyway, in my particular case I could also to not ask I could also avoid asking has*Element
properties to really limit data retrieved.
The main problem there is to understand the error :)
I get this error on a gravsearch query: