ICOS-Carbon-Portal / pylib

Python library for direct access to ICOS time series data.
12 stars 3 forks source link

Disable cache control #121

Closed ZogopZ closed 1 year ago

ZogopZ commented 1 year ago

Sometimes using SPARQL to fetch data might fail. The python request will succeed, giving you an OK status_code, but the actual response is an incomplete json object. This happens due to the response being too big in size, and the cache being limited to a specific number of bytes. Example run:

from icoscp.sparql.runsparql import RunSparql

object_specification = '<http://meta.icos-cp.eu/resources/cpmeta/atcLosGatosL0DataObject>' + '<http://meta.icos-cp.eu/resources/cpmeta/atcPicarroL0DataObject>'
query = (
    f"prefix cpmeta: <http://meta.icos-cp.eu/ontologies/cpmeta/>\n"
    f"prefix prov: <http://www.w3.org/ns/prov#>\n"
    f"select ?dobj ?hasNextVersion ?spec ?fileName ?size ?submTime ?timeStart ?timeEnd\n"
    f"where {{\n"
    f"  VALUES ?spec {{{object_specification}}}\n"
    f"  ?dobj cpmeta:hasObjectSpec ?spec .\n"
    f"  ?dobj cpmeta:hasSizeInBytes ?size .\n"
    f"  ?dobj cpmeta:hasName ?fileName .\n"
    f"  ?dobj cpmeta:wasSubmittedBy/prov:endedAtTime ?submTime .\n"
    f"  ?dobj cpmeta:hasStartTime | (cpmeta:wasAcquiredBy/prov:startedAtTime) ?timeStart .\n"
    f"  ?dobj cpmeta:hasEndTime | (cpmeta:wasAcquiredBy/prov:endedAtTime) ?timeEnd .\n"
    f"  FILTER NOT EXISTS {{[] cpmeta:isNextVersionOf ?dobj}}\n"
    f"  }}\n"
    f"order by desc(?submTime)\n"
)
raw_data = RunSparql(sparql_query=query, output_format='json').run()

and the error:

SPARQL RESPONSE TOO LARGE TO BE CACHED.
The largest cacheable response size is 8388608 bytes.
Try running the query with 'Cache-Control: no-cache' to get full response