read_raw_history() returns not all values

[djdtimit]

Hey,

I'm trying to read historic values from the free Procys OPC UA Simulation Server for one day (15.04.2019 10:00:00 - 16.04.2019 10:00:00) and NodeId "ns=5;s=Counter1".

The Procys OPC UA Client returns approx. 86000 values for this period.

In Python I'm using the following code with setting "numvalues = 100000":

counter1 = opc_client.get_node("ns=5;s=Counter1")  
starttime = "15.04.2019 10:00:00"  
starttime_dt_format = datetime.datetime.strptime(starttime,"%d.%m.%Y %H:%M:%S")  
endtime = "16.04.2019 10:00:00"  
endtime_dt_format = datetime.datetime.strptime(endtime,"%d.%m.%Y %H:%M:%S")  
counter1.read_raw_history(starttime=starttime_dt_format, endtime=endtime_dt_format, numvalues=100000)  

But only 1000 values instead of approx. 86000 values are returned.

What is the reason for this difference and how could I get the whole 86000 values?

Thanks Tim

[schroeder-]

The reason is that the server can change the numvalues and only return 1000 and give the client an continuation point to continue the request. The current client api doesn't support that feature out of the box. But you can implement it your self using client.history_read.

Maybe it helps if have done this for multiple nodes.

from opcua import Client
from opcua import ua

def do_transaction_history(client, nodes, start, end, data_per_req, contes=None):
    if contes is None:
        contes = [b''] * len(nodes)
    details = ua.ReadRawModifiedDetails()
    details.IsReadModified = False
    details.StartTime = start
    details.EndTime = end
    details.NumValuesPerNode = data_per_req
    details.ReturnBounds = True

    params = ua.HistoryReadParameters()
    params.HistoryReadDetails = details
    params.TimestampsToReturn = ua.TimestampsToReturn.Both
    params.ReleaseContinuationPoints = False
    for n, c in zip(nodes, contes):
        valueid = ua.HistoryReadValueId()
        valueid.NodeId = n.nodeid
        valueid.IndexRange = ''
        valueid.ContinuationPoint = c
        params.NodesToRead.append(valueid)
        result = client.uaclient.history_read(params)
    return result

def test_procdata_history(client):
    start = datetime.datetime(2016, 3, 13, 5, tzinfo=datetime.timezone.utc)
    end = datetime.datetime(2020, 3, 13, 5, tzinfo=datetime.timezone.utc)
    nodes = [client.get_node(ua.NodeId(33001 + x, 4)) for x in range(10)]
    more_data = True
    cnt = 0
    contes = None
    data_per_req = history_per_request
    values = []
    while more_data:
        print(data_per_req)
        result = do_transaction_history(client, nodes, start, end, data_per_req, contes)
        cnt += len(result[0].HistoryData.DataValues)
        more_data = False
        values += result[0].HistoryData.DataValues
        more_data = result[0].StatusCode.is_good() and result[0].ContinuationPoint is not None
        if cnt >= history_limit:
            more_data = False
        if history_limit - cnt < history_per_request:
            data_per_req = history_limit - cnt
        if more_data:
            contes = [res.ContinuationPoint for res in result]

[zerox1212]

The reason the server might only supply 1000 items is for server performance. The server doesn't want to make timeouts for other clients while it tries to retrieve such a huge number of values.

[kozachynskyi]

Hi, it's not relevant any more, right @schroeder- @zerox1212 ? Can we close the issue?