How can I archive data in openPDC

[siaDaniel]

I create a new device as virtual device, and I choose the PPA as historian. Then I create some measurements for the device.

By using the CSVInputAdapter, I save data to those measurements I just created. But I find the openPDC doesn't archive my data online or offline. I create a CSVOutputAdapter to confirm this. When I stop the inputadapter, I can't get the output data from outputadapter any more, this is why I said it's failed for online. When I restart the openPDC service, I can't get the output data neither, this is why I said it's failed for offline.

So I'm confused about what the openPDC does. Is it just a real-time data transfer tool and doesn't save data in files or other place? If it is true, what the option "Historian" for when I create the device?

Please help me to figure this. Do I use it in a totally wrong way? How can I archive data in openPDC both online and offline?

Thanks very much.

[StephenCWills]

At its core, openPDC is, as you said, a real-time data transfer tool. However, the openPDC does support the transfer of data to historian archives, databases, and CSV data files, some of which are hosted directly in the openPDC's adapters, so it would be somewhat incorrect to say that it doesn't save data to files or other places.

The problem you are having is more likely a configuration error of some kind. The openPDC has a number of diagnostic tools available to help you gather more details about the error. I would recommend looking into the ErrorLog.txt file which should be located in the openPDC's installation folder to see if there are any errors being written there. You may also consider running the openPDC Console so you can monitor the status of the openPDC service.

I'm sure I don't understand what you mean by "online" and "offline," but if you are using a CSVOutputAdapter to test, make sure that you have configured your filesystem permissions so that the openPDC has sufficient rights to write to the CSV file. By default, the openPDC runs as the NT SERVICE\openPDC user so if you are trying to write the file to My Documents or Desktop or something like that, it likely will not have the necessary permissions to be allowed to do that.

[siaDaniel]

@StephenCWills Thank you for your help. I had some personal issues yesterday, so I didn't reply you in time. Sorry about that.

I do the csv adapter test again, and the error log shows an error about "GSF.ServiceProcess.ServiceHelper.RemotingServer_ReceiveClientDataComplete(sender As Object, e As EventArgs`3)", I suppose it may not about the input or the output Adapter. And the openPDCConsole shows the system running correctly.

I know the openPDC can save data in files, for example csv file, and other places, for example relational databases. And it does save data in csv file successfully, so I can confirm there is no permission issue.

Actually I'm sure I don't understand how the historian works. What is the historian's function.

The offline mentioned in my question is about history data. I mean, if I restart the service, the data I "saved"(if openPDC saves data) in openPDC before can not be found any more. Doesn't the openPDC put the history data in any places? Does historian help to do this? If I misunderstand it, please tell me what the historian really does. And if the openPDC really saves history data.

Maybe I can't make myself clear that much. Look forward to your answer.

[siaDaniel]

@StephenCWills I read your answer again.

If you are indicating that the historian is a "place" that saves data, just like files and databases which I choose as an outputadapter type. If I want history data, I must get data from there, because openPDC doesn't save history data itself, so it won't provide history data when I required.

The only way I can get history data is reading data from databases, files or historians. openPDC doesn't offer that. Is it true?

I read some infomation from wikipedia: "The project includes a historian for data archival. The files produced by the historian can later be analyzed using Hadoop.[3] The historian is also designed to be used in distributed systems, having a single node dedicated to archival and several others sending concentrated measurements to the central archive. This design even allows for its use in any distributed system enabling one PDC to filter measurements to another that is running a resource intensive adapter.[9] The local historian provides web services which can be used to monitor the data being archived in real-time."

Is the "historian" in wikipedia just the historian I choose when I create a device, for example PPA.

Am I asking a very stupid question?

[StephenCWills]

@siaDaniel Your question is not stupid at all. The openPDC does include a local data historian which can be configured to store your input data and can be used in a variety of ways. The simplest and most common way to access that data is to use the Historian Trending Tool, which provides a simple interface through which you can view archived data on a line chart or export to a CSV file. The Historian Trending Tool should have been included when you installed the openPDC on your system.

Let me also mention, the local historian will write to the Archive directory located in the openPDC installation folder by default. If the openPDC is properly configured to archive your data, you will most likely see a file in that folder called ppa_archive.d.

[siaDaniel]

@StephenCWills Thanks a lot.

As you say, I find the file /Archive/ppa_archive.d. And use the HistorianPlaybackUtility(I have a running problem with HistorianView) to export data to a csv file, and the data is exactly what I have saved by openPDC.

I can consider historian as a default storage for openPDC, it will save data whether I want it or not. But I still have a little question. Can I read archive data through my custom output adapter? That is to say, does GSF offer some methods or interfaces to analyse archive data files. Because I can get real-time data by my custom output adapter for now, and I want history data as well.

Thanks!

[StephenCWills]

@siaDaniel Yes, it is possible. The first thing I'll mention is the Historian API, which you should be able to use in any .NET application to programmatically query data from an archive for any given time range. The Historian Trending Tool and Historian Playback Utility are both examples of applications that use this interface. You can find the source code for each of these applications in the Grid Solutions Framework.

https://github.com/GridProtectionAlliance/gsf/tree/master/Source/Tools/HistorianView https://github.com/GridProtectionAlliance/gsf/tree/master/Source/Tools/HistorianPlaybackUtility

Alternatively, you can request that the openPDC use the archive to stream historic data through your adapter in what we call a temporal session. You can see an example of this type of processing in the openPDC Manager's Graph Measurements page when you configure the "Historical Playback" controls to process a historic time range.

The only thing you need to do in your adapter to make this happen is to return true from the SupportsTemporalProcessing property. This will indicate to the system that your adapter should be included in the temporal session created by the openPDC Manager.

public override bool SupportsTemporalProcessing
{
    get
    {
        return true;
    }
}

You can also query historic data from a temporal session into another application programmatically using the Subscriber API. There is a sample application in GSF demonstrating the usage of the Subscriber API, though it doesn't show how to use it to access history.

https://github.com/GridProtectionAlliance/gsf/blob/master/Source/Applications/DataSubscriberTest/Program.cs

[siaDaniel]

@StephenCWills Got it. I really appreciate your help very much.

[Palberz]

on a somehow similar topic, I need your help guys. I want to automatically save each hour of the data as a separate csv file with 13 columns (voltage a, its angle, voltage b, its angle, voltage c, its angle, current a, its angle, current b, its angle, current c, its angle, frequency) for each data frame. But, with the current configuration, I only have the data as a single column and 13 rows for each frame. Beside, each file is only for 5 minutes. what should I do?