Include functionality to handle problematic timestamps and clock errors

[cdeline]

This should be a high priority. From our analyst intercomparison, the majority of differences between analyst has to do with incorrect timestamps, and incorrect application of daylight saving time shift.

[mdeceglie]

To start, it would be helpful to generate a specific list of the problems we would like to help users avoid. Two on my list are:

1. Time zone of pandas index is completely wrong, doesn't point to the correct moment (e.g. UTC time) of the measurement
2. Time zone is geographically correct but DST is not handled the same was as it was by the logger/DAQ

We should consider requiring timezone aware pandas indices (or at least raising a warning).

[wholmgren]

I'd be interested in seeing your solution incorporated into pvlib's new iotools subpackage. Happy to collaborate on this if you're interested. Also would be useful for our new Solar Forecasting 2 project. cc @cwhanse

[wholmgren]

I should add that if you guys are in a rush or just want to retain more control over your workflow then keeping it within rdtools makes sense too.

[mikofski]

I wonder if any of the timeseries qc tests/filters tools from Pecos could be useful? sorry, if irrelevant. cc @kaklise

[kaklise]

That sounds interesting, let me know if you want to collaborate on that.

Kate

From: Mark Mikofski [mailto:notifications@github.com] Sent: Monday, October 8, 2018 2:59 PM To: NREL/rdtools rdtools@noreply.github.com Cc: Klise, Katherine A kaklise@sandia.gov; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [NREL/rdtools] Include functionality to handle problematic timestamps and clock errors (#44)

I wonder if any of the timeseries qc tests/filters tools from Pecoshttps://pecos.readthedocs.io/en/latest/qc_tests.html could be useful? sorry, if irrelevant. cc @kaklisehttps://github.com/kaklise

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/NREL/rdtools/issues/44#issuecomment-427977160, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ARGr45xyqQPmCHcMubGZfW_DSnfP8Yoaks5ui7yagaJpZM4PpBhI.

[mdeceglie]

@wholmgren and @kaklise we'd be happy to work together. Perhaps once the solution takes shape we can figure out which package would be the best home to make sure it reaches its audience and contributes to a well-thought-out ecosystem for PV analysis and modeling.

Step 0 is to come up with some examples of the problems we're trying to solve, we'll work on that.

(thanks for the suggestion @mikofski, not irrelevant!)

[mdeceglie]

This kind of QA functionality may be a better fit for https://github.com/pvlib/pvanalytics (which didn't exist when we first considered it)