QC updates in Pecos

[kaklise]

I'm working on updates to Pecos that will facilitate streaming analysis of continuous datasets. This work also includes the ability to include user defined QC functions and create latex formatted reports. Most of these changes do not impact the Pecos functions that are used by MHKiT, with the exception of a few minor changes to input arguments. The custom QC functions are available using the object-oriented and functional approach.

These updates are located on my fork in the IDAES branch (https://github.com/kaklise/pecos/tree/IDAES). I plan to push these updates to the main fork soon and then tag a new release in about a month. I'll coordinate with the MHKiT team to make sure these updates are compatible.

[kaklise]

The updates have been merged into the master branch of Pecos. Before I tag the release, we'll need to coordinate minor updates needed on the MHKiT side.

[ssolson]

@kaklise thanks for coordinating this with us! In an email you state the following need to take place:

In MHKiT-Python:

• No changes are needed in the MHKiT python package
• The check_delta test used in the qc_example jupyter notebook uses the input argument ‘window’ which is no longer optional (you could change window=window to window) but the example runs.

In MHKiT-MATLAB,

• check_delta, the order of key and window need to be updated, window is no longer optional
• check_outlier, the order of key and window need to be updated (window is still optional). An additional argument has been added to use streaming analysis. This is set to False by default to mimic previous behavior. More info on streaming vs. static analysis is included in the updated Pecos documentation. Now that I look at the Matlab docs, I realize that you have to set previous arguments before later ones are used. If it makes life easier, I could move streaming to the end.
• Run QC MATLAB Live example to make sure it works.

This looks like a pretty easy update on the Python side. I will pull the pecos master, remove the window option, and check that everything is working smoothly for me on the Python side @rpauly18 would you take point of the MATLAB side?

[rpauly18]

@ssolson will do on updating the Matlab side. The question will be, do we want to tag a new release, especially on pip to reflect these changes? We should plan to make these changes available as soon as @kaklise tags the new Pecos release.

[ssolson]

Sound good to me. v0.3.1?

[rpauly18]

Yes, v0.3.1 would be fine with me.

[rpauly18]

@kaklise I have the changes ready on the Matlab side ready to be merged as soon as Pecos is released. Let us know when that has happened.

[rpauly18]

@ssolson we also will need to update the requirements.txt to require the latest pandas since in the Matlab side I added the new optional argument for check_outlier. Can you make that update along with the python example notebook?

[ssolson]

Can you specify the version of Pandas that is required now?

[rpauly18]

Can you specify the version of Pandas that is required now?

Sorry, I meant to type Pecos, not pandas. Do you know what version Pecos is being tagged too?

[kaklise]

Pecos will be tagged for version 0.1.9

[kaklise]

Pecos 0.1.9 has been tagged and is available to install through PyPI.

[rpauly18]

@ssolson I have not seen any changes on the Python side yet, will you have time to submit those soon? If not, I can get tot them first thing tomorrow morning.

[ssolson]

The changes are "Optional" as stated above. My new and previously uncommunicated plan (sorry) was to update the call to window in the QC example following the PECOS release. Does that sound like a good idea?

[ssolson]

Looks like it is tagged so I will go ahead 😁

[ssolson]

Closed with #79. I think you should be good to go with the MATLAB side now?

...the final step would then be tagging a new release. Which should be a relatively straightforward rebuilding of the docs and tagging. Does this sound right to you?

[rpauly18]

Yes, we will need to rebuild the docs and do the tags. Do you know how to tag on GitHub and PyPi or would you like for me to handle it?

[ssolson]

I do not know how, I would like to work with you to learn but it looks you have already tagged it. When I click on the documentation v0.3.0 still shows up. Will this update later?

[rpauly18]

I have only tagged on GitHub so far. I remembered that we need to change the version in init.py, which I have submitted a PR for. I will email you the instructions for tagging on PyPi. Then I will merge and tag the Matlab side. I want to update the Docs last because there will be changes on the Matlab side that will need to be incorporated too. If you want we can jump on a call later this afternoon ~3 or 3:30 to discuss the workflow in more detail.