SNL-WaterPower / WecOptTool-MATLAB

WEC Design Optimization Toolbox
GNU General Public License v3.0
12 stars 9 forks source link

Calculate standard error on PS results #203

Closed H0R5E closed 1 year ago

H0R5E commented 3 years ago

Description

This commit uses the central value theorem to facilitate calculation of the error on the mean results of the PS controller, using the standard error from a normal distribution.

Two new functions are added to the WecOptTool.math package called standardErrorMeasure and standardErrorReduce, the first of which measures the standard error after a given number of samples and the second of which keeps sampling until a desired level of error is found.

An example script to show how the standard error varies with the stroke of the RM3 device using the PS controller is added to the RM3 example directory called PSError.m. The results of this simulation are shown here:

RM3_PS_standard_error

Fixes #56

Checklist:

H0R5E commented 3 years ago

Hi Ryan,

I implemented your algorithm, but I'm not sure it's behaving the way you expected it to do. The diagram below shows a simulation with the tolerance set to 1e-6.

ryan_algorithm

I don't think this is converging as you were expecting, but I'm not really surprised because of the random numbers.

I'm also not sure what the tolerance is telling you. I set the tolerance to 1e-4 and ran the algorithm 10 times. The range on the results was 69039.8 Watts, but 1e-4 * max(power) is 395.2 Watts, so the tolerance is kind of misleading.

ryan_algorithm_2

How do you want to proceed?

H0R5E commented 3 years ago

I've added a metric which calculates the sum of the standard errors over the sum of the mean (per frequency component). It produces results as such:

RM3_PS_standard_error_of_mean

I'll do the reverse and set a limit on the error and show how many iterations are required.

H0R5E commented 3 years ago

OK, so here is the same situation recording the number of iterations required:

RM3_PS_standard_error_N

So, I really think the question is whether the user wants to set a particular accuracy and wait for the realisations, or choose a number of realisations and suffer the inaccuracy. I suggest we offer both options.

H0R5E commented 3 years ago

Just a note that this would need to be updated if #170 is merged first and visa-versa.

H0R5E commented 3 years ago

As part of the testing process I noticed that when summing the errors the results were somewhat better than expected. Turns out, that's because summing the standard error is not the correct approach and taking the 2norm is the correct way to go. I've implemented that and the results are now more consistent.

H0R5E commented 3 years ago

Add option when reducing error for max samples - return error.

Done.

H0R5E commented 3 years ago

@ryancoe