Trial run to investigate error effects of non-zero delta

[greenTara]

We will use two scripts to investigate the effects of delta:

1. the "MM" script (which runs faster than the QMM script) to get heatmaps, and corresponding numerical values with taumeta on the horizontal axis, with values of just error (not performance), and just for bayes (not for naive), averaged over numruns runs.
2. the QMM script to get parallel results to the first, but for delta = 1/8

[greenTara]

The purpose is to determine what numruns needs to be to distinguish delta=0 from delta = 1/8.

[greenTara]

First we print out the errors for the MM scripts after averageing over 2, 4 and 8 runs. We can use this to evaluate how close we are to the actual expectation of the error, and to predict what the error would be for 16 runs.

[greenTara]

Second we print out the parallel errors for the QMM scripts with delta = 1/8. This should give us enough information to select a value of numruns that will give us sufficiently accurate computation of the expectation of the error, so that we can compare the effect of changing delta.

[alexlafleur]

Average Errors Run 1-2: [array([[-3.47353914, -4.45701716, -5.13950525], [-3.97987578, -5.02509966, -5.44087834], [-4.47290525, -5.57498663, -5.80400199]]), array([[-3.45201229, -4.42831078, -5.12893552], [-3.97987578, -5.02509966, -5.44087834], [-4.48758795, -5.58262855, -5.80293912]]), array([[-2.88837582, -3.74672001, -4.61105779], [-3.41353809, -4.3673471 , -5.23309221], [-3.9909425 , -4.98843404, -5.71775497]])] Average Errors Run 1-4: [array([[-3.36643247, -4.09021552, -4.92575815], [-3.8716255 , -4.60920369, -5.24532033], [-4.35712849, -5.09488531, -5.47841397]]), array([[-3.34063415, -4.06403929, -4.90725631], [-3.8716255 , -4.60920369, -5.24532033], [-4.37134826, -5.10382162, -5.48322792]]), array([[-2.76058521, -3.54037912, -4.41049051], [-3.30974249, -4.0918174 , -5.04491004], [-3.87862866, -4.77148279, -5.58563878]])] Average Errors Run 1-8: [array([[-3.22223761, -4.03251531, -5.03787018], [-3.71168053, -4.53556666, -5.37636787], [-4.1659853 , -4.96390431, -5.59587952]]), array([[-3.1949843 , -4.00695361, -5.01645588], [-3.71168053, -4.53556666, -5.37636787], [-4.17670124, -4.9678966 , -5.61611683]]), array([[-2.69137939, -3.52381605, -4.37071104], [-3.21742656, -4.04663648, -5.05207792], [-3.71112431, -4.65122694, -5.60605293]])]

[greenTara]

These errors indicate that the accuracy is about 10% with 8 runs, as in all entries, the value appears to have two significant figures. We can expect the expectation to converge as the square root of the number of runs, so that means double the runs has a multiplier of 1/squareroot(2). To get another significant figure, we need about 5 such multipliers, so that means a factor of 32 on the runs (256 runs).

On Tue, Nov 15, 2016 at 9:43 AM, Alexandra La Fleur < notifications@github.com> wrote:

Average Errors Run 1-2: [array([[-1.88093458, -2.71686497, -3.60755385], [-1.91357182, -2.74574782, -3.6551762 ], [-1.92765883, -2.7580714 , -3.68488187]]), array([[-1.87799924, -2.71596494, -3.60113799], [-1.91357182, -2.74574782, -3.6551762 ], [-1.92781358, -2.75790778, -3.68358005]]), array([[-1.79948771, -2.64111551, -3.50863986], [-1.86683799, -2.71856082, -3.57676251], [-1.92068519, -2.76510473, -3.68955744]])]

Average Errors Run 1-4: [array([[-1.79463279, -2.60093794, -3.54837208], [-1.83963474, -2.63129166, -3.59782583], [-1.86559807, -2.64440289, -3.62798217]]), array([[-1.79123639, -2.59900763, -3.54310805], [-1.83963474, -2.63129166, -3.59782583], [-1.86615054, -2.64411639, -3.62679358]]), array([[-1.71341504, -2.52645986, -3.42589508], [-1.78298235, -2.61703027, -3.5439346 ], [-1.83540226, -2.63508334, -3.59925761]])]

Average Errors Run 1-8: [array([[-1.80306574, -2.60908356, -3.54753706], [-1.86936242, -2.66617601, -3.61087225], [-1.90097156, -2.69888865, -3.64969267]]), array([[-1.79932429, -2.60714174, -3.54045783], [-1.86936242, -2.66617601, -3.61087225], [-1.90177187, -2.69878874, -3.64981979]]), array([[-1.70647284, -2.51479206, -3.31557204], [-1.81694475, -2.61188808, -3.56503651], [-1.87082977, -2.7155192 , -3.60628365]])]

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ag-csw/LDStreamHMMLearn/issues/25#issuecomment-260659279, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMMVQIM7yqjAe1oWjfUl1qUzBQ-5gomks5q-cURgaJpZM4Kyf4j .

[greenTara]

How long did it take to do these 8 runs with the MM script?

[alexlafleur]

It took about 2 minutes

[alexlafleur]

MM Delta=0

MM Delta=0 Error Values

Average Errors Run 1-2: [array([[-4.06213075, -4.91986446, -5.9423074 ], [-4.49030057, -5.34600912, -6.38005423], [-4.89197218, -5.71637183, -6.76279817]]), array([[-4.03666902, -4.89278453, -5.91098235], [-4.49030057, -5.34600912, -6.38005423], [-4.897968 , -5.72638015, -6.75866188]]), array([[-3.45419691, -4.450008 , -5.17642037], [-4.03666902, -4.89278453, -5.91098235], [-4.62381138, -5.16156467, -6.27132667]])]

[alexlafleur]

Parameter Overview for ETA: taumeta: 2 eta: 8 scale_window : 16 shift: 16 window_size: 256 num_estimations: 256 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 65792 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for ETA: taumeta: 2 eta: 16 scale_window : 16 shift: 32 window_size: 512 num_estimations: 120 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 61952 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for ETA: taumeta: 2 eta: 32 scale_window : 16 shift: 64 window_size: 1024 num_estimations: 52 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 54272 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for ETA: taumeta: 4 eta: 8 scale_window : 16 shift: 32 window_size: 512 num_estimations: 120 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 61952 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for ETA: taumeta: 4 eta: 16 scale_window : 16 shift: 64 window_size: 1024 num_estimations: 52 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 54272 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for ETA: taumeta: 4 eta: 32 scale_window : 16 shift: 128 window_size: 2048 num_estimations: 18 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 38912 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for ETA: taumeta: 8 eta: 8 scale_window : 16 shift: 64 window_size: 1024 num_estimations: 52 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 54272 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for ETA: taumeta: 8 eta: 16 scale_window : 16 shift: 128 window_size: 2048 num_estimations: 18 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 38912 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for ETA: taumeta: 8 eta: 32 scale_window : 16 shift: 256 window_size: 4096 num_estimations: 1 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 8192 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for scale_window: taumeta: 2 eta: 16 scale_window : 8 shift: 32 window_size: 256 num_estimations: 128 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 33024 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for scale_window: taumeta: 2 eta: 16 scale_window : 16 shift: 32 window_size: 512 num_estimations: 120 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 61952 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for scale_window: taumeta: 2 eta: 16 scale_window : 32 shift: 32 window_size: 1024 num_estimations: 104 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 107520 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for scale_window: taumeta: 4 eta: 16 scale_window : 8 shift: 64 window_size: 512 num_estimations: 60 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 31232 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for scale_window: taumeta: 4 eta: 16 scale_window : 16 shift: 64 window_size: 1024 num_estimations: 52 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 54272 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for scale_window: taumeta: 4 eta: 16 scale_window : 32 shift: 64 window_size: 2048 num_estimations: 36 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 75776 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for scale_window: taumeta: 8 eta: 16 scale_window : 8 shift: 128 window_size: 1024 num_estimations: 26 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 27648 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for scale_window: taumeta: 8 eta: 16 scale_window : 16 shift: 128 window_size: 2048 num_estimations: 18 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 38912 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for scale_window: taumeta: 8 eta: 16 scale_window : 32 shift: 128 window_size: 4096 num_estimations: 2 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 12288 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for NUM_TRAJ: taumeta: 2 eta: 16 scale_window : 16 shift: 32 window_size: 256 num_estimations: 128 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 33024 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for NUM_TRAJ: taumeta: 2 eta: 16 scale_window : 16 shift: 32 window_size: 256 num_estimations: 60 len_trajectory: 2177 num_trajectories: 4 num_trajectories_len_trajectory: 8708 NAIVE window_size * num_estimations 15616 BAYES window_size + num_estimations_shift 2176

[[ 1. 1. 1. ..., 3. 3. 3.] [ 2. 2. 2. ..., 3. 3. 2.] [ 1. 0. 0. ..., 2. 3. 1.] [ 2. 2. 3. ..., 2. 2. 2.]] Parameter Overview for NUM_TRAJ: taumeta: 2 eta: 16 scale_window : 16 shift: 32 window_size: 256 num_estimations: 26 len_trajectory: 1089 num_trajectories: 8 num_trajectories_len_trajectory: 8712 NAIVE window_size * num_estimations 6912 BAYES window_size + num_estimations_shift 1088

[[ 1. 1. 1. ..., 3. 3. 3.] [ 2. 2. 2. ..., 2. 2. 2.] [ 1. 0. 0. ..., 1. 1. 1.] ..., [ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 1. ..., 1. 3. 3.] [ 0. 0. 0. ..., 2. 2. 3.]] Parameter Overview for NUM_TRAJ: taumeta: 4 eta: 16 scale_window : 16 shift: 64 window_size: 512 num_estimations: 60 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 31232 BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.] [ 3. 3. 3. ..., 3. 3. 3.]] Parameter Overview for NUM_TRAJ: taumeta: 4 eta: 16 scale_window : 16 shift: 64 window_size: 512 num_estimations: 26 len_trajectory: 2177 num_trajectories: 4 num_trajectories_len_trajectory: 8708 NAIVE window_size * num_estimations 13824 BAYES window_size + num_estimations_shift 2176

[[ 1. 1. 1. ..., 3. 3. 3.] [ 3. 3. 3. ..., 2. 3. 3.] [ 2. 2. 2. ..., 3. 3. 3.] [ 2. 2. 2. ..., 2. 2. 2.]] Parameter Overview for NUM_TRAJ: taumeta: 4 eta: 16 scale_window : 16 shift: 64 window_size: 512 num_estimations: 9 len_trajectory: 1089 num_trajectories: 8 num_trajectories_len_trajectory: 8712 NAIVE window_size * num_estimations 5120 BAYES window_size + num_estimations_shift 1088

[[ 1. 1. 1. ..., 3. 3. 2.] [ 3. 3. 3. ..., 1. 1. 1.] [ 2. 2. 2. ..., 2. 2. 1.] ..., [ 0. 0. 2. ..., 1. 1. 1.] [ 2. 2. 2. ..., 3. 3. 1.] [ 1. 1. 1. ..., 1. 1. 1.]] Parameter Overview for NUM_TRAJ: taumeta: 8 eta: 16 scale_window : 16 shift: 128 window_size: 1024 num_estimations: 26 len_trajectory: 4353 num_trajectories: 2 num_trajectories_len_trajectory: 8706 NAIVE window_size * num_estimations 27648 BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.] [ 2. 2. 2. ..., 1. 1. 1.]] Parameter Overview for NUM_TRAJ: taumeta: 8 eta: 16 scale_window : 16 shift: 128 window_size: 1024 num_estimations: 9 len_trajectory: 2177 num_trajectories: 4 num_trajectories_len_trajectory: 8708 NAIVE window_size * num_estimations 10240 BAYES window_size + num_estimations_shift 2176

[[ 3. 3. 3. ..., 1. 1. 1.] [ 2. 2. 2. ..., 1. 1. 1.] [ 2. 2. 2. ..., 1. 1. 1.] [ 1. 1. 1. ..., 2. 2. 3.]] Parameter Overview for NUM_TRAJ: taumeta: 8 eta: 16 scale_window : 16 shift: 128 window_size: 1024 num_estimations: 0 len_trajectory: 1089 num_trajectories: 8 num_trajectories_len_trajectory: 8712 NAIVE window_size * num_estimations 1024 BAYES window_size + num_estimations_shift 1024