Functional Test Results

[atareen]

Functional Tests for Suftware

@jbkinney @ChenCSHL

This issue contains the results of running functional tests on suftware from code in the jbk_dev branch. The initial model for these tests is to find limiting values of each of the parameters in density object constructor for as many distributions and datasets possible, starting with the normal distribution.

The numbers under the parameter names indicate the individual type of test I ran. [PASS] or [FAIL] above the parameter name denotes whether a parameter works successfully or not.

Note that this series of tests is on-going work, I am putting them here so we are familiar with the current of suftware. Note also that these tests are currently being done with python 3. python 2 will follow soon.

The results of the tests and notes for each of the parameters are recorded as follows:

**data:**

    The first set of tests are done with a Gaussian distribution. Once parameter ranges
    are known for this distribution, these tests will be run for all other data sets.
    in loops

[PASS] alpha:

    1) alpha works between 1 and 4.
        values of alpha outside this range return the following message,
        'Input check failed. Parameter "alpha" must be 1 <= alpha <= 4: alpha = -1'
        which seems reasonable.
    2) Non-integer values return the message:
        'Input check failed. Parameter "alpha" must be an integer: alpha = <class \'float\'>'
        which could be cleaned up a little.

    Notes: We may need to auto-correct wrong parameter values or ignore them just so the
          program does not exit.

[PASS] grid:

    1) grid = None works
    2) Manual grids are failing. For the Gaussian distribution. I tried
        test_grid = np.linspace(-3,3,100),np.linspace(-2,2,100),np.linspace(-1,1,100)...
        and a variety of combinations.

    Note: this parameter still fails after the latest update. 

[PASS] grid_spacing:

    1) negative values for grid spacing fail.
    2) a value of 0.0 and 0 fails.
    3) 1.5, 1.4 ,1.3, 1.2, 1.1, 1, 1e-1, 1e-2 work
    4) 1.6 and bigger values fail
    5) 1e-3 fail, will try smaller values after this is fixed.

    Notes: Need to handle broken cases and need to impose limits on this parameter.

[PASS] num_grid_points:

    1) for alpha = 1, num_grid_points = 2 and 3 fails. >= 4 and <=1000 works as expected
    2) for num_grid_points < 2*alpha, (alpha = 2 and 3), we get a message
        'Input check failed. Parameter "num_grid_points" must between [2*alpha, 1000]: 
         num_grid_points= 5', which I think is reasonable. This test works for all values between 6 and 
        1000.
    3) num_grid_points prints a message if the type of the parameter isn't an integer:
        'Input check failed. Parameter "num_grid_points" must be an integer: num_grid_points = <class 
         \'float\'>', which is reasonable

    Notes: Need to fix functional test #1 for this parameter.

[PASS] bounding_box:

    1) assertion errors if bounding box is list or tuple. Bounding box shouldn't just be an np.array
    2) np.array([False,True]): this works as a legitimate bbox, we should probably not allow this
    3) limiting case 1: np.array([0,0.01]) breaks
    4) limiting case 2: np.array([-34,34]) breaks also
    5) Values of bbox as np array between the limiting cases work well.

    Notes: Need check if entries in np.array entered as a bbox are valid. Currently not handled.
    e.g. np.array([-3,-4]) should return a reasonable message as opposed to breaking.

[PASS] periodic: 1) This parameter works for both True and False. 2) If I set this to something other than a boolean value, I get a reasonable message: 'Input check failed. Parameter "periodic" must be of type boolean: periodic = <class \'float\'>'

[FAIL] Z_evaluation_method:

    1) Any value of this parameter other than ['Lap', 'Lap+Imp', 'Lap+Fey'] produces the message
        "Input check failed. Z_evaluation_method must be in ['Lap', 'Lap+Imp', 'Lap+Fey']: 
            Z_evaluation_method = 1"
            which is reasonable
    2) The values 'Lap' and 'Lap+Fey' work, 'Lap+Imp' doesn't work.

[FAIL] num_samples_for_Z:

    Notes: This parameter will be ignored until we fix the issue with Z_evaluation method (that issue 
    needs to be fixed in deft_core)

[PASS] max_t_step:

1) negative values and 0.0 and 0 give the following reasonable message:
    'Input check failed. Parameter "max_t_step" must be > 0: max_t_step = 0'
2) instances different from positive floats give a reasonable message, e.g.:
    'Input check failed. Parameter "max_t_step" must be a number: max_t_step = <class \'str\'>'
3) 1e-4, 1e-3,1e-2,1e-1...1e1 all the way upto 1e8 works.

[PASS] print_t:

    1) True and False works. Instances besides boolean return a reasonable message:
        'Input check failed. Parameter "print_t" must be a boolean: print_t = <class \'float\'>'

    Notes: We should make the printed values more readable for the user with this variable is
        set to true

[PASS] tolerance:

    1) values other than float produce a reasonable message, e.g.:
        'Input check failed. Parameter "tolerance" must be a float: tolerance = <class \'bool\'>'
    2)  Negative floats are handled appropriately, returning the following message:
        'Input check failed. Parameter "tolerance" must be > 0: tolerance = -1.0'
    3) Tolerance works well between 1e-10 and 1e10, so long as the input is a float. 

[FAIL] resolution:

    1) negative values and instances other than floats are handled appropriately e.g.:
            'Input check failed. Parameter "resolution" must be > 0: resolution = 0'
    2) 1e-5 fails.
    3) 1e-4 and higher values work. I haven't found a reasonable upper limit.

    Notes: Need to handle #2 for this to pass.

[PASS] seed:

    1) None works. Negative and floating values are handled appropriately. e.g.
           'Input check failed. Parameter "seed" must be either None or an integer: seed = <class v 
             \'float\'>'
    2) positive integers work well

    Notes: We should probably impose lower and upper limits on this parameter. Right now I couldn't
    find a reasonable upper and lower limit.

Need to test the following parameteres

num_posterior_samples:
sample_only_at_l_star:
max_log_evidence_ratio_drop:

Each of the methods in the suftware child classes will also be functionally test.

The skeleton code used for these tests is:

import numpy as np
import matplotlib.pyplot as plt
import sys
import suftware as su
data = su.simulate_density_data(distribution_type='gaussian',N=100)
Q = su.Density(data)

[jbkinney]

@atareen Fixed issue with grid

[jbkinney]

@atareen Fixed grid_spacing issue Fixed num_grid_points issue Fixed bounding_box issue. Let's not worry about [False, True] working.

[atareen]

@jbkinney : Very nice. alpha = 1 and num_grid_points = 2 is failing still.

I like the updated error messages.

[atareen]

All functional tests are passing so I am closing the issue.