Fix the test that checks Q,A are deterministic

[brando90]

I recently noticed an issue with Test_all_funcs_with_seed_are_deterministic. Locally it looks as:

class Test_all_funcs_with_seed_are_deterministic(unittest.TestCase):
    '''
    Checks that given a seed the framework and your code indeed behave deterministically.
    If this does not pass then your code is random in a way the framework cannot
    control. This is bad.
    '''

    def __init__(self,qa_constructor,args,kwargs):
        super().__init__()
        self.qa_constructor = qa_constructor
        self.args = args
        self.kwargs = kwargs

    def runTest (self):
        # TODO how do I stop hardcoding the tests here, just go over your
        print()
        qagenerator = self.qa_constructor()
        #
        seed = 0 # random.randint()
        original_latex_flag = qagenerator.use_latex
        q_original,a_original = qagenerator.get_single_qa(seed=seed)
        # the latex flag should still be as original
        self.assertEqual(original_latex_flag,qagenerator.use_latex)
        #
        qagenerator.use_latex = True
        seed = 1 # random.randint()
        original_latex_flag = qagenerator.use_latex
        q_original,a_original = qagenerator.get_single_qa(seed=seed)
        # the latex flag should still be as original
        self.assertEqual(original_latex_flag,qagenerator.use_latex)
        #
        qagenerator.use_latex = False
        seed = 2 # random.randint()
        original_latex_flag = qagenerator.use_latex
        q_original,a_original = qagenerator.get_single_qa(seed=seed)
        # the latex flag should still be as original
        self.assertEqual(original_latex_flag,qagenerator.use_latex)

    # def runTest (self):
    #     # TODO how do I stop hardcoding the tests here, just go over your
    #     qagenerator = self.qa_constructor()
    #     seed = 0 # random.randint(0,500)
    #     q_original,a_original = qagenerator.get_single_qa(seed=seed)
    #     for i in range(500):
    #         # set seed
    #         qagenerator.seed_all(seed)
    #         # get variables for qa and register them for the current q,a
    #         variables, variables_consistent = qagenerator._create_all_variables()
    #         # get concrete qa strings
    #         q = qagenerator.Q(*variables,*variables_consistent)
    #         a = qagenerator.A(*variables,*variables_consistent)
    #         #
    #         print()
    #         print('q_original: ',q_original)
    #         print('q: ',q)
    #         #
    #         self.assertEqual( q,q_original )
    #         self.assertEqual( a,a_original )

    #
    # def test_keywords(self):
    # TODO
    #     self.assertTrue(len(self.QAFormat.keywords) > 0)

but thats not what it should be I believe. Lets try to fix it.

[brando90]

It should behave as its comment describes it:

    '''
    Checks that given a seed the framework and your code indeed behave deterministically.
    If this does not pass then your code is random in a way the framework cannot
    control. This is bad.
    '''

[brando90]

oh I think I figured out what I went to do. What I meant to do is check that a specific instance of QA always produces the same Q,A given a specific seed, so it tries to generate the same QA multiple times and if it doesn't match what it should be complains.

        for i in range(500):
            # set seed
            qagenerator.seed_all(seed)
            # get variables for qa and register them for the current q,a
            variables, variables_consistent = qagenerator._create_all_variables()
            # get concrete qa strings
            q = qagenerator.Q(*variables,*variables_consistent)
            a = qagenerator.A(*variables,*variables_consistent)
            #
            self.assertEqual( q,q_original )
            self.assertEqual( a,a_original )

[brando90]

I wrote in the past:

        qag = self.qa_constructor()
        seed = 0 # random.randint(0,500)
        #q_original,a_original = qag.get_single_qa(seed=seed)
        a,b = qag.get_symbols(2)
        qag.seed_all(seed)
        print('a,b: ', a,b)
        #
        qag = self.qa_constructor()
        seed = 0 # random.randint(0,500)
        qag.seed_all(seed)
        #q_original,a_original = qag.get_single_qa(seed=seed)
        a,b = qag.get_symbols(2)
        print('a,b: ', a,b)

though it doesn't actually check for determinism cuz QA object will remember the list of symbols its already used (cuz it tries to avoid duplicates I believe). Just throwing this here just in case we realize a analogous test that does more or less the same thing? maybe its not necessary.