search

brando90 / MathNet-large-scale-Mathematics-Dataset-for-Machine-Learning

1 stars 0 forks source link

write a formal unit test that shows that resetting seed works #62

Open brando90 opened 7 years ago

brando90 commented 7 years ago

Seems that setting the seed to form consistent format does not work. Can we check why? Whats up? it might imply there is a bigger bug with the seeding methodoly:

https://github.com/brando90/eit_proj1/blob/master/main_proj_lib/qagen/qagen.py https://github.com/brando90/eit_proj1/blob/master/main_proj_lib/qagen/qaops_test.py

brando90 commented 7 years ago

seems sympy changes the seed without our permission. This reveals a bigger flaw. If we allow the users to use arbitrary code then it could always be that they use some weird library that even if we reset their seed uses something like os.urand (that is indepedent of software) and do randomness. So basically we would need to check statically if they ever generate randomness without our permission. This is equivalent to the halting problem so we need a different solution.

brando90 commented 7 years ago

These are the unit tests we want:

  1. Seeding should guarantee that it does change the random state of their code (even if the code did it by itself). Here we want to guarantee that variation is given no matter what.

  2. if their variable initializers, Q, A dont return deterministic stuff given a seed, then they have a bug and warn them. Here we want that A always solves the same question correctly given the question,

  3. have Q and A returns delayed executions. Write a test for it. This helps to check that the user is actually using seqg, perg, choiceg. This test is only meant to aid the user and remind them they should return a choiceg object etc. Note resolve can always be used here as it goes through its