the parallel Fitter does not produce results equivalent to the serial Fitter

This #187 comment points out that 1-core and 8-core Fitter runs created different size kb/simData_Fit_1.cPickle files. This is dubious.

Are the results equivalent? Does it just mean some data fields are in a different order and data-dependent encoding (compression or variable-length offsets or whatever) accounts for the different sizes?

Make these objects support an == test so it's straightforward to test that the cPickle contents are functional equivalent.
Write a test program. It will be slow although we could speed it up by using the Fitter's debug feature if we make that feature select a consistent transcription factor rather than an arbitrary one.
Run the test in the nightly build, or the PR build, or maybe less often.

Agreed on all three points. Nightly build would probably be frequent enough.

I'm not sure how practical it is (@tahorst would have a much better idea), but functionalizing these operations would make it easier to avoid serial/parallel differences and would likely cut down on parallelization overhead (right now a huge chunk of memory has to be copied in/out). If functionalization isn't practical, then maybe we need to trim sim_data down. Ideally sim_data would strictly be model parameters and raw_data would be, as the name implies, data taken directly from primary sources and loaded into Python objects. (A third piece which often gets associated with both objects is 'structural' data like the reaction stoichiometries, which aren't parameters in the normal sense.)

Now that I think about it, maybe raw_data should be a module instead of a class/instance, since it (should be) static. Then it's much easier to load piecemeal (which is great for e.g. parallel operations that don't need to see most of the raw_data). In contrast, one big sim_data object makes more sense because those parameter values all belong together, and need to be passed simultaneously to initialize a simulation. Just some food for thought - historically, raw_data and sim_data were originally the same object.

I think supporting == and running it as part of the build that gets run any time a new commit is added to master would be good. That build already runs a serial and parallel fitter task so it would be a simple comparison. I think they should be equivalent but it's possible something gets set within the function so functionalizing everything would make things more clear.

As @jmason42 points out elsewhere, seeding the random number generators differently is one potential cause for different parallel/serial Fitter results.

The parallel Fitter does not produce equivalent output, at least not on macOS, judging by the output from diff_simouts.py.

Below, out/manual/ has the output from a parallel Fitter run w/12 worker processes then one simulation generation, while out/x4/ has the output from a serial Fitter run then one simulation generation.

Parallel Fitter run: 3m 44s Serial Fitter run: 15m 55s -- 4.3x

Comparing: ('out/manual/wildtype_000000/000000/generation_000000/000000/simOut/', 'out/x4/wildtype_000000/000000/generation_000000/000000/simOut/')

{'BulkMolecules/counts': Arrays are not equal (mismatch 7.04940792673e-06%) x: array([[0, 0, 0, ..., 1, 1, 1], [0, 0, 0, ...,...,
 'EnzymeKinetics/actualFluxes': Arrays are not equal (mismatch 6.95512548153%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+00, ...,
 'EnzymeKinetics/countsToMolar': Arrays are not equal (mismatch 0.271923861319%) x: array([[0.000000e+00], [1.364496e-06], [1.364496e...,
 'EnzymeKinetics/metaboliteConcentrations': Arrays are not equal (mismatch 0.271923861319%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+00,...,
 'EnzymeKinetics/metaboliteCountsFinal': Arrays are not equal (mismatch 0.00192853802353%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+0...,
 'EnzymeKinetics/metaboliteCountsInit': Arrays are not equal (mismatch 0.00192853802353%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+0...,
 'EnzymeKinetics/targetFluxes': Arrays are not equal (mismatch 0.263426240653%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+00,...,
 'FBAResults/deltaMetabolites': Arrays are not equal (mismatch 0.000723201758831%) x: array([[ 0.0000e+00, 0.0000e+00, 0.0000e+00, ....,
 'FBAResults/externalExchangeFluxes': Arrays are not equal (mismatch 15.9860330017%) x: array([[ 0.000000e+00, 0.000000e+00, 0.000000e+00,...,
 'FBAResults/homeostaticObjectiveValues': Arrays are not equal (mismatch 0.00289280703531%) x: array([[0. , 0. , 0. , ..., 0. , 0. , 0. ], [0....,
 'FBAResults/kineticObjectiveValues': Arrays are not equal (mismatch 6.20166978246%) x: array([[ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. ...,
 'FBAResults/objectiveValue': Arrays are not equal (mismatch 99.9660095173%) x: array([[0. ], [0.021562], [0.016488],... y: array(...,
 'FBAResults/reactionFluxes': Arrays are not equal (mismatch 7.96058783244%) x: array([[0. , 0. , 0. , ..., 0. , 0. , 0. ], [0. , ...,
 'FBAResults/reducedCosts': Arrays are not equal (mismatch 22.7123313665%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+00, ...,
 'FBAResults/shadowPrices': Arrays are not equal (mismatch 98.7348790566%) x: array([[ 0.000000e+00, 0.000000e+00, 0.000000e+00,...,
 'GrowthLimits/aaRequestSize': Arrays are not equal (mismatch 0.249263539542%) x: array([[0.000000e+00, 0.000000e+00, 0.000000e+00,...,
 'Main@endTime': (u'2018-11-28 14:09:16', u'2018-11-27 23:53:58'),
 'Main@startTime': (u'2018-11-28 14:00:03', u'2018-11-27 23:45:34'),
 'Mass/cellMass': Arrays are not equal (mismatch 0.271923861319%) x: array([[1339.133243], [1339.133239], [1339.133235...,
 'Mass/cellVolume': Arrays are not equal (mismatch 0.271923861319%) x: array([[1.217394], [1.217394], [1.217394],... y: ...,
 'Mass/dryMass': Arrays are not equal (mismatch 0.271923861319%) x: array([[403.237634], [403.32835 ], [403.391711],....,
 'Mass/growth': Arrays are not equal (mismatch 0.238014280857%) x: array([0.090716, 0.063361, 0.043779, ..., 0.16633...,
 'Mass/instantaniousGrowthRate': Arrays are not equal (mismatch 0.306018361102%) x: array([0.001125, 0.000785, 0.000543, ..., 0.00023...,
 'Mass/processMassDifferences': Arrays are not equal (mismatch 0.0183025675888%) x: array([[ 0.000000e+00, 0.000000e+00, 0.000000e+0...,
 'Mass/relProcessMassDifferences': Arrays are not equal (mismatch 0.0235318726141%) x: array([[ 0.000000e+00, 0.000000e+00, 0.000000e+0...,
 'Mass/waterMass': Arrays are not equal (mismatch 0.271923861319%) x: array([[ 935.89561 ], [ 935.804889], [ 935.741524...,
 'ReplicationData/criticalMassPerOriC': Arrays are not equal (mismatch 0.271923861319%) x: array([[0. ], [0.686735], [0.686735],... y: array...,
 'RnaDegradationListener/DiffRelativeFirstOrderDecay': Arrays are not equal (mismatch 20.4962610469%) x: array([[0. ], [0.094397], [0.098627],... y: array(...,
 'RnaDegradationListener/FractionActiveEndoRNases': Arrays are not equal (mismatch 0.271923861319%) x: array([[0. ], [0.050895], [0.050949],... y: array...,
 'RnaSynthProb/rnaSynthProb': Arrays are not equal (mismatch 86.7990600192%) x: array([[0. , 0. , 0. , ..., 0. , 0. , 0. ], [0. , ...}

Hmm looks like the major issue is rnaSynthProb. I'd expect that difference to eventually trickle down to some of the other processes. Any ideas how to troubleshoot? I would guess that one of the functions in fit_sim_data_1 that are parallelized updates something in place in addition to returning values. Might be difficult to track down exactly where it's different without a tool to compare sim_data objects.

Yes, to troubleshoot we need testable hypotheses.

Hypothesis: Parallelized functions in fit_sim_data_1 update something in place in addition to returning values.
- Test: Write code to compare sim_data objects, then compare snapshots before/after each of the parallelized function calls.
- Test: Write-protect the sim_data object (e.g. putting immutable wrappers like frozendict around dictionaries), then try calling the parallelized functions.
- Test: Pickle+unpickle the sim_data data (to make a fully deep copy) to send to the parallelized functions, so they can't modify the callers. See if that changes the result. (Isn't this what happens when passing args to a function in a worker process?)
Hypothesis: Something directly or indirectly called is initializing a random number generator from external state like the time.
- Test: Set breakpoints on those random number initializers or monkey-patch them, then run the Fitter and Simulation.
Hypothesis: [TODO]

Hypothesis: Some third party/native library has state that is shared between python processes and is sensitive to access order/count.
- Test: ??? [TODO]

Through analysis we can definitely narrow down the search for where the differences arise. If we want to compare sim_data then https://github.com/CovertLab/wcEcoli/pull/282 is already most of the way there. I can put together a quick tree diff for that, it should tell us where the problem is.

I do know that a function operating in parallel cannot reach back and modify anything in the parent process. At least, that is how multiprocessing works (any necessary communication is mediated entirely through its Queue class), but that library is dead to me now. So that should cut out a whole class of hypotheses.

Hypothesis: Some third party/native library has state that is shared between python processes and is sensitive to access order/count. [Ryan's hypothesis, above.]
- Test: exec those python processes instead of forking them.
Hypothesis: The bug is in the serial Fitter, e.g. some processing steps unintentionally modify the sim_data that goes to later steps, which the same steps running in subprocesses shouldn't be able to do.
- Test: Write-protect the sim_data, then try calling the serial steps.
- Test: Pickle+unpickle the sim_data (to make a fully deep copy) to send to each step so it can't modify the caller's data. See if that changes the result.

Yes, diff trees would be useful, esp. if computed at intermediate stages.

Yes, diff trees would be useful, esp. if computed at intermediate stages.

Okay, I am diffing sim_data objects in https://github.com/CovertLab/wcEcoli/pull/395 and I compared a serial fitter output to the parallel fitter output and got no differences. Like I mention in the PR that may be because I am not comparing Unums correctly, I am eliminating that possibility now.

I believe it is this library? https://github.com/trzemecki/Unum He does not describe comparison, but using python == fails with these values so I am currently using np.array_equal until I find something better.

Ah found it, you can call number() on them to get the numpy array out. Okay, testing again....

Stupid question: are we sure that two runs of the serial fitter are identical? Likewise for the parallel fitter.

That's an excellent question!

I have verified that the fitter output (sim_data) for both serial and parallel is identical, at least as far as diff_trees is able to discern.

I have verified that the fitter output (sim_data) for both serial and parallel is identical, at least as far as diff_trees is able to discern.

Huh, that's interesting. I would have expected the parallel fitter to exhibit arbitrary behavior, given these issues. I suppose there is a good chance that the execution order might be the same between two parallel runs. Maybe it would change if we reordered the parallel execution calls (scramble the order, or just reverse it - either would almost guarantee a different execution order).

Of 3 serial and 3 parallel Fitter runs on Sherlock, one run produced different results while the other 5 produced equivalent results (as reported by the latest compareFitter.py).

Of 5 serial runs and 5 parallel runs on my MacBook, none of them seemed to produce equivalent results, although I mostly only compared adjacent runs.

Hypothesis: Are these differences at least partly due to a run-time environment sensitivity?

Q. @prismofeverything Are you still running with a BLAS installation overriding Apple's Accelerate framework? Try python runscripts/debug/summarize_environment.py

Hypothesis: Is something initializing random number generators differently?

[BTW we could upgrade NumPy 1.14.5 to 1.15.4. There are API changes, e.g. multidimensional indexing with a list or anything but a tuple is deprecated. It switched from nose to pytest. NumPy 1.17 will drop support for Python 2.7.]

Some Fitter run times

On a Sherlock compute node with 4 CPUs requested, a serial run takes about 27 minutes while a parallel run with 4 worker processes (-c4) takes about 3.6 m.
- How can it get a 7x speedup from 4 workers???
On 2018 MBP, a serial run takes about 16 m. Parallel runs with 2, 4, 8, or 12 workers all took about 3.6 m. That's a 4x speedup.

Some Fitter diffs

Sherlock serial run s1 compared to `s2:

{'process': {'equilibrium': {'derivativesJacobianSymbolic': (Matrix([
[-1.0*kr[0],  2.0*kf[0]*y[1]**1.0*y[2]**1.0,  1.0*kf[0]*y[1]**2.0,          0,                    0,                    0,          0,                    0,                    0,          0,                     0,                                                                                         0,          0,                     0,                     0,          0,                 ...,
                                                         --),
                         'derivativesSymbolic': (Matrix([
[                                                                                                                     1.0*kf[0]*y[1]**2.0*y[2]**1.0 - 1.0*kr[0]*y[0]**1.0],
[                                                                                                                    -1.0*kf[0]*y[1]**2.0*y[2]**1.0 + 1.0*kr[0]*y[0]**1.0],
[                                              ...,
                                                 --)},
         'two_component_system': {'derivativesFitterJacobianSymbolic': (Matrix([
[-100000000.0*y[6]**1.0, 0, 0,  500.0*y[4]**1.0,      500.0*y[3]**1.0,               0, -100000000.0*y[0]**1.0, 0,                0,                       0,                   0,               0,                                               0,                       0,                 0,                       0,                   0,               0,                                        ...,
                                                                        --),
                                  'derivativesFitterSymbolic': (Matrix([
[                                      -100000000.0*y[0]**1.0*y[6]**1.0 + 500.0*y[3]**1.0*y[4]**1.0],
[                                                                                                 0],
[                                                                                                 0],
[                                       100000000.0*y[0]**1.0*y[6]**1.0 - 500.0*y[3]*...,
                                                                --),
                                  'derivativesJacobianSymbolic': (Matrix([
[-100000000.0*y[6]**1.0, 0, 0,  500.0*y[4]**1.0,                                                                                                                                                                                                                500.0*y[3]**1.0,               0, -100000000.0*y[0]**1.0, 0,                                                                           ...,
                                                                  --),
                                  'derivativesSymbolic': (Matrix([
[                                                                                                                                                                                                                                                                                -100000000.0*y[0]**1.0*y[6]**1.0 + 500.0*y[3]**1.0*y[4]**1.0],
[ 170000.0*y[10]**1.0*y[4]**1.0 + 0.0001*y[14]**1.0*y[4...,
                                                          --)}}}

Sherlock serial run s2 compared to s3, s3 to parallel run p1, p1 to p2, or p2 to p3:
```
{}
```

Comparing sherlock-s2 to MBP s1; or MBP s1 to s2, s2 to s3, s3 to s4, s4 to s5, s5 to p1, p1 to p2, p2 to p3, p3 to p4, p4 to p5, they all look about like this, ± 'constants' ± 'expectedDryMassIncreaseDict' ± 'two_component_system' derivatives:

{'constants': {'darkATP': (43.579121091227336, 43.57912109122734)},
'expectedDryMassIncreaseDict': {'minimal_plus_sam': (375.7426732524626, 375.74267325246257)},
'pPromoterBound': {'CPLX-125__active': {'CPLX-125': (0.8879493735067311, 0.8879493735067314)},
                'CPLX-125__inactive': {'CPLX-125': (0.00559403326464808, 0.0055940332646480136)},
                'CPLX0-228__active': {'CPLX0-228': (0.9846137922755411, 0.9846137922755414), 'PC00027': (0.9999999999999999, 1.0)},
                'CPLX0-228__inactive': {'CPLX0-228': (0.4999739791454931, 0.49997397914549313)},
                'CPLX0-7740__active': {'CPLX0-7740': (0.6556887843199597, 0.6556887843199593)},
                'CPLX0-7740__inactive': {'CPLX0-7740': (0.001697125220452854, 0.001697125220452855)},
                'CPLX0-7796__active': {'CPLX0-7796': (0.46688416785817616, 0.46688416785817605)},
                'MONOMER0-155__active': {'MONOMER0-155': (0.8963502216484903, 0.8963502216484904)},
                'MONOMER0-155__inactive': {'MONOMER0-155': (0.5026152919010923, 0.5026152919010922)},
                'MONOMER0-162__active': {'MONOMER0-162': (0.9720860470840941, 0.9720860470840944)},
                'MONOMER0-162__inactive': {'MONOMER0-162': (0.6304391620366636, 0.6304391620366637)},
                'PC00027__active': {'CPLX0-228': (0.4999739791454931, 0.4999739791454932)},
                'PC00027__inactive': {'CPLX0-228': (0.4999739791454931, 0.49997397914549313)},
                'PD00288__active': {'PHOSPHO-ARCA': (0.0015876600692842188, 0.0015876600692842175)},
                'PD00288__inactive': {'PHOSPHO-ARCA': (0.00474137931034483, 0.004741379310344829)}},
'process': {'equilibrium': {'ratesRev': Arrays are not equal (mismatch 17.8571428571%) x: array([2.560000e-08, 2.000000e-07, 4.000000e-04, 3.430000e-20, 1.000000e-06, 3.250000e-07, 1.000000e-06, 1.000000e-06, 4.951815e-06, 1.000000e-06, 1.0...},
         'metabolism': {'concentrationUpdates': {'moleculeSetAmounts': {'CYTIDINE[c]': (0.002912814492167465, 0.002912814492167457),
                                                                        'LEU[c]': (0.0016430960569510228, 0.0016430960569510256),
                                                                        'S-ADENOSYLMETHIONINE[c]': (0.013234537880789276, 0.013234537880789274),
                                                                        'TRP[c]': (0.0004504088763323458, 0.0004504088763323492),
                                                                        'TYR[c]': (0.0005920144255532174, 0.0005920144255532243)}}},
         'rna_decay': {'KmConvergence': Arrays are not equal (mismatch 30.4738920579%) x: array([1. , 1. , 1. , ..., 0.999953, 0.999946, 0.999946]) y: array([1. , 1. , 1. , ..., 0.999953, 0.999946, 0.999946]),
                       'KmFirstOrderDecay': Arrays are not equal (mismatch 81.8999561211%) x: array([0.000226, 0.000226, 0.000226, ..., 0.113296, 0.113295, 0.113295]) y: array([0.000226, 0.000226, 0.000226, ..., 0.113296, 0.113295, 0.113295]),
                       'StatsFit': {'LossKm': (59.43121115880196, 59.43121115880152),
                                    'LossKmOpt': (2.618122949327173e-06, 2.6181211968401286e-06),
                                    'ResEndoRNKm': (0.44807840150863676, 0.4480784015086372),
                                    'ResEndoRNKmOpt': (1.6513797440609324e-08, 1.6513802547635237e-08),
                                    'ResKmOpt': (8.363088069618385e-06, 8.363090562513165e-06),
                                    'ResScaledKm': (1.1757756332375394e-09, 1.1757756332375407e-09),
                                    'ResScaledKmOpt': (4.3485446939959514e-17, 4.3485460032431424e-17)}},
         'transcription': {'rnaData': (STRUCTURED ARRAY:
array([('6S-RNA[c]', 2.01017105e-03, 183, [41, 51, 50, 41], 58865.634, False,  True, False, False, False, False, False, False, False, 'AUUUCUCUGAGAUGUUCGCAAGCGGGCCAGUCCCCUGAGCCGAUAUU...,
                                       STRUCTURED ARRAY:
array([('6S-RNA[c]', 2.01017105e-03, 183, [41, 51, 50, 41], 58865.634, False,  True, False, False, False, False, False, False, False, 'AUUUCUCUGAGAUGUUCGCAAGCGGGCCAGUCCCCUGAGCCGAUAUU...),
                           'rnaExpression': {'CPLX-125__active': Arrays are not equal (mismatch 89.8639754278%) x: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]) y: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]),
                                             'CPLX-125__inactive': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX-172__active': Arrays are not equal (mismatch 85.3444493199%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX-172__inactive': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-228__active': Arrays are not equal (mismatch 91.2242211496%) x: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]) y: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]),
                                             'CPLX0-228__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7671__active': Arrays are not equal (mismatch 88.8328214129%) x: array([0. , 0. , 0. , ..., 0.00734 , 0.008408, 0.008408]) y: array([0. , 0. , 0. , ..., 0.00734 , 0.008408, 0.008408]),
                                             'CPLX0-7671__inactive': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7705__active': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7705__inactive': Arrays are not equal (mismatch 90.1491882405%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008419, 0.008419]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008419, 0.008419]),
                                             'CPLX0-7740__active': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7740__inactive': Arrays are not equal (mismatch 89.8200965336%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7796__active': Arrays are not equal (mismatch 79.5963141729%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7796__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7916__active': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'CPLX0-7916__inactive': Arrays are not equal (mismatch 82.8652917946%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'FNR-4FE-4S-CPLX__active': Arrays are not equal (mismatch 92.2553751645%) x: array([0. , 0. , 0. , ..., 0.007348, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007348, 0.008418, 0.008418]),
                                             'FNR-4FE-4S-CPLX__inactive': Arrays are not equal (mismatch 88.9205792014%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'MONOMER0-155__active': Arrays are not equal (mismatch 91.7068889864%) x: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]) y: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]),
                                             'MONOMER0-155__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'MONOMER0-160__active': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'MONOMER0-160__inactive': Arrays are not equal (mismatch 86.8363317244%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'MONOMER0-162__active': Arrays are not equal (mismatch 88.4159719175%) x: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]) y: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]),
                                             'MONOMER0-162__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PC00010__active': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PC00010__inactive': Arrays are not equal (mismatch 83.106625713%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PC00027__active': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PC00027__inactive': Arrays are not equal (mismatch 86.0465116279%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008419, 0.008419]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008419, 0.008419]),
                                             'PD00288__active': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PD00288__inactive': Arrays are not equal (mismatch 85.4322071084%) x: array([0. , 0. , 0. , ..., 0.007348, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007348, 0.008418, 0.008418]),
                                             'PD00519__active': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PD00519__inactive': Arrays are not equal (mismatch 91.9043440105%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-ARCA__active': Arrays are not equal (mismatch 87.5603334796%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-ARCA__inactive': Arrays are not equal (mismatch 91.6630100921%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008419, 0.008419]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008419, 0.008419]),
                                             'PHOSPHO-BAER__active': Arrays are not equal (mismatch 88.1307591049%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-BAER__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-BASR__active': Arrays are not equal (mismatch 85.717419921%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-BASR__inactive': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-DCUR__active': Arrays are not equal (mismatch 90.3905221588%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-DCUR__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-NARL__active': Arrays are not equal (mismatch 84.4010530935%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PHOSPHO-NARL__inactive': Arrays are not equal (mismatch 86.7924528302%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PUTA-CPLX__active': Arrays are not equal (mismatch 87.0557261957%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'PUTA-CPLX__inactive': Arrays are not equal (mismatch 88.8328214129%) x: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]) y: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]),
                                             'basal': Arrays are not equal (mismatch 81.3953488372%) x: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]) y: array([0. , 0. , 0. , ..., 0.007349, 0.008418, 0.008418]),
                                             'no_oxygen': Arrays are not equal (mismatch 90.5221588416%) x: array([0. , 0. , 0. , ..., 0.007347, 0.008417, 0.008417]) y: array([0. , 0. , 0. , ..., 0.007347, 0.008417, 0.008417]),
                                             'with_aa': Arrays are not equal (mismatch 87.9333040807%) x: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407]) y: array([0. , 0. , 0. , ..., 0.007339, 0.008407, 0.008407])},
                           'rnaSynthProb': {'CPLX-125__active': Arrays are not equal (mismatch 89.381307591%) x: array([0. , 0. , 0. , ..., 0.006508, 0.007456, 0.007456]) y: array([0. , 0. , 0. , ..., 0.006508, 0.007456, 0.007456]),
                                            'CPLX-125__inactive': Arrays are not equal (mismatch 89.8200965336%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX-172__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX-172__inactive': Arrays are not equal (mismatch 89.8639754278%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX0-228__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.006506, 0.007453, 0.007453]) y: array([0. , 0. , 0. , ..., 0.006506, 0.007453, 0.007453]),
                                            'CPLX0-228__inactive': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX0-7671__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.006517, 0.007466, 0.007466]) y: array([0. , 0. , 0. , ..., 0.006516, 0.007464, 0.007464]),
                                            'CPLX0-7671__inactive': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX0-7705__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005906, 0.006765, 0.006765]) y: array([0. , 0. , 0. , ..., 0.005906, 0.006765, 0.006765]),
                                            'CPLX0-7705__inactive': Arrays are not equal (mismatch 96.0070206231%) x: array([0. , 0. , 0. , ..., 0.005908, 0.006768, 0.006768]) y: array([0. , 0. , 0. , ..., 0.005908, 0.006768, 0.006768]),
                                            'CPLX0-7740__active': Arrays are not equal (mismatch 89.8420359807%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX0-7740__inactive': Arrays are not equal (mismatch 86.1781483107%) x: array([0. , 0. , 0. , ..., 0.005909, 0.006769, 0.006769]) y: array([0. , 0. , 0. , ..., 0.005909, 0.006769, 0.006769]),
                                            'CPLX0-7796__active': Arrays are not equal (mismatch 89.0741553313%) x: array([0. , 0. , 0. , ..., 0.005905, 0.006765, 0.006765]) y: array([0. , 0. , 0. , ..., 0.005905, 0.006765, 0.006765]),
                                            'CPLX0-7796__inactive': Arrays are not equal (mismatch 88.6573058359%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX0-7916__active': Arrays are not equal (mismatch 89.8420359807%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'CPLX0-7916__inactive': Arrays are not equal (mismatch 96.0070206231%) x: array([0. , 0. , 0. , ..., 0.005906, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005906, 0.006766, 0.006766]),
                                            'FNR-4FE-4S-CPLX__active': Arrays are not equal (mismatch 88.4817902589%) x: array([0. , 0. , 0. , ..., 0.00436 , 0.004995, 0.004995]) y: array([0. , 0. , 0. , ..., 0.00436 , 0.004995, 0.004995]),
                                            'FNR-4FE-4S-CPLX__inactive': Arrays are not equal (mismatch 90.8293111014%) x: array([0. , 0. , 0. , ..., 0.005904, 0.006763, 0.006763]) y: array([0. , 0. , 0. , ..., 0.005904, 0.006763, 0.006763]),
                                            'MONOMER0-155__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.006503, 0.00745 , 0.00745 ]) y: array([0. , 0. , 0. , ..., 0.006499, 0.007444, 0.007444]),
                                            'MONOMER0-155__inactive': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005909, 0.006769, 0.006769]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'MONOMER0-160__active': Arrays are not equal (mismatch 89.8420359807%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'MONOMER0-160__inactive': Arrays are not equal (mismatch 88.7011847301%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]),
                                            'MONOMER0-162__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.006509, 0.007457, 0.007457]) y: array([0. , 0. , 0. , ..., 0.006509, 0.007457, 0.007457]),
                                            'MONOMER0-162__inactive': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PC00010__active': Arrays are not equal (mismatch 89.8420359807%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PC00010__inactive': Arrays are not equal (mismatch 91.7946467749%) x: array([0. , 0. , 0. , ..., 0.005911, 0.006772, 0.006772]) y: array([0. , 0. , 0. , ..., 0.005911, 0.006772, 0.006772]),
                                            'PC00027__active': Arrays are not equal (mismatch 96.0289600702%) x: array([0. , 0. , 0. , ..., 0.005906, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005906, 0.006766, 0.006766]),
                                            'PC00027__inactive': Arrays are not equal (mismatch 96.0070206231%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PD00288__active': Arrays are not equal (mismatch 88.0868802106%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PD00288__inactive': Arrays are not equal (mismatch 90.1711276876%) x: array([0. , 0. , 0. , ..., 0.005919, 0.00678 , 0.00678 ]) y: array([0. , 0. , 0. , ..., 0.005919, 0.00678 , 0.00678 ]),
                                            'PD00519__active': Arrays are not equal (mismatch 88.5914874945%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PD00519__inactive': Arrays are not equal (mismatch 96.0070206231%) x: array([0. , 0. , 0. , ..., 0.005906, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005906, 0.006766, 0.006766]),
                                            'PHOSPHO-ARCA__active': Arrays are not equal (mismatch 96.0070206231%) x: array([0. , 0. , 0. , ..., 0.004384, 0.005022, 0.005022]) y: array([0. , 0. , 0. , ..., 0.004384, 0.005022, 0.005022]),
                                            'PHOSPHO-ARCA__inactive': Arrays are not equal (mismatch 96.0070206231%) x: array([0. , 0. , 0. , ..., 0.005946, 0.006812, 0.006812]) y: array([0. , 0. , 0. , ..., 0.005946, 0.006812, 0.006812]),
                                            'PHOSPHO-BAER__active': Arrays are not equal (mismatch 88.3940324704%) x: array([0. , 0. , 0. , ..., 0.005908, 0.006768, 0.006768]) y: array([0. , 0. , 0. , ..., 0.005908, 0.006768, 0.006768]),
                                            'PHOSPHO-BAER__inactive': Arrays are not equal (mismatch 88.6573058359%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PHOSPHO-BASR__active': Arrays are not equal (mismatch 86.243966652%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PHOSPHO-BASR__inactive': Arrays are not equal (mismatch 89.8420359807%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PHOSPHO-DCUR__active': Arrays are not equal (mismatch 83.5234752084%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]),
                                            'PHOSPHO-DCUR__inactive': Arrays are not equal (mismatch 88.6573058359%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PHOSPHO-NARL__active': Arrays are not equal (mismatch 89.1619131198%) x: array([0. , 0. , 0. , ..., 0.005903, 0.006762, 0.006762]) y: array([0. , 0. , 0. , ..., 0.005903, 0.006762, 0.006762]),
                                            'PHOSPHO-NARL__inactive': Arrays are not equal (mismatch 88.6353663888%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006767, 0.006767]),
                                            'PUTA-CPLX__active': Arrays are not equal (mismatch 88.5256691531%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'PUTA-CPLX__inactive': Arrays are not equal (mismatch 85.2786309785%) x: array([0. , 0. , 0. , ..., 0.006511, 0.007459, 0.007459]) y: array([0. , 0. , 0. , ..., 0.006511, 0.007459, 0.007459]),
                                            'basal': Arrays are not equal (mismatch 83.1505046073%) x: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]) y: array([0. , 0. , 0. , ..., 0.005907, 0.006766, 0.006766]),
                                            'no_oxygen': Arrays are not equal (mismatch 88.5914874945%) x: array([0. , 0. , 0. , ..., 0.004327, 0.004957, 0.004957]) y: array([0. , 0. , 0. , ..., 0.004327, 0.004957, 0.004957]),
                                            'with_aa': Arrays are not equal (mismatch 92.9574374726%) x: array([0. , 0. , 0. , ..., 0.006502, 0.007449, 0.007449]) y: array([0. , 0. , 0. , ..., 0.006502, 0.007449, 0.007449])},
                           'rnaSynthProbFraction': {'minimal': {'mRna': (0.215102022139732, 0.21510202213973215),
                                                                'rRna': (0.1662521599248865, 0.16625215992488654)},
                                                    'minimal_minus_oxygen': {'mRna': (0.4250232302408263, 0.42502323024082644),
                                                                             'rRna': (0.12178796808686032, 0.12178796808686024),
                                                                             'tRna': (0.45318880167231335, 0.45318880167231346)},
                                                    'minimal_plus_amino_acids': {'mRna': (0.1351401216147142, 0.13514012161471425),
                                                                                 'rRna': (0.18318918747364846, 0.18318918747364837),
                                                                                 'tRna': (0.6816706909116377, 0.6816706909116375)},
                                                    'minimal_plus_arabinose': {'mRna': (0.21509068484832672, 0.21509068363369815),
                                                                               'rRna': (0.16625456131874533, 0.16625456157602023),
                                                                               'tRna': (0.618654753832928, 0.6186547547902814)},
                                                    'minimal_plus_cytidine': {'tRna': (0.6189324701049762, 0.6189324701049764)},
                                                    'minimal_plus_ferric': {'mRna': (0.21509300987369143, 0.21509300987369148),
                                                                            'rRna': (0.16625406884647548, 0.16625406884647545)},
                                                    'minimal_plus_indole': {'mRna': (0.2149555483165392, 0.21495554831653935),
                                                                            'rRna': (0.1662831850901503, 0.16628318509015025)},
                                                    'minimal_plus_nitrate': {'mRna': (0.21554584376818242, 0.21554584376818253),
                                                                             'rRna': (0.1661581524150794, 0.16615815241507936)},
                                                    'minimal_plus_sam': {'mRna': (0.21530861604309437, 0.21530861604309454),
                                                                         'rRna': (0.16620840050184033, 0.16620840050184027),
                                                                         'tRna': (0.6184829834550649, 0.6184829834550651)},
                                                    'minimal_succinate': {'mRna': (0.21505939065871796, 0.215059390658718),
                                                                          'rRna': (0.16626118985743743, 0.16626118985743737)}},
                           'rnaSynthProbRProtein': {'minimal': Arrays are not equal (mismatch 92.7272727273%) x: array([0.001149, 0.000799, 0.001202, 0.001047, 0.002444, 0.001968, 0.000596, 0.001872, 0.001298, 0.001337, 0.001035, 0.001455, 0.001125, 0.000876, 0.0...,
                                                    'minimal_minus_oxygen': Arrays are not equal (mismatch 92.7272727273%) x: array([0.002367, 0.001557, 0.00242 , 0.002086, 0.005054, 0.004058, 0.001193, 0.003835, 0.002692, 0.002674, 0.002104, 0.00291 , 0.002231, 0.001751, 0.0...,
                                                    'minimal_plus_amino_acids': Arrays are not equal (mismatch 94.5454545455%) x: array([0.000945, 0.000707, 0.00102 , 0.0009 , 0.001998, 0.001615, 0.000509, 0.001551, 0.001057, 0.001145, 0.000868, 0.001246, 0.000974, 0.00075 , 0.00...,
                                                    'minimal_plus_arabinose': Arrays are not equal (mismatch 100.0%) x: array([0.001149, 0.000799, 0.001202, 0.001047, 0.002444, 0.001968, 0.000596, 0.001872, 0.001298, 0.001337, 0.001035, 0.001455, 0.001125, 0.000876, 0.000884, 0...,
                                                    'minimal_plus_cytidine': Arrays are not equal (mismatch 90.9090909091%) x: array([0.001148, 0.000799, 0.001201, 0.001046, 0.002442, 0.001966, 0.000595, 0.001871, 0.001297, 0.001336, 0.001035, 0.001454, 0.001125, 0.000875, 0.0...,
                                                    'minimal_plus_ferric': Arrays are not equal (mismatch 90.9090909091%) x: array([0.001149, 0.000799, 0.001202, 0.001047, 0.002444, 0.001968, 0.000596, 0.001872, 0.001298, 0.001337, 0.001035, 0.001455, 0.001125, 0.000876, 0.0...,
                                                    'minimal_plus_indole': Arrays are not equal (mismatch 100.0%) x: array([0.001148, 0.000799, 0.001201, 0.001046, 0.002442, 0.001966, 0.000595, 0.001871, 0.001297, 0.001336, 0.001035, 0.001454, 0.001125, 0.000875, 0.000883, 0...,
                                                    'minimal_plus_nitrate': Arrays are not equal (mismatch 92.7272727273%) x: array([0.001152, 0.000801, 0.001205, 0.001049, 0.00245 , 0.001972, 0.000597, 0.001877, 0.001301, 0.00134 , 0.001038, 0.001459, 0.001128, 0.000878, 0.0...,
                                                    'minimal_plus_sam': Arrays are not equal (mismatch 94.5454545455%) x: array([0.001151, 0.000801, 0.001204, 0.001049, 0.002448, 0.001971, 0.000597, 0.001876, 0.0013 , 0.00134 , 0.001037, 0.001458, 0.001127, 0.000877, 0.00...,
                                                    'minimal_succinate': Arrays are not equal (mismatch 81.8181818182%) x: array([0.001149, 0.000799, 0.001202, 0.001047, 0.002443, 0.001967, 0.000596, 0.001872, 0.001298, 0.001337, 0.001035, 0.001455, 0.001125, 0.000876, 0.0...},
                           'rnaSynthProbRnaPolymerase': {'minimal': Arrays are not equal (mismatch 100.0%) x: array([0.015282, 0.002854, 0.002924]) y: array([0.015282, 0.002854, 0.002924]),
                                                         'minimal_minus_oxygen': Arrays are not equal (mismatch 100.0%) x: array([0.022691, 0.004237, 0.004342]) y: array([0.022691, 0.004237, 0.004342]),
                                                         'minimal_plus_amino_acids': Arrays are not equal (mismatch 100.0%) x: array([0.01107 , 0.002067, 0.002118]) y: array([0.01107 , 0.002067, 0.002118]),
                                                         'minimal_plus_arabinose': Arrays are not equal (mismatch 100.0%) x: array([0.015281, 0.002854, 0.002924]) y: array([0.015281, 0.002854, 0.002924]),
                                                         'minimal_plus_cytidine': Arrays are not equal (mismatch 100.0%) x: array([0.015253, 0.002848, 0.002919]) y: array([0.015253, 0.002848, 0.002919]),
                                                         'minimal_plus_ferric': Arrays are not equal (mismatch 100.0%) x: array([0.015281, 0.002854, 0.002924]) y: array([0.015281, 0.002854, 0.002924]),
                                                         'minimal_plus_indole': Arrays are not equal (mismatch 100.0%) x: array([0.015264, 0.002851, 0.002921]) y: array([0.015264, 0.002851, 0.002921]),
                                                         'minimal_plus_nitrate': Arrays are not equal (mismatch 100.0%) x: array([0.015304, 0.002858, 0.002929]) y: array([0.015304, 0.002858, 0.002929]),
                                                         'minimal_plus_sam': Arrays are not equal (mismatch 100.0%) x: array([0.0153 , 0.002857, 0.002928]) y: array([0.0153 , 0.002857, 0.002928]),
                                                         'minimal_succinate': Arrays are not equal (mismatch 100.0%) x: array([0.015278, 0.002853, 0.002924]) y: array([0.015278, 0.002853, 0.002924])}},
         'transcription_regulation': {'recruitmentData': {'hV': Arrays are not equal (mismatch 83.2866293034%) x: array([0. , 0. , 0. , ..., 0.004152, 0.003853, 0.003853]) y: array([0. , 0. , 0. , ..., 0.004152, 0.003853, 0.003853])}}}}

That is very strange that some are consistent but others aren't. Those execution times in parallel seem way too short. I've never seen anything less than ~8 min or so. Did you check the actual timestamps or just the script total time that gets printed at the end? I've seen some weird stats on that at times.

Hypothesis: Is something initializing random number generators differently?

Along those lines, are we sure everything is using the proper seed? If there is one thing that isn't using our seeded object then it could throw everything off downstream. Might be less likely if there were that many that were the same but if it's one number that is stochastically rounded it could be likely.

It's good to see that the error is (always?) in the last few bits of the mantissa, but troubling that there is any error at all. The MBP-specific issues make me think this is a library thing - maybe a SciPy dependency? I can't make any sense of that Sherlock diff.

@tahorst Good point on the seeding. I had thought there was more random number generation, but it looks like the only RNG is in the calls to the Cythonized wholecell.utils.mc_complexation.mccFormComplexesWithPrebuiltMatrices, which ought to be deterministic (and hopefully platform-independent) given @1fish2's rewrite to use a numpy.random.RandomState object in the Cython code. That said, maybe the Cython code would be a good thing to check - do we have a determinism test for that code?

Hey @1fish2, here is the result of me running python runscripts/debug/summarize_environment.py on the macbook:

multiprocessing 0.70a1
multiprocessing.cpu_count(): 8

numpy 1.14.5
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blis_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE

os 
os.environ:
  'BOOST_NUMPY_LIB': --
  'HOME': '/Users/rspangler'
  'LIBRARY_PATH': --
  'PI_HOME': --
  'PYENV_ROOT': '/Users/rspangler/.pyenv'
  'PYTHONPATH': '/Users/rspangler/Code/wcEcoli'
  'SHERLOCK': --
os.getcwd(): /Users/rspangler/Code/wcEcoli
os.uname(): ('Darwin', 'omniomnibus', '17.7.0', 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64', 'x86_64')

scipy 1.0.1
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3']
    define_macros = [('NO_ATLAS_INFO', 3)]
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

sys 
sys.platform: darwin
sys.prefix: /Users/rspangler/.pyenv/versions/wcEcoli2
sys.version: 2.7.14 (default, Jul 17 2018, 09:59:22) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
sys.api_version: 1013
sys.version_info: sys.version_info(major=2, minor=7, micro=14, releaselevel='final', serial=0)

So looks like I am still using the openblas build for numpy, but accelerate for scipy? Which could account for the differences we are seeing.

I have yet to create any differences caught by diff_trees on either ubuntu or mac now, but cmp does show a difference:

(wcEcoli2) rspangler@omniomnibus:~/Code/wcEcoli$ cmp out/serial/kb/simData_Fit_1.cPickle out/serial2/kb/simData_Fit_1.cPickle 
out/serial/kb/simData_Fit_1.cPickle out/serial2/kb/simData_Fit_1.cPickle differ: char 147089, line 287

This could be dictionary hashing order? Or something else? The contents do not differ at least.

I will run a few sims with these, but I am starting to suspect the Accelerate framework for numpy since that is the only tangible difference between our systems I can think of.

Those execution times in parallel seem way too short. I've never seen anything less than ~8 min or so. Did you check the actual timestamps or just the script total time that gets printed at the end? I've seen some weird stats on that at times.

Indeed, the printed timestamps and the iTerm2 timestamps are larger than those numbers. In the code, time.clock() measures the “processor time” ... “this is the function to use for benchmarking Python or timing algorithms” ... except when they use subprocesses. (LOL) I'm fixing that.

This is better:

$ python runscripts/manual/runFitter.py p8 -c8
Mon Dec  3 14:45:06 2018: RunFitter at /Users/jerry/dev/wcEcoli/out/p8
{'Arguments': {'cached': False,
               'cpus': 8,
               'debug': False,
               'ribosome_fitting': True,
               'rnapoly_fitting': True,
               'sim_outdir': 'p8',
               'sim_path': '/Users/jerry/dev/wcEcoli/out/p8',
               'verbose': False}}
...
Mon Dec  3 14:52:32 2018: Elapsed time 445.9 secs (0:07:25.926408); 300.9 secs (0:05:00.947440) in process

Hmm, I'll simplify it:

Mon Dec  3 15:24:13 2018: Elapsed time 700.81 secs (0:11:40.811176); 298.89 in process

-c8 took 445.9 secs; 300.9 secs in process with no cached km.cPickle. -c2 took 700.81 secs; 298.89 in process with no cached km.cPickle.

Key deltas (details below) in summarize_environment.py between my MBP and @prismofeverything's:

numpy is linked the Accelerate framework
macOS Mojave
Python 2.7.15 (as in requirements.txt)
LLVM 9.1.0

The entire computation ought to be deterministic, including dictionary hashing (unless the hash bucket sizes depend on available RAM) and floating point computation. The Sherlock diff suggests there might be more than one cause of variations.

Proposed next steps:

I'll switch numpy to openblas. Ryan, please send me the instructions you used. Do you have info on how to switch back?
Ryan, please update to Python 2.7.15 and retest: PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 2.7.15
Ryan, please update Xcode tools and retest: xcode-select --install
If you're up for switching to Mojave, be sure that FUSE is up to date beforehand and reinstall Xcode tools afterwards.

lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
openblas_clapack_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
blis_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE

os.uname(): ('Darwin', 'Jerrys-MacBook-Pro.local', '18.2.0', 'Darwin Kernel Version 18.2.0: Fri Oct  5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64', 'x86_64')

scipy 1.0.1
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3']
    define_macros = [('NO_ATLAS_INFO', 3)]
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_info:
  NOT AVAILABLE
...

sys
sys.version: 2.7.15 (default, Jul 26 2018, 17:13:52)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]

I meant to say pretty please when sugar on top. :-)

From the Python 2.7.15rc1 release notes:

Prevent unwanted behavior in _random.Random.seed() in case the argument has a bad abs() method. Patch by Oren Milman.

Preserve generator state when _random.Random.setstate() raises an exception. Patch by Bryan Olson.

Support glibc 2.24 on Linux: don't use getentropy() function but read from /dev/urandom to get random bytes, for example in os.urandom(). On Linux, getentropy() is implemented which getrandom() is blocking mode, whereas os.urandom() should not block.

... and 2 more references to "random".

Work around a gc.disable() race condition in the subprocess module that could leave garbage collection disabled when multiple threads are spawning subprocesses at once. Users are strongly encouraged to use the subprocess32 module from PyPI on Python 2.7 instead, it is much more reliable.

Issue #166: Use the subprocess32 pip instead of subprocess. But I think we only use this Python library for little command line calls like "git rev-parse HEAD".

Update zlib to 1.2.11.

I'll switch numpy to openblas. Ryan, please send me the instructions you used. Do you have info on how to switch back? Ryan, please update to Python 2.7.15 and retest: PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 2.7.15 Ryan, please update Xcode tools and retest: xcode-select --install

Currently reinstalling my pyenv for this to update python to 2.7.15, must have missed that update on this computer. This does mean I will be back to the accelerate version of numpy and can test that next. To compile numpy with openblas I followed this: https://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration

If you're up for switching to Mojave, be sure that FUSE is up to date beforehand and reinstall Xcode tools afterwards.

I don't think I'm ready for Mojave yet.... this is an old system and currently stable, I'm going to leave that alone. Let's see how the tests work out here.

Cool.

Whoa, that's a lot of steps to build OpenBLAS and then compile numpy with it! Many ways to go wrong. It'd help if there are pip install options, but I'm not finding any. (The Intel Python Distribution is sure to have all the subtle linkages and dependencies right...)

Some of these treediffs show individual numbers differing in the low order digits. This is different between runs, not between machines, which is puzzling. The numpy array diffs don't show the actual differences, which could be in the elided values or in the rounded digits. compareFitter.py might need adjustable verbosity.

Occasionally, some Matrix derivatives like 'process/equilibrium/derivativesJacobianSymbolic' are present in some runs and absent in others such as my Sherlock runs s1 vs. s2 (group readable in $SCRATCH). Can sporadic non-zero values determine whether to include a Matrix or not?

Yep, getting diffs now. Now the question is: was it going back to accelerate? Or upgrading to python 2.7.15? I will do those steps to compile numpy against openblas and rule out the last possibility.

Nice!

Good news:

It's no longer necessary to create an Intel dev account, download the whole Intel Distribution for Python, rationalize its packages with ours, nor use conda to install their packages and figure out how to add our packages and how conda coexists with pip.
- Just create a new pyenv virtualenv (or pip uninstall numpy scipy scikit-learn -y), do pip install intel-numpy intel-scipy tbb4py, then pip install our other packages. I drafted intel-requirements.txt instructions to build this virtualenv.
- summarize_environment.py reports that numpy and scipy are linked to lapack_opt_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/localdisk/tools/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/localdisk/tools/intel//compilers_and_libraries_2016.3.170/mac/mkl', '/localdisk/tools/intel//compilers_and_libraries_2016.3.170/mac/mkl/include', '/localdisk/tools/intel//compilers_and_libraries_2016.3.170/mac/mkl/lib', '/opt/anaconda1anaconda2anaconda3/include'] etc.
- Switching between these virtualenvs works fine. Maybe run make clean compile after switching.
- One can then run python -m tbb [-h] [-p P] [-b] [-v] [-m] script ... to coordinate CPU usage across thread and processes to avoid slowdown due to oversubscription contention for cores.
So far, parallel runs pi10 vs. pi11 differed only in the derivative matrices, pi10 vs. pi12 and pi11 vs. pi12 were typical long diffs; but pi12, si6, and si7 all match (despite different file sizes and contents)!

Not great news:

It doesn't seem to run faster, but it is easier to install than compiling numpy on BLAS.

So the output varies between runs, even with the serial Fitter.

I'm changing compareFitter.py to allow some floating point tolerance, but the NumPy functions which support that don't handle inf and NaN values, nor structured dtypes. There are lots of inf values in these arrays.

I'm seeing the problem frequently with macOS Mojave on Python 2.7.15 with Accelerate framework, Python 2.7.14 with Accelerate framework, Python 2.7.15 with Intel NumPy, SciPy, numba, numexpr, Scikit-learn, and TBB on MKL, and less often on Sherlock with Python 2.7.15 on BLAS.

Investigating this issue, there are some sources of non-determinism:

Concurrent computations are non-deterministic in the order that floating-point operations are performed, which can result in non-bit-exact results across runs.
numpy.empty() returns a new array without initializing entries. If we fail to set all those entries, the results could be unpredictable.
- In my measurements, empty() takes about as long as zeros(), so don't use it! --> Something's wrong in my timing experiments...
- ~~TODO: I'll replace all calls on np.empty() with np.zeros().~~
I don't know if the parallelism within NumPy, BLAS, etc. are careful to avoid non-deterministic results.
- TODO: Try some experiments.
- I asked on SO: https://stackoverflow.com/q/53626778/1682419. This already yielded links to NumPy non-determinism bugs.

Okay, just tried with openblas again and python 2.7.15 this time. No diffs. I'll run a couple more but it is starting to look pretty convincing that accelerate is at least one component of the problem.

Not sure what to say about sherlock except that it has been historically pretty unreliable and I have observed some highly suspicious behavior (a file descriptor being corrupted from one login node, but fine from another when accessing the same file, among other things. I have a whole spreadsheet of the lab's collective sherlock problems). I wouldn't be surprised if it was a different source of variability on the sherlock system than the mac environment.

That's good to know.

Maybe Mojave or Core i9 is another component of the problem. MKL didn't fare much better than Accelerate. I was looking into sources of nondeterminism.

I should test openblas. Is there a sane way to install it? ... This NumPy Issue says openblas is in the numpy wheels distribution.

BTW something was wrong with my timing measurements. So np.empty() is faster than np.zeros() by about 6x.

Couple thoughts, probably not helpful:

Re: NumPy determinism and Sherlock; when running lots of optimization problems (hundreds of runs, about 15 minutes of heavy NumPy computation) I experienced almost determinism, as in most of the time a seeded run would produce the same results. I was running these things mostly on owners so after failing to find a reproducible case I just chalked it up to hardware differences and moved on with my life.
Re: np.empty; even if it is faster, avoiding it outside of non-intensive code is probably a best practice. Even better, perhaps, would be strictly initializing arrays from complete sets of values (i.e. a list) rather than building the array and then filling it in, although that's not always convenient.
I'm a little troubled by the fact that we have any non-finite numbers in sim_data. Those may be worth documenting/inspecting/redressing.

I should test openblas. Is there a sane way to install it?

I just followed the steps in the accepted answer in that SO post.... I had to do some acrobatics to get gfortran installed but otherwise it was straightforward enough. I've done it twice now if that's any testament : )

Also, just checked again on ubuntu (where numpy is built against openblas by default) and I get no diffs.

Measuring performance is tricky! Calling np.empty in a loop can discard the array on each iteration and then simply reallocate the same one or two memory nodes. That measurement is independent of the requested array size.

This test shows that numpy.empty() does return an uninitialized array and it can easily and quickly recycle a recent array:

np.arange(100); np.empty(100, int); np.empty(100, int)
np.arange(100, 200.0); np.empty(100, float); np.empty(100, float)

To prevent recycling (measured on MBP i9):

>>> from timeit import timeit

>>> timeit('l.append(numpy.empty(100000))', 'import numpy; l = []')
3.0151820182800293
>>> timeit('l.append(numpy.empty(10000))', 'import numpy; l = []')
1.1399481296539307
>>> timeit('l.append(numpy.empty(10000))', 'import numpy; l = []')
1.244724988937378

>>> timeit('l.append(numpy.zeros(100000))', 'import numpy; l = []')
3.100321054458618
>>> timeit('l.append(numpy.zeros(100000))', 'import numpy; l = []')
3.4001340866088867
>>> timeit('l.append(numpy.zeros(10000))', 'import numpy; l = []')
54.84381604194641  # WAT?
>>> timeit('l.append(numpy.zeros(1000))', 'import numpy; l = []')
5.2339441776275635  # HUH?

Initializing an ndrarry from a list can take 500x as long. Setting number=10000 instead of the default 1M:

>>> timeit('l.append(numpy.array(x))', 'import numpy; l = []; x = 100000 * [1.1]', number=10000)
19.110800981521606

(Trying that on Sherlock can easy get a job killed for lack of memory.)

Sherlock seems much slower at zeroing out an array. (I think that happens inside C calloc.)

>>> timeit('l.append(numpy.zeros(10000))', 'import numpy; l = []', number=1000)
0.06631803512573242
>>> timeit('l.append(numpy.zeros(10000))', 'import numpy; l = []', number=1000)
0.06618595123291016

>>> timeit('l.append(numpy.empty(10000))', 'import numpy; l = []', number=1000)
0.00645899772644043
>>> timeit('l.append(numpy.empty(10000))', 'import numpy; l = []', number=1000)
0.006451845169067383

>>> timeit('l.append(numpy.zeros(10001))', 'import numpy; l = []', number=1000)
0.06612896919250488
>>> timeit('l.append(numpy.zeros(10001))', 'import numpy; l = []', number=1000)
0.06485104560852051

>>> timeit('l.append(numpy.empty(10002))', 'import numpy; l = []', number=1000)
0.006468057632446289
>>> timeit('l.append(numpy.empty(10002))', 'import numpy; l = []', number=1000)
0.006554126739501953

>>> timeit('l.append(numpy.zeros(10002))', 'import numpy; l = []', number=1000)
0.06684994697570801
>>> timeit('l.append(numpy.zeros(10002))', 'import numpy; l = []', number=1000)
0.06575298309326172

Yay, NumPy + SciPy on OpenBLAS is producing serial and parallel Fitter runs with results that match each other and most Sherlock runs!

Summary:

OpenBLAS 0.3.4 fixes several threading bugs that cause nondeterministic results. (Reportedly, at least one of these only shows up when you have 3+ cores.)
The OpenBLAS head of the source tree as of Dec 3 fixes a macOS compatibility bug.
NumPy on this OpenBLAS fixes most of the discrepancies. SciPy on OpenBLAS fixes the derivatives matrices mismatches.
I'll document installation steps later. It was not easy to figure out.

Very interesting. I wonder if library linking issues also explain the discrepancies I saw on Sherlock. Wouldn't shock me - did I tell you two about the time I found different versions of git installed on different nodes?

I'm wondering same thing about discrepancies on Sherlock.

runscripts/debug/summarize_environment.py prints numpy sections like this on both Sherlock and my local BLAS pyenv:

lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']   # or ['/usr/local/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c

while the scipy sections look like this on Sherlock:

lapack_opt_info:
    libraries = ['openblas']
    library_dirs = ['/usr/local/lib']
    language = f77

vs. local:

lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c

I'll try reinstalling scipy in the Sherlock pyenv wcEcoli3 to compare.

(Different versions of git on different Sherlock nodes? Ugh. And they're old -- the stash commands are different.)

Instructions to fix the non-determinism on your local machine by installing openblas + numpy + scipy are now in the wiki.

In brief: Compile openblas from source [the 0.3.5 release will be easier], then create a ~/.numpy-site.cfg file, then pip install numpy==1.14.5 scipy==1.0.1 --no-binary numpy,scipy.

Sherlock's pyenv wcEcoli3 now has OpenBLAS compiled from current source (0.3.4+) with NumPy and SciPy compiled for that.

Several serial and parallel runs produced the same output, except occasionally the derivatives matrices are different -- but did match an older run. (See the first part of issuecomment-443575466 above for an example.) So scratch my hypothesis about the derivatives matrices.

Is the fixtures/endo_km/km.cPickle part of the computation where this difference comes in?

I could be wrong (@ggsun and @heejochoi may know better) but I think those derivatives are generated when instantiating sim_data, and are not modified by the fitter at all. But I've always had an extremely difficult time following this logic. Anyway, if I'm right, then the issue can be tested without running the fitter. Furthermore I'm inclined to believe that the errors crop up in sympy since this is almost the only place where that library is called.

Yay, NumPy + SciPy on OpenBLAS is producing serial and parallel Fitter runs with results that match each other and most Sherlock runs!

Yeah, nice! So, this is the second issue with the "accelerate" framework (the first being the segfaults with multiprocessing), why is apple trying to replace openblas? Its aversion to goodness and light? Openblas has been working for decades and widely considered unimprovable, which apple's demonstration supports so far. Maybe it's Fortran prejudice ; )

I know I will be compiling everything to use openblas from now on.

Thanks for all the investigation on this @1fish2, time to merge treediff yet?

OpenBLAS fixed the threading bugs that seemed to lead to non-determinism only in the last week or two.

Does that fix the fork segfaults?

Does it fix the parallel slowdown? Intel's tbb can fix the over-subscription problem.

Yes, it's time to merge in the treediff branch.

We could probably use this fix to close the parallel fitter issue but open a new one about the derivatives.

I think Apple pushed Intel and Moto for vector instructions. My guess is Accelerate was designed to use them when nothing else did.

Does that fix the fork segfaults?

Before, switching to openblas shifted the segfault to later in the operation. So probably it cleared up one of the segfaults and revealed a different one. That said, in that environment I had only compiled numpy against openblas, scipy was still using the accelerate framework, so perhaps that would fix the problem now that we know about the issues with accelerate.

AFAIK the main problem is fixed by making NumPy and SciPy use OpenBLAS 0.3.4+, so I'll close this Issue. See also

404 -- update the Sherlock pyenv wcEcoli2 to use OpenBLAS 0.3.4+ (already in wcEcoli3)
405 -- we're still getting differing derivative matrices from the Fitter
406 -- a regular check to catch differing output

CovertLab / wcEcoli