Model with subConnParams not replicable on diff num cores

[salvadord]

• Don and Salvador noticed diff M1 results on diff cores
• Salvador tried on GCP with 32, 64, 128 cores and found diff results; but if set subConnParams to False the results are identical
• Salvador tried with single PT cell on 2 vs 4 cores and found diff results, but identical if no subConnParms:

[salvadord]

tried without groupSunMechs but still different results

[salvadord]

tried 'density': 'uniform' and still got different results

[salvadord]

Fixed by sorting conns before distributing them. Testing on GCP now

[salvadord]

tested M1 model with 64 vs 128 cores and identical

[salvadord]

• Fixed by ordering conns; and by indexing connGroups with unambiguous connGroupLabel instead of index
• PTcell example shows identical results (8k inputs, connGroup, 3 vs 7 cores)
• M1 model 64 vs 128 not identical (prev comparison was bug because had subconn = false)
• Simplified M1 model (30x30um) showed identical conns

[salvadord]

• Is it possible the subconn isssue is fixed? unlikely since showed identical results with subconn=off
• track down different features of subconn: -- X groupedSyns -- X loc precision -- X uniform vs 2D dist -- multiple rules interacting

[salvadord]

Tried without groupedSyns and still got different (v54_batch16 vs 17) (though smaller difference)

[salvadord]

tried 'uniform' distribution for all rules, but still differen (128 vs 256 cores)

[salvadord]

can't be loc precision because not using groupedSyns!

[salvadord]

add v['loc'] to sort

[salvadord]

test subconn ordering of conns before uniform distribution

[salvadord]

comparing saved output json files (conns and spikes) showed no difference in conns (even though spikes times different)! properly compared? maybe not because nested? becasue ignored order? try comparing h.netcon info? would only show for cells in node 0

[salvadord]

• try oneSynPerConn
• make sure json comparison is right (identical conns) -- if so, compare h.netcons somehow (save from each node?)
• try without weightNorm?

[salvadord]

• oneSynPerConn - same difference
• trying to save h.netcon info to compare

[salvadord]

h.netcon is done per node, so doesn't have info of presyn gid if presyn cell in different node.
ah! actually can use netcon.srcgid() !

compared across all nodes by saving to json and merging files... identical h.list('netcon') info!!

[salvadord]

Comparing output of psection now across all nodes.

Other things to try:

• mike hines method to find out 1st spike difference
• order of connections
• something to do with vecstim play() ?
• rounding error?

[salvadord]

Summary:

• M1 model spike output is different for different number of cores (same code, only change num cores)
• Only happens if use subconnParams (rules to distribute synapses along sections, executed after all conns have been created; happens even with simplest rule, i.e. uniform distribution)
• Cannot reproduce in smaller networks (minimum >3k cells) or single PT cell with 10k inputs
• Output generated by netpyne (netParams, simConfig, net with all cells etc) is identical
• h.list('Netcon') is identical (calculated by saving output from each core to json, merging and comparing with DeepDiff)
• sec.psection() output for all sections is identical (calculated by saving output from each core to json, merging and comparing with DeepDiff)

[salvadord]

• Check what is 1st spike difference -- difference in time: 491.34 vs 491.29 ms (same cell id)
• Check Vecstims are identical -- yes
• Gradually remove subConnParams -- if keep subConnParams for 5-comp then see diffs (but not for >500 comp cells); suggests issue is when placing multiple syns on same section/segment

[salvadord]

Reduced to minimal model with 1 cell with 5 compartments and 100 netstims spiking at 15hz, all connected to the cell (no subconn rules; fixed delay of 5 ms). If netstims target soma: sims identical on 3 vs 4 cores if netstims target Adend1: voltage differs up to 0.1 mV after 1 sec on 3 vs 4 cores

[salvadord]

Next steps:

• reproduce in minimal netpyne model
• reproduce in NEURON

[salvadord]

manage to show reproducibility issue in a very simple model: a single 5-compartment cell with 100 NetStims as input; if run for 1 sec on 3 vs 4 cores get different voltages (up to 3mV difference!)

requires very specific tuning: e.g. syns have to be on ‘Adend1’ sec and temperature has to be 34ºC however, still not really sure where the discrepancy is coming from…

from netpyne import specs, sim
# Network parameters
netParams = specs.NetParams()  # object of class NetParams to store the network parameters
## Population parameters
netParams.popParams['IT5B'] = {'cellType': 'IT', 'numCells': 1, 'cellModel': 'HH'}
netParams.popParams['input'] = { 'numCells': 100, 'rate': 15, 'noise': 1.0, 'start':0, 'cellModel': 'NetStim'}
## Cell property rules
netParams.loadCellParamsRule(label='IT5B_reduced', fileName='IT5B_reduced_cellParams.pkl')
netParams.cellParams['IT5B_reduced']['conds'] = {'cellType': 'IT'}
## Synaptic mechanism parameters
netParams.synMechParams['AMPA'] = {'mod': 'Exp2Syn', 'tau1': 0.05, 'tau2': 5.3, 'e': 0}  # excitatory synaptic mechanism
## Cell connectivity rules
netParams.connParams['input->IT5B'] = {  #  S -> M label
    'preConds': {'pop': 'input'},   # conditions of presyn cells
    'postConds': {'pop': 'IT5B'}, # conditions of postsyn cells
    'probability': 1.0,             # probability of connection
    'weight': 0.5,              # synaptic weight
    'delay': 5,                     # transmission delay (ms)
    'synMech': ['AMPA'],
    'oneSynPerNetcon': True,
    'sec': ['Adend1']}              # synaptic mechanism
# Simulation options
simConfig = specs.SimConfig()       # object of class SimConfig to store simulation configuration
simConfig.duration = 1*1e3          # Duration of the simulation, in ms
simConfig.dt = 0.05         # Internal integration timestep to use
simConfig.verbose = False  # Show detailed messages
simConfig.seeds = {'conn': 4321, 'stim': 1234, 'loc': 4321}
simConfig.hParams = {'celsius': 34, 'v_init': -80}
simConfig.recordTraces = {'V_soma':{'sec':'soma','loc':0.5,'var':'v'}}  # Dict with traces to record
simConfig.recordStep = 0.1          # Step size in ms to save data (eg. V traces, LFP, etc)
simConfig.saveJson = True   
simConfig.filename = 'model3'  # Set file output name

simConfig.analysis['plotRaster'] = True             # Plot a raster
simConfig.analysis['plotTraces'] = {'include': [0]}             # Plot recorded traces for this list of cells
# Create network and run simulation
sim.createSimulateAnalyze(netParams = netParams, simConfig = simConfig)

[salvadord]

testing if dt = 0.025 and recordStep = 0.025 fixes issue in orig model