cross validation segmentation with various neural states

[chongxi]

cross-validation is important to test the performance of the various decoder construction. API from spiketag output spike count vector and the pc.ts pc.pos is required to segment the input into:

train_X, train_y valid_X, valid_y test_X, test_y

In such a simple way: (could be more complex with low_speed and/or low_mua states)

dec.partition(training_range=[0.0, .5],valid_range=[0.5, 0.6],testing_range=[0.6, 1.0])
(X_train, y_train), (X_valid, y_valid), (X_test, y_test) = dec.get_partitioned_data()

https://github.com/chongxi/spiketag/commit/1d24a165dc6f26d38f438f885b03ad16746d046a

Several factors need to be considered before and when dividing data:

• [x] Sometimes we only sort the stable section of the recording, in this case, data alignment is required. The alignment according to the replay_offset time and ephys_start, ephys_end time pc.align_with_recording(ephys_start, ephys_end, replay_offset) https://github.com/chongxi/spiketag/commit/dec7832becd1c6e775cbf6fa37d258ca119d058b

• [x] Make sure the low speed period is removed from the training data and the valid data ps.low_speed_idx automatically calculate during pc.initialize() and dec.get_partitioned_data() https://github.com/chongxi/spiketag/commit/b228b13e3f4ac73cb66c144f569f7070d958aeb7 https://github.com/chongxi/spiketag/commit/71211230377ffbc22a2c84aa296ee35e2ffa2e43 https://github.com/chongxi/spiketag/commit/68e488d4b9677c1ffe7dbfcaed318b5c20ba5519

• [x] Make sure the low speed period can be removed from the testing data (for decoding MUA bursts during ripples, which happen when the animal is not moving)

  low_speed_cutoff = {'training':True, 'testing':False}
  (X_train, y_train), (X_valid, y_valid), (X_test, y_test) = dec.get_partitioned_data(low_speed_cutoff, v_cutoff=25)

  https://github.com/chongxi/spiketag/commit/71211230377ffbc22a2c84aa296ee35e2ffa2e43 https://github.com/chongxi/spiketag/commit/68e488d4b9677c1ffe7dbfcaed318b5c20ba5519

• [ ] Make sure that during the MUA burst periods we use different decoding parameters (because it is essentially a different neural state)

• [ ] Make sure that during the low MUA periods we use different decoding method because the decoding will not be accurate with just a few spikes in total (e.g. <3 spikes from 80 neurons) per frame (20ms).

[chongxi]

More details:

• [x] One should be able to set the range for training, validation, and testing using percentage

training_range = [0.4, 0.9]
valid_range    = [0.9, 1.0]
testing_range  = [0.0, 0.5]

• [x] The parameter start and end can be calculated from the specified range.

• [x] A function is needed to output the corresponding segmented spike count vector (scv). Presumably the pc.get_scv(start, end)

• [x] A converted time range (in secs) is needed to compute the representation using start and end in the pc.get_fields(start, end)

Then the new Fr = pc.fields contains the representations from start to end (secs)

[chongxi]

check frame 79350, 81749 and 82493 in dusty data (an immediate prediction) 557.270 seconds 562397*1ms step (5ms code)