search

AllenInstitute / ipfx

computes intrinsic cell features from intracellular electrophysiology data
https://ipfx.readthedocs.io/en/latest/
Other
26 stars 36 forks source link

unequal number of datapoints in NWB conversion #473

Open ozsat opened 4 years ago

ozsat commented 4 years ago

I noticed that when I convert DAT to NWB (ipfx.bin.run_x_to_nwb_conversion.convert), the voltage samples and current samples are not equal. In the current samples, 5 of the given points are always missing from the beginning. It seems to be a major problem because the ipfx cant process those NWB files. I checked the raw data and it contains an equal number of points (I also attached a picture that shows the discrepancy in the NWB file).

voltage_samples_nwb stimulus_samples_nwb

When I run run_pipeline on the generated NWB data the following error message appears:

INFO:root: INFO:root: Extract QC features INFO:root:======================================================= INFO:root: INFO:root:Ontology is not provided, using default E:\anaconda3\lib\site-packages\ipfx\defaults\stimulus_ontology.json E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:28: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead blowout_sweep_number = data_set.get_sweep_number(ontology.blowout_names) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpinbath warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpbreakn warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Long square warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Rheobase warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Short square warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Capacitance warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Chirp test warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Chirp warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpexpend warnings.warn("Could not find stimulus: %s" % tag) E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:57: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead bath_sweep_number = data_set.get_sweep_number(ontology.bath_names) E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:99: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead seal_sweep_number = data_set.get_sweep_number(ontology.seal_names,"VoltageClamp") E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:151: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead breakin_sweep_number = data_set.get_sweep_number(ontology.breakin_names,"VoltageClamp") WARNING:root:Could not compute input/access resistance ratio (sr: None, ir:: None) WARNING:root:Blowout is not available WARNING:root:Electrode 0 is not available WARNING:root:Seal is not available WARNING:root:Breakin sweep not found Traceback (most recent call last): File "e:/ipfx/File_reader.py", line 74, in output_dir=Output_Full_Path, File "e:/ipfx/File_reader.py", line 60, in run_pipelinefrom_nwb_file write_spikes) File "E:\anaconda3\lib\site-packages\ipfx\bin\run_pipeline.py", line 27, in run_pipeline stimulus_ontology_file File "E:\anaconda3\lib\site-packages\ipfx\bin\run_sweep_extraction.py", line 69, in run_sweep_extraction sweep_features = sweep_qc_features(ds) File "E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py", line 278, in sweep_qc_features sweep = data_set.sweep(sweep_num) File "E:\anaconda3\lib\site-packages\ipfx\dataset\ephys_data_set.py", line 240, in sweep enforce_equal_length=True, File "E:\anaconda3\lib\site-packages\ipfx\dataset\ephys_data_set.py", line 431, in _voltage_current f"found {len(voltage)} voltage samples, " ValueError: found 302000 voltage samples, but 301995 current samples

The DAT file as an example: https://drive.google.com/file/d/1GgNunvFPqRpjz9A6DAj9jF7uFA_owmtR/view?usp=sharing The generated NWB: https://drive.google.com/file/d/1eHzVY_2F6fyWNtE0R1OyhkAtZ2a-JMWt/view?usp=sharing

Environment:

tmchartrand commented 4 years ago

@ozsat I'm guessing you closed this based on using the fix implemented on my fork? we should keep this issue open to track the work of merging that fix into the primary repository. If you closed it for some other reason, please let us know!

tmchartrand commented 1 year ago

This is fixed at the new x-to-nwb conversion repo (see above), but should maybe be left for reference until the faulty code is removed here (#492).