Open rly opened 1 year ago
This loss of precision is actually not a problem; the noise level of the recordings is several microvolts. Also, is it really useful to know the units of the data and not to have to convert. As such, I'd like for us to keep this as is...
@rly @lfrank
I support @rly's suggestion. It may be the case that the quantization noise from early conversion is not large, but there is no reason to add noise to our data. It also doesn't require more disk space since everything would remain int16
. The only downside seems to be that it is unintuitive, but right now we still have to multiply the data by the value in conversion
field of the ElectricalSeries
because the unit is volt
not microvolt
; if we're going to do this, then we might as well multiply by the actual ADC-to-microvolt conversion factor. This can be done automatically in spyglass so the user only sees data in microvolts. Finally, the conversion factor carries useful information: e.g. if it is 0.000000195 then we can infer that the data was acquired with Intan amplifier chips. This can be useful for understanding the data acquisition process later.
Currently, rec_to_nwb does the following:
rec_to_binaries
which converts the raw ephys voltage data from .rec to .mda format (dtype = int16; ADC units)"raw_data_to_volts"
key from the metadata YAML. according to a jan 2021 slack message from loren, this value should always be set to 0.000000195 (or 1.95e-7)0.195
). this matches the value stored in the .rec xml file headers (rawScalingToUv="0.19500000000000001"
)0.195
) and then sets the dtype to int16, which truncates any values after the decimal point (0.99 -> 0)1e-6
conversion factor, used to convert the data to voltsBecause of the data transformation in Step 4 above, there is a loss of precision. Let's say the original .rec file data has values:
then after multiplying by 0.195 to get the data in uV, the values are:
then after setting the dtype to int16, the values are:
Note the loss of precision in the resulting output. If the original data has unique values -100 to 100 (201 possible), then the converted NWB file will have unique values -19 to 19 (39 possible). This could have an impact on spike sorting and LFP filtering - probably a small impact, but still I think some impact?
For the above reason, it is more common to store the raw, untransformed int16 ephys data (ADC units) from an acquisition system as the ElectricalSeries data, and store the
conversion
factor (here: 0.000000195). However, NWB users (such as Spyglass) have to remember to multiple the data by theconversion
factor to get the data in volts. (The NWB team is working on improving this messaging...). Note that this makes using the data just a little slower and converting the data just a little faster.I suggest that the ephys data be stored in the original ADC units, because currently some precision is lost, and the cost of multiplying during use is small.