Dark pixel indices discrepancy (USB4000 / FLAME-T)

fohrloop commented 4 years ago

spectrometer and system information

model: FLAME-T (recognized as USB4000). Host Interface: FLAME-T 4.00.4, FPGA: 4.05.6.
operating system: Win10 64bit
python version: 3.7.7 32-bit
python-seabreeze version: 1.1.0
installed-via: pip

current problem

Seems that there is some discrepancy in the dark pixels indices.

USB4000 / FLAME-T Manual values:

usb4000-oem-data-sheet.pdf
Same values in the FLAME-T docs: Flame-Manual.pdf

Values using pyseabreeze

[ 5 6 7 8 9 10 11 12 13 14 15]

The problem is

The manual says the dark pixels are pixels 6-18. It also means that there are 13 dark pixels.
Pyseabreeze says the dark pixels are pixels 5-15. It also means that there are 11 dark pixels.

If the manual uses indexing starting from 1 (and in python it is from 0), the starting pixel is correct, but the range should be 5-17 in pyseabreeze.

steps to reproduce

import seabreeze
seabreeze.use('pyseabreeze')

from seabreeze.spectrometers import Spectrometer
spec = Spectrometer.from_first_available()

s = spec.features['spectrometer'][0]

print(s.get_electric_dark_pixel_indices())

this will print

[[ 5  6  7  8  9 10 11 12 13 14 15]]

Proposed fix

Change

#pyseabreeze.devices.py

    # spectrometer config
    dark_pixel_indices = (
        DarkPixelIndices.from_ranges((5, 16)),
    )  # as in seabreeze-3.0.9

into

#pyseabreeze.devices.py

    # spectrometer config
    dark_pixel_indices = (
        DarkPixelIndices.from_ranges((5, 18)),
    )  # as in seabreeze-3.0.9

(this should add the two pixels to the dark pixels)

ap-- commented 4 years ago

Good catch! :smiley:

But this is a bit more complicated sadly. I'd recommend reading https://sourceforge.net/p/seabreeze/discussion/906079/thread/b7214f2d/ where I asked about this some time ago.

I will probably opt for this being a wont-fix Because there's a chance that the two added pixels are not edc pixels for all spectrometers out there.

The scientifically best way in my opinion would be to write a script that interactively guides you through the necessary steps to measure which of the pixels on the spectrometer are actually edc pixels. They should be non-zero and not influenced by broad spectrum light sources.

fohrloop commented 4 years ago

Ok I see it is complicated. I have previously been using the Ocean Optics 2000 4000 Labview driver and there also the dark pixels were 6-18. The following table contains the driver v1.4.4. settings:

name	decimal number	start black pixel	stop black pixel	start optical pixel	stop optical	All Pixels	not usable	note reference
USB2000	4098	2	24	26	2074	3648	0-1
ADC1000	4100	0	0	26	2047	2048
HR2000	4106	2	24	26	2047	2048	0-1	Note 1
HR4000	4114	6	18	22	3669	3648	1-5 ,3670-3681
HR2000+	4118	0	17	20	2047	2048	18-19
QE65000	4120	0 , 1038	3 , 1043	10	1033	1044	4-9 , 1034-1037	Note 2
USB2000+	4126	2	17	20	2047	2048	18-19	Note 3
USB4000	4130	6	18	22	3669	3648	1-5,3670-3681
NIRQuest512	4134	NaN		0	511	1024
NIRQuest256	4136	NaN		0	255	512
MAYA Pro	4138	1,2064	3,067	10	2047	2068	0,4-9
MAYA	4140	0,2072	7,079	16	2063	2080	8,15-2064-2071
Torus	4160	0	17	20	2047	2028	18-19
Flame	4126	2	17	20	2047	2048	0	0

Note 1: The HR2000 only digitizes the first 2048 pixels
Note 2: In the QE6500 are the pixels 038..1047 (black pixels) in software moved to the back of the second set of black pixels.
Note 3: Per Ocean Optics tech support USB2000+ data sheet incorrectly labels 0-17 as dark pixels. Pixels 0 and 1 contain garbage and should not be used [AC]. The dark pixels are 2-17 [AC].
Note 4: All Pixel values are from the Obsoleted Read Spectra VI

(and yes, there seems to be something wrong with the MAYA and MAYA Pro dark pixel indices, since they should be a comma separated list of intervals, as in QE65000. This is exactly how it was in the Labview driver. The end stop black pixels should probably be 3,2067 and 7,2079. The Flame only applies to Flame-S, since that matches with the total number of pixels.)

One option would be to let the user to choose to use the pixels 6 -> 18 instead of the default 5->15. Anyway, I believe the dark current effect can be mitigated with 10 or 13 pixels in almost equal accuracy. One thing which could be considered is to calculate dark pixel median instead of mean (like the Labview driver does). This would make the dark pixel count more robust to outliers. (or, that could be an option).

One more thing which is quite fishy with all of these indices is that for example with HR4000 and USB4000 (and FLAME-T) the indices start from one, and yet they are all interpreted the same way as with other devices (where indices start from zero).

ap-- commented 4 years ago

I thought about this a bit more, and I'd be happy to accept a PR that allows providing a list/tuple of indices to Spectrometer.intensities() via correct_dark_counts :smiley:

That way everyone who's concerned about dark count correction can tweak the indices as they wish. :sparkling_heart:

This would only touch code here:

https://github.com/ap--/python-seabreeze/blob/964f0f15cc981229986e3f41ccbc1f5b6b4cda5b/src/seabreeze/spectrometers.py#L137-L198

Cheers, Andreas

ap-- / python-seabreeze