search

Ulm-IQO / qudi-iqo-modules

A collection of qudi measurement modules originally developed for experiments on colorcenters in semiconductor materials.
GNU General Public License v3.0
11 stars 28 forks source link

Wavemeter Toolchain #59

Open ksenkalla opened 1 year ago

ksenkalla commented 1 year ago

Feature Description

Rewrite the wavemeter toolchain in such a way that the wavemeter can use the data instream interface.

Related Problem

Porting from old core did not help as the "slow counter" doesn't exist anymore

Considered Alternatives

Alternative would be using the laser scanning directly using voltage to scan the laser. But this doesn't help to monitor the wavemeter

Additional Context

The following steps are considered to be done to get the toolchain ready:

Contact Details

katharina.senkalla@uni-ulm.de

qku commented 1 year ago

Hello, I tested the hardware file after moving to analog channels and it seems to work fine with the wavemeter I am using!

I saw that you increased the hardcoded number of digits behind the decimal in the time series gui. I guess it would be useful to have this as a config option or even accessible in the gui itself at some point.

qku commented 1 year ago

Regarding the plan from here: are we planning to stick to the hardware file implementing the instreamer interface? I understand that you have some plans for a custom logic and gui, but the hardware file will stay like this (with further improvements), right?

ksenkalla commented 1 year ago

Hello, I tested the hardware file after moving to analog channels and it seems to work fine with the wavemeter I am using!

I saw that you increased the hardcoded number of digits behind the decimal in the time series gui. I guess it would be useful to have this as a config option or even accessible in the gui itself at some point.

Thats a good point, I will add this to the things we want to do.

Regarding the plan from here: are we planning to stick to the hardware file implementing the instreamer interface? I understand that you have some plans for a custom logic and gui, but the hardware file will stay like this (with further improvements), right?

Yes exactly: For the future the hardware file can be adjusted so that all functions of the wavemeter can be covered. So on top of the already implemented methods further functions can be implemented (as e.g. setting the acquisition timing etc.) We were setting it up like this with the notion that we try to standardize the modules, in sense of that any other wavemter can be used with the same toolchain. And we found that every wavemeter at least can read continuously some wavelength, which is basically an instream of wavelengths.

qku commented 1 year ago

Hi @ksenkalla , I just played around with the latest state of your code. Unfortunately I cannot get the histogram GUI to work. What has to be running for it to work properly? How is it connected to the time series GUI/logic? At this point, I am pretty clueless about the idea behind this wavemeter histogram system.

qku commented 1 year ago

Independently of the histogram gui, I think it would be cool to improve the wavemeter hardware module implementing the time series interface, e.g. by adding support for multiple wavemeter channels and units. I would be happy to work on it myself and contribute via a PR. A robust base can facilitate various wavemeter use cases, which may be rather diverse from one setup/experiment to another. Maybe it is even worth to consider merging a working hardware file into main already even if the histogram functionality is not finished yet.

qku commented 1 year ago

A matter I would like to discuss further is interpolation. Interpolation is neccessary here because the data instreamer interface is designed for fixed data rates, correct? Wouldn't it make sense then to automatically set the data rate according to the exposure time set for the wavemeter? And also warn the user if the actual rate of data coming from the wavemeter is much slower than what e.g. the time series GUI is displaying to the user? A possible caveat might be that the data rate needs to be the same for all channels, correct?

Nick-Grimm commented 1 year ago

Hi @ksenkalla , I just played around with the latest state of your code. Unfortunately I cannot get the histogram GUI to work. What has to be running for it to work properly? How is it connected to the time series GUI/logic? At this point, I am pretty clueless about the idea behind this wavemeter histogram system.

Hi @qku, You can have a look at the wavemeter_rework.cfg (and in general files of the commit on 31st of March) in order to get an idea what has to be running. At the moment we are using an active wavemeter and an instream dummy (with one digital channel) for the hardware. The wavemeter logic is basically only connected to the timeserieslogic via a signal. This new signal (sigDataChangedWavemeter) emits new count values and the wavelength values are interpolated upon the number of counts. Therefore, for the wavemetergui to be running we need a running timeseries gui and an exposed wavemeter. If you have further question about the idea or how to get it running let us know. FYI, I just pushed the latest version of the code.

Nick-Grimm commented 1 year ago

A matter I would like to discuss further is interpolation. Interpolation is neccessary here because the data instreamer interface is designed for fixed data rates, correct? Wouldn't it make sense then to automatically set the data rate according to the exposure time set for the wavemeter? And also warn the user if the actual rate of data coming from the wavemeter is much slower than what e.g. the time series GUI is displaying to the user? A possible caveat might be that the data rate needs to be the same for all channels, correct?

For the moment the update rate of the wavemeter is determined by the data rate of the timeserieslogic/instream_dummy. Hence, interpolation is needed to connect this data rate and the rate given by the exposure time of the wavemeter. Indeed, it would be best if these rates are in the same order of magnitude. I am not sure if setting the exposure time such that it matches the data rate would avoid interpolation, since as you mention, the data rate from the timeseries is often faster. For this we also thought about a warning.

qku commented 1 year ago

@Neverhorst I really like how you improved the instream interface in your dev branch! I started to make the wavemeter hardware module fit the new requirements and make use of the TIMESTAMP timing (see this branch).

So far you assumed a single timestamp per sample. However, the wavemeter gives out timestamps per channel, since those are not measured at the same time. Without looking into the possible implications to time series logic and gui, could we allow different timestamps per channel? I guess having differing x-axes could make plotting readings of multiple channels potentially more complex.

Furthermore, is it really necessary to require the methods read_data_into_buffer and read_available_data by the interface and expose them this way to logic? They are not used anywhere so far for existing implementations afaik. The wavemeter hardware module continuously receives data from the hardware and puts it into an internal buffer. Calling these functions with a new buffer argument seems artificial. read_data is the important method I believe and maybe the only one required. Or am I missing something?

Neverhorst commented 1 year ago

@qku Thank you, but be advised this branch is still subject to a lot of change. So maybe don't implement a hardware module in detail, yet. I try to finish it over the weekend and open a PR.

Regarding your questions/remarks:

  1. After our last discussion I gave it a bit more thought. My conclusion was, that timestamped sample streams with a sample timing that does not correlate with each other (independent sample streams) do not belong in the same hardware module. Completely independent sample streams can only be processed by a dedicated experiment logic anyway. There is no (meaningful) general way for joint processing or even display of an rabitrary number of independent streams as far as I can see. In other words there need to be a minimum of correlation between different streams in order to be considered logically grouped (in a hardware module). Independent streams warrant a parallel configuration (e.g. 2 in-streamer GUIs). For specialized measurements that absolutely need to process independent data streams together, you would need a logic module that connects to 2 (or more) separate streaming hardware modules. I'm happy to test this opinion in a deeper discussion if you like.

  2. The different ways of getting data from the hardware module have their applications. Just because so far there is only the TimeSeriesReaderLogic does not mean other modules might not need it. Hardware interfaces should not be subject to change once they are released unless absolutely necessary. That is why I tried to cover most use cases already.

    • read_data_into_buffer is the most efficient way if you want to receive a fixed amount of samples (possibly blocking)
    • read_available_data is the most efficient way if you want to return as fast as possible with whatever number of samples available (non-blocking)
    • read_data and read_single_point are the "lazy" way of retrieving data if speed and efficiency is of no concern (e.g. a slow data stream). It allows simpler logic modules due to no buffer management but the downside is that each call allocates a potentially huge new array (slow).

    Continuous streaming of data must almost always be done with care in order to ensure sufficient throughput depending on the application.
    However most hardware implementations can probably do something like the DataInStreamDummy module and just implement read_data_into_buffer and derive the other read methods in a trivial way. So not much more programming to be done in order to satisfy the different interface methods.

qku commented 1 year ago

@Neverhorst I am happy to see your continued work on the time time series toolchain! Thank you for explaining your thought process. This could be part of the documentation maybe!

I agree with your reasoning about the minimal required correlation between individual channels on a single data instream module. This will complicate the multi-channel wavemeter implementation a little, but I already have an idea how to do it.

I tried if two callback procedures can run simultaneously @ksenkalla. Unfortunately, this doesn't work. So simply having one hardware module per wavemeter channel is no option. However, what we could do is have one main wavemeter data instream module which gathers measurements and timestamps of multiple channels in data channels (mabye the RANDOMsampling timing could be used here?) and dedicated single-channel data instream modules (interfuses?) which rearrange the readings and make them available as TIMESTAMP sampled data.

Let me know what you think about this!

Neverhorst commented 1 year ago

Hi @qku,

from our discussion last time I understood the multichannel wavemeter is acquiring the different channels in a round-trip style with a fixed time interval. If this is true, with the new interface you have basically 2 options:

  1. If the round-trip interval is really fixed, you could use the CONSTANT sample timing mode and just use this interval
  2. If the round-trip interval is non-constant, you could just always use the timestamp for a specific channel for all channels and use the TIMESTAMP sample timing mode.
               Timestamp 1                                                   Timestamp 2
                    │                                                             │
                    ▼                                                             ▼
┌───────────────────┬───────────────────────┬────────────────┬┬───────────────────┬───────────────────────┬────────────────┐
│   wavelength 1    │      wavelength 2     │  wavelength 3  ││   wavelength 1    │      wavelength 2     │  wavelength 3  │
└───────────────────┴───────────────────────┴────────────────┴┴───────────────────┴───────────────────────┴────────────────┘

|----------------------- sample interval --------------------|
qku commented 1 year ago

Thank you for your input @Neverhorst.

What you are saying is true: the round-trip time composing the individual channel exposure times and switching times is fixed. In an earlier version of my implementation, I estimate the sampling rate based on the wavemeter settings. However, there are a few caveats which make me feel uncomfortable with implementing the CONSTANT sample timing mode as in your first suggestion:

  1. Whenever a channel is activated or deactivated, the sample rate changes.
  2. It is possible to change the exposure time "behind qudi's back" in the gui from the manufacturer. While there are ways to detect and or prevent this, they will make the implementation more complex.
  3. I am not sure on how to implement a request to change the sample rate. Should this trigger a proportional change of all exposure times? Should this be not allowed at all?

My last suggestion came from the assumption that it is critical to preserve all timestamps that come from the wavemeter. However, after giving it some thought, I believe just keeping the timestamps from a single channel should satisfy all synchronization needs to an extent that is possible to implement on this software level.

I would therefore prefer to go with your second suggestion. Even if the sample rate does not change regularly, I believe it would make the implementation simpler and more robust against errors.

qku commented 1 year ago

I am about to be done with my work on the wavemeter hardware module. How far are you with your work on gui and logic @ksenkalla @Nick-Grimm ? Depending on your progress, I would prefer to open a separate PR for the hardware module directly to main. What do you think @Neverhorst ?

Neverhorst commented 1 year ago

If the new hardware module will implement the DataInStreamInterface, I see no problem in merging it in a separate PR. As long as a new hardware module has a corresponding interface and dummy module, it's fine.

qku commented 10 months ago

@ksenkalla @Nick-Grimm I am trying out your wavemeter scanner on our NVs. To get it to work, I had to change line 160 in wavemeter_scanning_logic_3.py to use the new name for the keyword: samples_per_channel (this was a recent change with the instream interface).

The module is running very smoothly for me. I have the wavemeter running remotely on another PC.

I find it confusing that the time series GUI has to run in the background. Why is that? Is it possible to activate the time series GUI automatically?

I will analyze the data and compare it to data that I acquired with similar settings with the old laser scanner from old qudi.

qku commented 10 months ago

I can do more detailed testing once you finalize things and open a PR!

Neverhorst commented 10 months ago

I find it confusing that the time series GUI has to run in the background. Why is that? Is it possible to activate the time series GUI automatically?

That is strange indeed. However you can always configure modules to automatically start with qudi using the global (list) config option startup_modules, e.g.:

global:
    # list of modules to load when starting
    startup_modules: 
        - my_time_series_gui
qku commented 10 months ago

I will analyze the data and compare it to data that I acquired with similar settings with the old laser scanner from old qudi.

I did the analysis. We get comparable center frequencies for our NV resonance both with hardware-timed scanning with the NIDAQ and software-timed scanning with the wavemeter logger. So it looks like we could also use this tool for our common scan speeds. Very cool!

Nick-Grimm commented 10 months ago

I find it confusing that the time series GUI has to run in the background. Why is that? Is it possible to activate the time series GUI automatically?

Indeed, the time series GUI does not have to run in the background. It would suffice if the time series reader logic is streaming and this could be started automatically. However, for the interplay when both applications are running I thought it would be clearer if the time series reader logic is solely handled by the GUI. But of course input is appreciated how to handle such things.

Nick-Grimm commented 10 months ago

I did the analysis. We get comparable center frequencies for our NV resonance both with hardware-timed scanning with the NIDAQ and software-timed scanning with the wavemeter logger. So it looks like we could also use this tool for our common scan speeds. Very cool!

Thats good to hear, thanks for testing already. I will finalize things with the wavemeter dummy and then open a PR for more detailed testing. 👍

qku commented 10 months ago

Indeed, the time series GUI does not have to run in the background. It would suffice if the time series reader logic is streaming and this could be started automatically.

How about connecting your main logic directly to the time series reader logic? That would start the time series logic automatically.

However, for the interplay when both applications are running I thought it would be clearer if the time series reader logic is solely handled by the GUI. But of course input is appreciated how to handle such things.

I don't understand why the time series GUI is required. The wavemeter data is displayed in the main GUI. How are you using the two GUIs - what do you do in which?

qku commented 10 months ago

I will finalize things with the wavemeter dummy and then open a PR for more detailed testing. 👍

Do we really need a dedicated dummy? The wavemeter implements the data instream interface. Why don't we just use the data instream dummy?

Nick-Grimm commented 10 months ago

How about connecting your main logic directly to the time series reader logic? That would start the time series logic automatically.

The main logic is connected to the time series reader logic. That activates the time series logic automatically, however it does not yet automatically start accumulating counts.

I don't understand why the time series GUI is required. The wavemeter data is displayed in the main GUI. How are you using the two GUIs - what do you do in which?

The time series GUI is used as a slow counter to display counts. In our workflow this runs constantly. The wavemeter GUI displays wavemeter data and eventually correlates them with the counts from the time series reader logic. And as the slow counter (time series GUI) runs constantly we e.g. don't want the wavemeter logic to stop the time series toolchain.

Nick-Grimm commented 10 months ago

Do we really need a dedicated dummy? The wavemeter implements the data instream interface. Why don't we just use the data instream dummy?

I don't think we need a dedicated dummy. The data instream dummy can be used. However it does no provide attributes such as _wm_start_time. That is for now required for the lower plot, but maybe there is a way to circumvent that.