Correlate timetag streams from two different correlators

[linzuzeng]

I guess we should allow a file to be linked to an aquisition device. then file1 could be linked to aquisition device1 with channels 0-5 and file2 could be linked to aquisition device2 with channels 6-10.

The channel number assignment for two time tagger in ETA is not yet written, but we should definitely support it. see #60

[Cyberfly100]

Zuzeng: In the ETA.run(), I would suggest using a dict like, {"devicename":"pathtofile"} so that it doesn't get too confusing about the correspondence of the files and the acquisition devices.

good idea, I think. it will not be immediately clear, which device belongs to which file on the home screen, though...

[linzuzeng]

True. Shall we add more parameters to the Acqusition Device Panel on the home screen, like path to the file and the what kind of Time Tagger is being used.

But then it would be confusing for batch mode and realtime mode...

[linzuzeng]

We should somehow make it clear that the Acquisiton Device is something like a placeholder that should be filled with either a timetag file or a memory buffer (for true realtime mode without writing to disk).

[Cyberfly100]

So far we have linked the aquisition device and file by just assuming there is only one and using ETA.run(filename). In hindsight, that was also not very transparent to the user. Maybe we can get away with it again until we come up with a better way? I like the way of linking them with a dict, but for it to be clear on the home screen, I think we would need to link them there and give only one of them to ETA.run(). Not sure what is better...

[Chris645]

Maybe it'll get transparent as soon as another Virtual Instrument is added. It'll describe that the two files are combined into a new variable (a dict) called cutfile_combined. cutfile_combined is then transferred to ETA.run(). I'm not sure if that makes sense ~

[linzuzeng]

Reply Edited: 12/12/19 19:14 Pacific, WA

So far we have linked the aquisition device and file by just assuming there is only one and using ETA.run(filename). In hindsight, that was also not very transparent to the user. Maybe we can get away with it again until we come up with a better way? I like the way of linking them with a dict, but for it to be clear on the home screen, I think we would need to link them there and give only one of them to ETA.run(). Not sure what is better...

How about this:

We add a new concept called "source". All VIs should read from that source and emits on virtual channels. On the Python side, we feed the "source" into ETA.run instead of cut_descriptor as suggested by @Chris645

The source is usually created using only one cut descriptor. But it can also be created using two or more cut descriptors. In the case that users need to merge two timetags, they can add a combined source using two cut descriptors.

Besides that, some of the virtual channels can be set as "sink", and events on those channels can be retrived from Python side and then written to disk for further processing, see https://github.com/timetag/ETA/issues/40#issuecomment-530227473

[linzuzeng]

I wonder if we should change the "acquisition device" button in main GUI to a better name like "Timetag Source", "Event Source" or "Source & Sink", because we need to use term "acquisition device" to refer to the individual timetag files that is later merged into one single "source".

Also, we can add a configuration button for the "source" that allows users to configure the acquisition devices for that source.

This issue might influence many aspects of ETA, so I decide to split it into many sub-issues. For GUI related stuff, see #101 For API related stuff, see #102

[linzuzeng]

We should also add a tool to preview the combined source, since it might become complicated in the future, see #64

[linzuzeng]

Now I wonder how would the simple_cut() and incremental_cut() should work with the combined source.

Assuing we use simple_cut, which cuts the file in equal length in size, we usually get inequal length in time because of the countrate fluctuation. I drew a sketch to illustrate the problem here. Red is a timetag file, purple is another timetag file, green line splits the simple_cuts.

In a non-parallel analysis mode where the context is mantained across each cut, this combined source will cause errors in most analysis, since the clips of different timetag timetags might not end at exactly the same moment in time.

Some file will stop earlier than the other in the first cut, and in the second cut this file will have leftover events that is older than the current time. It breaks the basic assumetion of almost all existing timetag analysis methods, that timetag can only grow in a monotonic order. Therefore, it will cause serious problems to start-stop masurement and many other types of analysis.

This problem will also prevent us from doing realtime on combined devices. 😭

[linzuzeng]

To address this problem, I suggest to introduce Time-based cutting. Unlike simple_cut(), it happens after file combination.

See https://github.com/timetag/ETA/issues/100#issuecomment-565261885

[gyger]

Whats missing for this issue to be used?

[linzuzeng]

Actually we are quite prepared for supporting this feature. The only missing parts are

1. some code on Python side for parsing the new RFILE tool, and
2. automatically feeding multiple Clips from a dictionary of generators.

I have implemented the first part, but unfortunately, it was left on an offline computer in another city. 😢 I will try to merge it in and implement the second part as soon as I get back.

Also, I would suggest to release ETA 0.6.6 first, as it has many API changes already, and save this for 0.6.7 or 0.7.0

[linzuzeng]

125 Merged to master

[linzuzeng]

Actually we are quite prepared for supporting this feature. The only missing parts are

1. some code on Python side for parsing the new RFILE tool, and

2. automatically feeding multiple Clips from a dictionary of generators.

I have implemented the first part, but unfortunately, it was left on an offline computer in another city. 😢 I will try to merge it in and implement the second part as soon as I get back.

Also, I would suggest to release ETA 0.6.6 first, as it has many API changes already, and save this for 0.6.7 or 0.7.0

Both of them have been implemented.

Correlation of zero file is tested to be working, see the simulation recipe in the current master. :) Haven't tested correlation of two files though, but I think it should work

@Chris645 sorry for letting you wait for so long

[linzuzeng]

I made a sample recipe to correlate from two correlators.

I don't have two files on my hand, so I made a recipe that has two RFILE tool to map the two channels from a same file to [0,1] and [2,3]. And then I use two dealy lines to further map [0,3] to [5,4]. The correlation results are exactly the same as running on one file with only chn [0,1].

But you can try setting a different path on one of the files to correlate two files

Correlation on multiple source.zip

[linzuzeng]

Changes is merged into master. :)