setup_notches records incorrect S21 data for some resonators

[dpdutcher]

Describe the bug

The resp_eta_scan (complex s21) in some tunefiles repeats the exact same complex s21 data across multiple resonators, and this s21 does not match that taken with an external VNA.

Caleb has a detailed issue of this filed within sodetlib here: Multiple resonators on a Tune file have the exact same complex 21 array ('resp_eta_scan' in the tunefiles) #300.

Below are plots generated by a visiting student at Princeton that also show the problem on three resonators: the blue line is data from an external VNA, and the orange line is the resp_eta_scan recorded by SMuRF. The data is centered around where we expect the resonance to be, but is wrong and identical between these three resonators.

To Reproduce

Steps to reproduce the behavior: Either (a) follow the code example in the linked sodetlib issue above, or (b) examine the resp_eta_scan data for each resonator in a tunefile. I don't think we know yet exactly how prevalent this is across tunefiles, but in Caleb's example there are 8 repeated S21s across just 44 resonators.

[swh76]

@dpdutcher two questions on this;

1) could you please send me that tune file in Caleb's post (/mnt/so1/data/pton-rd/legacy/smurfsrv/tune/1636578762_tune.npy).

2) are you usually seeing this when you run find_freq to generate the list of frequencies for setup_notches?

I just tried to reproduce with the following ;

S.freq_resp=S.fake_resonance_dict([5750.0, 5826.8, 5519.6, 5673.2, 5788.4, 5942.0, 5634.8, 5711.6, 5922.8, 5846.0, 5538.8, 5999.6, 5692.4, 5807.6, 5961.2, 5577.2, 5730.8, 5606.0, 5990.0, 5682.8, 5874.8, 5778.8, 5932.4, 5625.2, 5548.4, 5817.2, 5970.8, 5894.0, 5586.8, 5908.4, 5601.2, 5831.6, 5946.8, 5870.0, 5716.4, 5927.6, 5850.8, 5697.2, 5658.8, 5889.2, 5582.0, 5735.6, 5918.0, 5841.2, 5994.8, 5687.6, 5802.8, 5879.6, 5726.0, 5783.6, 5937.2, 5860.4, 5706.8, 5822.0, 5975.6, 5668.4, 5591.6, 5829.2, 5522.0, 5637.2, 5867.6, 5771.6, 5925.2, 5848.4, 5541.2, 5810.0, 5656.4, 5886.8, 5992.4, 5685.2, 5800.4, 5954.0, 5570.0, 5723.6, 5934.8, 5550.8, 5704.4, 5512.4, 5666.0, 5896.4, 5589.2, 5742.8, 5757.2, 5910.8, 5834.0, 5680.4, 5949.2, 5565.2, 5718.8, 5930.0, 5853.2, 5546.0, 5814.8, 5507.6, 5661.2, 5891.6, 5738.0, 5920.4, 5613.2, 5843.6, 5536.4, 5997.2, 5690.0, 5958.8, 5651.6, 5882.0, 5574.8, 5728.4, 5786.0, 5632.4, 5824.4, 5670.8, 5901.2, 5594.0, 5904.8, 5597.6, 5828.0, 5520.8, 5981.6, 5674.4, 5789.6, 5559.2, 5924.0, 5616.8, 5540.0, 5693.6, 5501.6, 5655.2, 5885.6, 5578.4, 5732.0, 5914.4, 5837.6, 5530.4, 5799.2, 5645.6, 5780.0, 5626.4, 5856.8, 5549.6, 5703.2, 5511.2, 5664.8, 5895.2, 5756.0, 5909.6, 5602.4, 5832.8, 5679.2, 5794.4, 5948.0, 5640.8, 5564.0, 5717.6, 5928.8, 5852.0, 5967.2, 5660.0, 5919.2, 5535.2, 5996.0, 5804.0, 5650.4, 5573.6, 5727.2, 5784.8, 5938.4, 5631.2, 5861.6, 5554.4, 5516.0, 5976.8, 5900.0, 5746.4, 5753.6, 5600.0, 5830.4, 5676.8, 5792.0, 5868.8, 5561.6, 5715.2, 5772.8, 5926.4, 5619.2, 5849.6, 5542.4, 5504.0, 5964.8, 5657.6, 5734.4, 5763.2, 5840.0, 5993.6, 5686.4, 5801.6, 5648.0, 5571.2, 5724.8, 5782.4, 5936.0, 5552.0, 5705.6, 5820.8, 5758.4, 5912.0, 5528.0, 5988.8, 5796.8, 5566.4, 5624.0, 5854.4, 5700.8, 5508.8, 5969.6, 5892.8, 5585.6, 5739.2, 5921.6, 5844.8, 5537.6, 5691.2, 5806.4, 5883.2, 5729.6, 5864.0, 5556.8, 5825.6, 5518.4, 5979.2, 5595.2])

S.setup_notches(3,new_master_assignment=True)

resp_eta_scans=[]
for k in S.freq_resp[3]['resonances'].keys():
   resp_eta_scans.append(S.freq_resp[3]['resonances'][k]['resp_eta_scan'][0])

print(len(resp_eta_scans)==len(np.unique(resp_eta_scans)))

but am not seeing any duplicates, at least in one attempt (the above fakes the find_freq input for 231 resonators with ~2 MHz spacing, adding some jitter to each resonator frequency). I also tried loading that tune file and don't see any duplicates there either.

[swh76]

Oh ok I think I may be able to reproduce it;

S.freq_resp=S.fake_resonance_dict([5746.4-0.3,5746.4-0.2,5746.4-0.1,5746.4,5746.4+0.1,5746.4+0.2,5746.4+0.3])

S.setup_notches(3,new_master_assignment=True)

resp_eta_scans=[]
for k in S.freq_resp[3]['resonances'].keys():
   resp_eta_scans.append(S.freq_resp[3]['resonances'][k]['resp_eta_scan'][0])

I'm seeing if multiple resonance frequency candidates are passed to setup_notches in the same subband (5746.4 MHz is the center of a 1.2 MHz subband in band 3), after 2nd entry, the complex s21 data is copies;

In [32]: resp_eta_scans
Out[32]: 
[(-0.015200138092041016+0.11574888229370117j),
 (0.07043766975402832+0.0944068431854248j),
 (0.07043766975402832+0.0944068431854248j),
 (0.07043766975402832+0.0944068431854248j),
 (0.07043766975402832+0.0944068431854248j),
 (0.07043766975402832+0.0944068431854248j),
 (0.07043766975402832+0.0944068431854248j)]

which is obviously a bug. Looking into it.

[swh76]

I think I've implemented the fix (at least for the problem I found) in branch issue737. Need to verify using Caleb's list of frequencies if it fixes everything he was seeing too.
To try to speed this up I co-opted an old single frequency scan routine eta_scan for scanning the unassigned channels, so I also need to confirm that channels scan the same way whether or not they're assigned or unassigned. Here's a snippet I've been using for testing that's probably only useful for me (I had to use our old eta_scan function to scan unassigned channels, but it turned out it has a different I/Q convention, so I used this to confirm I understood that):

################################################################
#### Compare scaled setup_notches scan to single eta_scan

#### Compare raw setup_notches scan to single eta_scan
plt.ion()
band=3
fc=5746.4
sweep_width=.3              
df_sweep=.002              
f_sweep = np.arange(-sweep_width, sweep_width, df_sweep)
n_channels = S.get_number_channels(band)
n_step  = len(f_sweep)
S.load_tune()

## setup_notches default algo
S.freq_resp=S.fake_resonance_dict([fc])
S.setup_notches(band,new_master_assignment=True, scan_unassigned=False)
idx=0

# get raw
Isn = S.get_eta_scan_results_real(band, count=n_step*n_channels)
Qsn = S.get_eta_scan_results_imag(band, count=n_step*n_channels)

Isn = S.get_eta_scan_results_real(band, count=n_step*n_channels)
Isn = np.asarray(Isn)
idx = np.where( Isn > 2**23 )
Isn[idx] = Isn[idx] - 2**24
Isn /= 2**23
Isn = Isn.reshape(n_channels, n_step)

Qsn = S.get_eta_scan_results_imag(band, count=n_step*n_channels)
Qsn = np.asarray(Qsn)
idx = np.where( Qsn > 2**23 )
Qsn[idx] = Qsn[idx] - 2**24
Qsn /= 2**23
Qsn = Qsn.reshape(n_channels, n_step)

Isn = Isn[S.freq_resp[band]['resonances'][0]['channel']]
Qsn = Qsn[S.freq_resp[band]['resonances'][0]['channel']]

## Apparently, setup_notches returns etaScanResultsReal.  
## This is populated by ./python/CryoDet/DspCoreLib/CryoDetCmbHcd/_SerialFindFreq.py.
## That function polls frequencyError.

# eta_scan runs runEtaScan in ./python/CryoDet/DspCoreLib/CryoDetCmbHcd/_CryoChannels.py
# which also polls frequencyError

## does this work?
tone_power=12
sb=np.abs(S.get_subband_centers(band)[1]-(fc-S.get_band_center_mhz(band))).argmin()
offset=(fc+-S.get_band_center_mhz(band))-S.get_subband_centers(band)[1][sb]
Ies,Qes = S.eta_scan(band, sb, offset + f_sweep, tone_power)

idx = np.where( Ies > 2**23 )
Ies[idx] = Ies[idx] - 2**24
Ies /= 2**23

idx = np.where( Qes > 2**23 )
Qes[idx] = Qes[idx] - 2**24
Qes /= 2**23

plt.figure()

end=10
Respes = (Qes[:end] - 1j*Ies[:end])

plt.plot(np.real(Respes),np.imag(Respes),label='eta_scan')
plt.plot(Isn[:end],Qsn[:end],'r--',label='setup_notches')
plt.axes().set_aspect('equal')
plt.legend()

plt.figure()

respsn=Isn + 1j*Qsn
respes=Ies + 1j*Qes
plt.plot(respes,label='eta_scan')
plt.plot(respsn,'r--',label='setup_notches')
plt.legend()

plt.show()

and here's a verification script (here I'm confirming that for 3 unassigned channels the eta_scan function seems to correctly scan them):

plt.ion()
band=3
fc=5746.4
sweep_width=.3              
df_sweep=.002              
f_sweep = np.arange(-sweep_width, sweep_width, df_sweep)
n_channels = S.get_number_channels(band)
n_step  = len(f_sweep)

## setup_notches default algo
S.freq_resp=S.fake_resonance_dict([fc-0.2,fc-0.1,fc])
S.setup_notches(band,new_master_assignment=True, scan_unassigned=True)

#In [9]: S.freq_resp[3]['resonances'][0].keys()
#Out[9]: dict_keys(['freq', 'eta', 'eta_scaled', 'eta_phase', 'r2', 'eta_mag', 'latency', 'Q', 'freq_eta_scan', 'resp_eta_scan', 'subband', 'channel', 'offset'])

for idx in range(3):
    plt.plot(S.freq_resp[3]['resonances'][idx]['freq_eta_scan'],S.freq_resp[3]['resonances'][idx]['resp_eta_scan'])