Loading old debug data from Tracking Setup

jlashner commented 2 years ago

Describe the bug

After attempting to load one of the debug dat files from tracking_setup, I learned that though we are able to read in the f, df, and sync signals, there is no info that tells you how to map those arrays to the correct channel number. From what I can tell, the decode_data function relies on the get_tone_frequency_offset_mhz function to determine the channel order and correctly map the tracking data to the channel number, but unfortunately, this data is not saved anywhere at the time of tracking and is not available in offline mode, so I don't think it's possible to determine the correct channel order.

So we are able to see what general tracking statistics look like from old tracking-setup data, but we are unable to go back and look at how well a specific channel was tracking at a certain point in time. I imagine this is something we want to be able to do, so I think we should save the channel order and any other info required to parse debug data in the take_debug_data function

To Reproduce

Try to load a debug .dat file that comes from an older instance of pysmurf.

Expected behavior

We want to be able to fully recover debug data

swh76 commented 1 year ago

Looking at this with @Prakamya01.

This works as a short term kludge:

S.get_tone_frequency_offset_mhz = lambda band : np.array([   0. ,  307.2,  153.6, -153.6,   76.8, -230.4,  230.4,  -76.8,
         38.4, -268.8,  192. , -115.2,  115.2, -192. ,  268.8,  -38.4,
         19.2, -288. ,  172.8, -134.4,   96. , -211.2,  249.6,  -57.6,
         57.6, -249.6,  211.2,  -96. ,  134.4, -172.8,  288. ,  -19.2,
          9.6, -297.6,  163.2, -144. ,   86.4, -220.8,  240. ,  -67.2,
         48. , -259.2,  201.6, -105.6,  124.8, -182.4,  278.4,  -28.8,
         28.8, -278.4,  182.4, -124.8,  105.6, -201.6,  259.2,  -48. ,
         67.2, -240. ,  220.8,  -86.4,  144. , -163.2,  297.6,   -9.6,
          4.8, -302.4,  158.4, -148.8,   81.6, -225.6,  235.2,  -72. ,
         43.2, -264. ,  196.8, -110.4,  120. , -187.2,  273.6,  -33.6,
         24. , -283.2,  177.6, -129.6,  100.8, -206.4,  254.4,  -52.8,
         62.4, -244.8,  216. ,  -91.2,  139.2, -168. ,  292.8,  -14.4,
         14.4, -292.8,  168. , -139.2,   91.2, -216. ,  244.8,  -62.4,
         52.8, -254.4,  206.4, -100.8,  129.6, -177.6,  283.2,  -24. ,
         33.6, -273.6,  187.2, -120. ,  110.4, -196.8,  264. ,  -43.2,
         72. , -235.2,  225.6,  -81.6,  148.8, -158.4,  302.4,   -4.8,
          2.4, -304.8,  156. , -151.2,   79.2, -228. ,  232.8,  -74.4,
         40.8, -266.4,  194.4, -112.8,  117.6, -189.6,  271.2,  -36. ,
         21.6, -285.6,  175.2, -132. ,   98.4, -208.8,  252. ,  -55.2,
         60. , -247.2,  213.6,  -93.6,  136.8, -170.4,  290.4,  -16.8,
         12. , -295.2,  165.6, -141.6,   88.8, -218.4,  242.4,  -64.8,
         50.4, -256.8,  204. , -103.2,  127.2, -180. ,  280.8,  -26.4,
         31.2, -276. ,  184.8, -122.4,  108. , -199.2,  261.6,  -45.6,
         69.6, -237.6,  223.2,  -84. ,  146.4, -160.8,  300. ,   -7.2,
          7.2, -300. ,  160.8, -146.4,   84. , -223.2,  237.6,  -69.6,
         45.6, -261.6,  199.2, -108. ,  122.4, -184.8,  276. ,  -31.2,
         26.4, -280.8,  180. , -127.2,  103.2, -204. ,  256.8,  -50.4,
         64.8, -242.4,  218.4,  -88.8,  141.6, -165.6,  295.2,  -12. ,
         16.8, -290.4,  170.4, -136.8,   93.6, -213.6,  247.2,  -60. ,
         55.2, -252. ,  208.8,  -98.4,  132. , -175.2,  285.6,  -21.6,
         36. , -271.2,  189.6, -117.6,  112.8, -194.4,  266.4,  -40.8,
         74.4, -232.8,  228. ,  -79.2,  151.2, -156. ,  304.8,   -2.4,
          1.2, -306. ,  154.8, -152.4,   78. , -229.2,  231.6,  -75.6,
         39.6, -267.6,  193.2, -114. ,  116.4, -190.8,  270. ,  -37.2,
         20.4, -286.8,  174. , -133.2,   97.2, -210. ,  250.8,  -56.4,
         58.8, -248.4,  212.4,  -94.8,  135.6, -171.6,  289.2,  -18. ,
         10.8, -296.4,  164.4, -142.8,   87.6, -219.6,  241.2,  -66. ,
         49.2, -258. ,  202.8, -104.4,  126. , -181.2,  279.6,  -27.6,
         30. , -277.2,  183.6, -123.6,  106.8, -200.4,  260.4,  -46.8,
         68.4, -238.8,  222. ,  -85.2,  145.2, -162. ,  298.8,   -8.4,
          6. , -301.2,  159.6, -147.6,   82.8, -224.4,  236.4,  -70.8,
         44.4, -262.8,  198. , -109.2,  121.2, -186. ,  274.8,  -32.4,
         25.2, -282. ,  178.8, -128.4,  102. , -205.2,  255.6,  -51.6,
         63.6, -243.6,  217.2,  -90. ,  140.4, -166.8,  294. ,  -13.2,
         15.6, -291.6,  169.2, -138. ,   92.4, -214.8,  246. ,  -61.2,
         54. , -253.2,  207.6,  -99.6,  130.8, -176.4,  284.4,  -22.8,
         34.8, -272.4,  188.4, -118.8,  111.6, -195.6,  265.2,  -42. ,
         73.2, -234. ,  226.8,  -80.4,  150. , -157.2,  303.6,   -3.6,
          3.6, -303.6,  157.2, -150. ,   80.4, -226.8,  234. ,  -73.2,
         42. , -265.2,  195.6, -111.6,  118.8, -188.4,  272.4,  -34.8,
         22.8, -284.4,  176.4, -130.8,   99.6, -207.6,  253.2,  -54. ,
         61.2, -246. ,  214.8,  -92.4,  138. , -169.2,  291.6,  -15.6,
         13.2, -294. ,  166.8, -140.4,   90. , -217.2,  243.6,  -63.6,
         51.6, -255.6,  205.2, -102. ,  128.4, -178.8,  282. ,  -25.2,
         32.4, -274.8,  186. , -121.2,  109.2, -198. ,  262.8,  -44.4,
         70.8, -236.4,  224.4,  -82.8,  147.6, -159.6,  301.2,   -6. ,
          8.4, -298.8,  162. , -145.2,   85.2, -222. ,  238.8,  -68.4,
         46.8, -260.4,  200.4, -106.8,  123.6, -183.6,  277.2,  -30. ,
         27.6, -279.6,  181.2, -126. ,  104.4, -202.8,  258. ,  -49.2,
         66. , -241.2,  219.6,  -87.6,  142.8, -164.4,  296.4,  -10.8,
         18. , -289.2,  171.6, -135.6,   94.8, -212.4,  248.4,  -58.8,
         56.4, -250.8,  210. ,  -97.2,  133.2, -174. ,  286.8,  -20.4,
         37.2, -270. ,  190.8, -116.4,  114. , -193.2,  267.6,  -39.6,
         75.6, -231.6,  229.2,  -78. ,  152.4, -154.8,  306. ,   -1.2])

f,df,strobe=S.decode_data('/data/smurf_data/tracking_setup/1671824252_tracking_setup/outputs/1671824252.dat')

Unfortunately there's not a real easy way to fix this without hardcoding this array in. @jlashner how do you feel about having it just use the above array if we're in offline mode (e.g. if initialized SmurfControl instance via S =pysmurf.client.SmurfControl(offline=True))?

jlashner commented 1 year ago

What are the consequences of hardcoding the array instead of using the one returned by rogue? Does that mean the f values are shifted by some amount since the center frequencies are not right?

I think hardcoding this into pysmurf doesn't seem right to me, if it would result in misleading data when you use offline mode. I think for now we just post this kludge (along with the caveat that the frequency values will be shifted) for people to use if they really need to, with the long term solution being to make sure to use sodetlib tracking functions if possible.

swh76 commented 1 year ago

@jlashner so these frequencies are the polyphase filter bank (PFB) subband center frequency offsets. They are the same for every band, so actually using the above values should be correct, for all bands. We parameterized them by band in pysmurf / rogue only to maintain flexibility in case, for some reason, some future application needed different bands to have different PFBs, but all existing SMuRF implementations use this identical PFB structure for every 500 MHz band.

As far as hardcoding is concerned, offline mode actually only works because of hardcodes which assume the user is running the standard SO-SMuRF firmware. See e.g. https://github.com/slaclab/pysmurf/blob/02a237bc0c06ebe223b7c697a61b90a5d1a02e70/python/pysmurf/client/command/smurf_command.py#L509-L510

and

https://github.com/slaclab/pysmurf/blob/02a237bc0c06ebe223b7c697a61b90a5d1a02e70/python/pysmurf/client/command/smurf_command.py#L541-L542

and many other places. This is needed if no rogue server is accessible because pysmurf holds no state of its own (or at least, that was our intention, to allow for seamless changes in firmware, if required).

In the short term, I don't think adding this as a hardcode, if it's useful for anyone, is any worse than the status quo. A possibly better fix we'd batted around in the past would be to include in pysmurf releases a yaml state dump from a system configured using that release which could be loaded in offline mode instead of these hardcodes. Some of the machinery for that is already in pysmurf (see e.g. the yml keyword in https://github.com/slaclab/pysmurf/blob/02a237bc0c06ebe223b7c697a61b90a5d1a02e70/python/pysmurf/client/command/smurf_command.py#L127-L133) but I don't think it's fully implemented, and there's no simple way to automatically generate a state dump with each release (because it'd be most easily done with hardware).

jlashner commented 1 year ago

Ah I see, I was mistaken and thought it was using resonator centers (which change with smurf usage) instead of subband centers which won't realistically change. In that case I'm fine with hardcoding it in.

slaclab / pysmurf