Update GUI default parameters

[adampbeardsley]

I don't think anyone uses the parameters that default in the GUI anymore. At wsu we've been doing 1415-1425 MHz, 10s integrations, 5 ints per tuning. I think that frequency range is good, but maybe we want more time resolution so we can resolve bursty interference?

Curious what others think and what they're doing.

[Alexandra202421]

I did a narrower frequency range with 7 10 second integrations around 1417 and 1423. The reason for this was to get more data closer to the 1420 mark, but I also noticed the integrations were overlapping quite a bit in the code. Do the integrations overlap if they are super close together? That would definitely change what the ideal settings would be. Also, does this issue include changing the default parameters inside the disk image/ gui.py? Is that where I would make the pull request from?

[adampbeardsley]

Note: Sorry, this comment got out of hand.

I'm not sure what you mean by "the integrations were overlapping"... correct me if I'm wrong, but I think you are talking about how the 2MHz tunings overlap in frequency, as seen in this plot, where each color is a different tuning:

That overlapping is on purpose, and it's done using a setting that's actually not exposed on the GUI (maybe something we want to change). We use the overlap because (a) the edges of the band aren't always super reliable because of the low response, and (b) the SDR sometimes changes the automatic gain control between tunings and having overlapping data allows us to correct for that.

Perhaps it's worth describing each parameter and give a little detail on the various trade-offs.

• initial and final frequencies:
  • The lowest and highest central frequencies. Note the actual lowest frequency will be about 1MHz lower than the initial freq, and the actual highest tuning is 1MHz below what you set here (because of how python counts... again maybe we want to change this).
  • Current defaults are 1390 MHz and 1450 MHz
  • More frequencies away from the 21cm line we care about allows us to better fit the bandpass that's common to all tunings. But this takes up observing time. So we need to find the balance here.
• Integration time:
  • How long to average for each data point.
  • Default is 0.5 s
  • Need to balance shorter times so we can resolve intermittent RFI vs longer times to reduce data volume.
• Number of Integrations:
  • How many integrations to perform at each frequency tuning before moving to the next.
  • Default is 20
  • Need to balance faster observing (fewer integrations) vs lower random noise (more integrations)
  • The total time spent on a tuning is given by (integration time) x (number of tunings), so these parameters are somewhat coupled. WSU has been using a total of 50s on each tuning and that's given very good signal to noise.
• Some hidden parameters that we might want to expose or change:
  • df -- spacing between frequency tunings. Currently 1 MHz.
  • samp_rate -- sample rate (bandwidth per tuning). Currently 2 MHz.
  • veclength -- how many frequency channels per frequency tuning. Currently 1024.

Finally, your last question - this issue is to change the defaults in the GUI, so in the gui.py file. I can help you figure out how to do that once we've decided on the actual parameters. It will also end up in the disk image, but that's a separate issue -- it just requires rebuilding the image once changes like this are settled.

[Alexandra202421]

Oh okay I guess I should have assumed that the over lap in frequency measurements was intentional, that makes sense.

Just out of curiosity, in the example plot that you sent, it doesn't look like the default range was used because the frequencies don't go down to 1390 or up to 1450; but say for the first integration (the far left blue curve), is the frequency its "told" to measure 1410? Like does the center of the peak note what frequency its measuring or is it based on the farthest left part of the curve and go up from there? ex with a sample rate of 2 Mhz and measured at 1420, would it actually collect data from 1419 to 1421? This doesn't change a whole lot in terms of default parameters I just want to make sure I'm understanding where the values are coming from.

I think the python index shift is fine but I think longer integrations would be beneficial here. I took a few data sets using the default 0.5s and a few at 0.8s and they all came out significantly worse than the 10s ones that I used for my plots in my memo. I think sticking with 50s total makes sense, maybe 10 tunings at 5 seconds each? Would it be worth going out and taking several data sets with various parameters just to see how the data changes?

[adampbeardsley]

You are correct about the frequency its "told" to measure. Here's another example from last week, where I actually know the parameters used: In this plot, we set the initial frequency to 1415 and the final to 1424. (actually the final was set to 1425 but we lost the last one.) But as you can see, the center frequencies start at 1415 and end at 1423, and the actual frequencies observed go 1414 to 1424.

I agree that the default times (total of 10s per frequency tuning) are probably too short.

Before anyone goes out to take a bunch of data, you could actually try taking a dataset we already have and just limit what data is actually used in the analysis. For example, I think we have a dataset that used the full 1390-1450 frequency range (abeardsley_Winona-HS-Park_2022.10.08_1_6:12_pm on the chart server). One could try narrowing limiting the analysis to using subsets to determine how many tunings are really needed to get the good bandpass estimate.

[Alexandra202421]

Okay Ill start messing around with reading in the Winona data to see how it changes the graphs!

[Alexandra202421]

Do I want to change the files going into the code or do I want to alter the code to cut off some of the files? I am having a hard time following all of the places the data is referenced in the code and feel like it might be easier to just shorten the number of pointings going in

[adampbeardsley]

If you're using the chart analysis tutorial, I think the easiest place is to do it in this section:

for i in range(ntrials):
    d, m = chart.analysis.read_run(directory=data_dir + paths[i])
    d = np.array(d)
    data.append(d)
    mdata.append(m)

Rather than appending d to the data variable, you could do something like data.append(d[10:20]) (I'm making up numbers here). That would only keep tunings [10, 20) for the rest of the notebook. You'd have to do something similar for the metadata... but I think m is a list so you need to first convert it to a numpy array to do the slicing: mdata.append(np.array(m[10:20]))

[Alexandra202421]

ahhh thank you! I was way overcomplicating it!

[Alexandra202421]

Also something that I've noticed: the code only works when there is more than one pointing being processed at a time. Is this something thats worth fixing? I don't think it would be too in depth to add a part to plot with only one pointing

[adampbeardsley]

ah shucks, could you make a separate issue for that, and include details about how it's failing? I had tried to make it work for a single pointing, but I guess I wasn't as successful as I thought.

[Alexandra202421]

It's just returning an axs plotting error because its expecting multiple axis, I can make an issue and fix it tomorrow!

[lmberkhout]

I think the default settings proposed are good, I think we should clean up and include that handy description Adam gave of what the tradeoffs are in the GUI walkthrough.

[adampbeardsley]

Closed with #228