All Antenna Sky-Subtraction 2 Methods

[tamerakassie]

Added the notebook that compares the ABBA and the 2 channel Time Differencing Sky Subtraction Methods, also updated with the bandpass plots

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:22Z ----------------------------------------------------------------

I would add a Markdown cell above with a short summary about what this notebook does.

# Sky-subtraction Method Comparison
In this notebook, we ...

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:23Z ----------------------------------------------------------------

Line #1.    mkssins.good_ant(fname)

Commit and push this file to the PR branch

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:25Z ----------------------------------------------------------------

Line #3.    ants = mkssins.ants_checked_L1(fname, path)

I am not sure what this does.

Line #5.    for ant in ants:
Line #6.        print(ant[0])

I am just wondering why you have to do [0] . What is in the ant list?

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:26Z ----------------------------------------------------------------

These are just warnings. They are quite annoying. I am pretty sure there is a way to suppress them. I will help you with this.

tamerakassie commented on 2024-03-07T18:02:17Z ----------------------------------------------------------------

okay, let me know how we can do this

piyanatk commented on 2024-03-08T07:09:38Z ----------------------------------------------------------------

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:27Z ----------------------------------------------------------------

Line #8.            vis_products[i] = (vis)
Line #9.            nds0_products[i] =(nd_s0)

• I am guessing both vis and nd_s0 are already Numpy arrays, right? So you don't need the parenthesis around them. The dictionary value can be a numpy array.
• I would use raw_vis_dict and raw_vis instead of vis_products and vis. I think it is a bit more clear.
• Also, nd_flags_dict and nd_flags

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:28Z ----------------------------------------------------------------

Just a thought that masks and flags are using interchangeably here. In the original SSINS algorithm, masks are derived from SS spectrum, while flags are converted from masks to apply to raw data. They have different size on the time dimension because sky subtraction was done through neighboring channel subtraction, so you loose a few time channels. It is probably fine in our case, but I would try to be consistent in the code, so we don't confuse ourselves. Maybe either use "masks" or "flags" for all boolean arrays in your code.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:23Z ----------------------------------------------------------------

Line #1.    mkssins.good_ant(fname)

Commit and push this file to the PR branch

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:25Z ----------------------------------------------------------------

Line #3.    ants = mkssins.ants_checked_L1(fname, path)

I am not sure what this does.

Line #5.    for ant in ants:
Line #6.        print(ant[0])

I am just wondering why you have to do [0] . What is in the ant list?

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:26Z ----------------------------------------------------------------

These are just warnings. They are quite annoying. I am pretty sure there is a way to suppress them. I will help you with this.

tamerakassie commented on 2024-03-07T18:02:17Z ----------------------------------------------------------------

okay, let me know how we can do this

piyanatk commented on 2024-03-08T07:09:38Z ----------------------------------------------------------------

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:27Z ----------------------------------------------------------------

Line #8.            vis_products[i] = (vis)
Line #9.            nds0_products[i] =(nd_s0)

• I am guessing both vis and nd_s0 are already Numpy arrays, right? So you don't need the parenthesis around them. The dictionary value can be a numpy array.
• I would use raw_vis_dict and raw_vis instead of vis_products and vis. I think it is a bit more clear.
• Also, nd_flags_dict and nd_flags

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:28Z ----------------------------------------------------------------

Just a thought that masks and flags are using interchangeably here. In the original SSINS algorithm, masks are derived from SS spectrum, while flags are converted from masks to apply to raw data. They have different size on the time dimension because sky subtraction was done through neighboring channel subtraction, so you loose a few time channels. It is probably fine in our case, but I would try to be consistent in the code, so we don't confuse ourselves. Maybe either use "masks" or "flags" for all boolean arrays in your code.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:25Z ----------------------------------------------------------------

Line #3.    ants = mkssins.ants_checked_L1(fname, path)

I am not sure what this does.

Line #5.    for ant in ants:
Line #6.        print(ant[0])

I am just wondering why you have to do [0] . What is in the ant list?

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:26Z ----------------------------------------------------------------

These are just warnings. They are quite annoying. I am pretty sure there is a way to suppress them. I will help you with this.

tamerakassie commented on 2024-03-07T18:02:17Z ----------------------------------------------------------------

okay, let me know how we can do this

piyanatk commented on 2024-03-08T07:09:38Z ----------------------------------------------------------------

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:27Z ----------------------------------------------------------------

Line #8.            vis_products[i] = (vis)
Line #9.            nds0_products[i] =(nd_s0)

• I am guessing both vis and nd_s0 are already Numpy arrays, right? So you don't need the parenthesis around them. The dictionary value can be a numpy array.
• I would use raw_vis_dict and raw_vis instead of vis_products and vis. I think it is a bit more clear.
• Also, nd_flags_dict and nd_flags

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:28Z ----------------------------------------------------------------

Just a thought that masks and flags are using interchangeably here. In the original SSINS algorithm, masks are derived from SS spectrum, while flags are converted from masks to apply to raw data. They have different size on the time dimension because sky subtraction was done through neighboring channel subtraction, so you loose a few time channels. It is probably fine in our case, but I would try to be consistent in the code, so we don't confuse ourselves. Maybe either use "masks" or "flags" for all boolean arrays in your code.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:26Z ----------------------------------------------------------------

These are just warnings. They are quite annoying. I am pretty sure there is a way to suppress them. I will help you with this.

tamerakassie commented on 2024-03-07T18:02:17Z ----------------------------------------------------------------

okay, let me know how we can do this

piyanatk commented on 2024-03-08T07:09:38Z ----------------------------------------------------------------

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:27Z ----------------------------------------------------------------

Line #8.            vis_products[i] = (vis)
Line #9.            nds0_products[i] =(nd_s0)

• I am guessing both vis and nd_s0 are already Numpy arrays, right? So you don't need the parenthesis around them. The dictionary value can be a numpy array.
• I would use raw_vis_dict and raw_vis instead of vis_products and vis. I think it is a bit more clear.
• Also, nd_flags_dict and nd_flags

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:28Z ----------------------------------------------------------------

Just a thought that masks and flags are using interchangeably here. In the original SSINS algorithm, masks are derived from SS spectrum, while flags are converted from masks to apply to raw data. They have different size on the time dimension because sky subtraction was done through neighboring channel subtraction, so you loose a few time channels. It is probably fine in our case, but I would try to be consistent in the code, so we don't confuse ourselves. Maybe either use "masks" or "flags" for all boolean arrays in your code.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:27Z ----------------------------------------------------------------

Line #8.            vis_products[i] = (vis)
Line #9.            nds0_products[i] =(nd_s0)

• I am guessing both vis and nd_s0 are already Numpy arrays, right? So you don't need the parenthesis around them. The dictionary value can be a numpy array.
• I would use raw_vis_dict and raw_vis instead of vis_products and vis. I think it is a bit more clear.
• Also, nd_flags_dict and nd_flags

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:28Z ----------------------------------------------------------------

Just a thought that masks and flags are using interchangeably here. In the original SSINS algorithm, masks are derived from SS spectrum, while flags are converted from masks to apply to raw data. They have different size on the time dimension because sky subtraction was done through neighboring channel subtraction, so you loose a few time channels. It is probably fine in our case, but I would try to be consistent in the code, so we don't confuse ourselves. Maybe either use "masks" or "flags" for all boolean arrays in your code.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:28Z ----------------------------------------------------------------

Just a thought that masks and flags are using interchangeably here. In the original SSINS algorithm, masks are derived from SS spectrum, while flags are converted from masks to apply to raw data. They have different size on the time dimension because sky subtraction was done through neighboring channel subtraction, so you loose a few time channels. It is probably fine in our case, but I would try to be consistent in the code, so we don't confuse ourselves. Maybe either use "masks" or "flags" for all boolean arrays in your code.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:29Z ----------------------------------------------------------------

I would define

mask_dir = '/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'

Better

mask_dir = Path('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/')

Then, in your try-except, you can do

mask_dir / f'{fname}_{dish}_mask2'

The / work as a path "joiner" for path lib, and you won't have to worry about missing or having extra / .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:30Z ----------------------------------------------------------------

Line #9.            d3 = pickle.load(open('/idia/projects/hi_im/raw_vis/MeerKLASS2021/level1/mask/checked/'+fname+'_'+dish+'_mask', 'rb'))

You should also use open within with block, so the file is automatically "close" (so other people/process can access it). See an example here https://docs.python.org/3/library/functions.html#open

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:31Z ----------------------------------------------------------------

See above comment about "flags" and "masks". Maybe just call it masks_L1 or flags_L1 .

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:32Z ----------------------------------------------------------------

I am guessing masked_vis is a Numpy masked array with L1 and nd flags applied, and masked_data is a dictionary of that with antenna as keys, right? In that case, I would call the latter masked_vis_dict

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:33Z ----------------------------------------------------------------

• This would be a good point to plot a waterfall of the raw visibility with and without existing flags applied to show.
• You are doing a lot of array printings to check if things are working. This is good, but once you are sure that your notebook work, you should remove the unnecessary printing lines to tidy things up a bit.

tamerakassie commented on 2024-03-07T18:17:28Z ----------------------------------------------------------------

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

piyanatk commented on 2024-03-08T07:07:21Z ----------------------------------------------------------------

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:34Z ----------------------------------------------------------------

Make sure your plot is labelled.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:35Z ----------------------------------------------------------------

Seeing a peak at frequency channel 1200 in the all-antenna-averaged spectrum, I am thinking that not all antennas are being flagged at this frequency channel, so when you averaged over antennas, it produce a peak. Plotting other antenans should gives us some hints. Although, are you averaging the waterfall or the bandpass? I would do the former before calculating the bandpass.

review-notebook-app[bot] commented 6 months ago

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

piyanatk commented on 2024-03-06T09:44:36Z ----------------------------------------------------------------

Just looking at this, the noise RMS (the fluctuations) does go down after averaging over antennas. It would be a good check if antenna averaging works as expected by looking at the RMS number

tamerakassie commented 6 months ago

okay, let me know how we can do this

---

View entire conversation on ReviewNB

tamerakassie commented 6 months ago

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

piyanatk commented 6 months ago

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

piyanatk commented 6 months ago

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[tamerakassie]

okay, let me know how we can do this

---

View entire conversation on ReviewNB

[tamerakassie]

just a question, will it be a good idea to plot all receiver waterfalls ? or just for a few?

---

View entire conversation on ReviewNB

[piyanatk]

All receivers would be nice, but it might be too many plots. You will probably have to combine them into one big multi-panel plots, say 13 rows x 5 columns. Something like

# Normalisation for all panels
norm = LogNorm(min= ..., max= ...)

ncols = 5
fig, axes = plt.subplots(13, ncols, figsize=(15, 40), sharex='all', sharey='all)
# Loop over antennas, plotting with plot_waterfall
for i, ant in enumerate(axes.flat):
    # Check if i is in good antenna list. If so, plot. Otherwise, left the subplot empty
    if i in good_ants_int:
        plot_waterfall(data, ax=ax)
fig.colorbar(axes)

--- View entire conversation on ReviewNB

[piyanatk]

https://stackoverflow.com/questions/14463277/how-to-disable-python-warnings/14463362#14463362

---

View entire conversation on ReviewNB