Test -1.1.0

[lwhitler]

This notebook validates RIMEz against the first generation of pyuvsim reference simulations. Test criteria are still TBD and it's still being updated somewhat. Specifically, there is evidence for source position errors from RIMEz, but there may be some assumptions we missed so we are following that up. All paths are also on the ASU cluster (Enterprise) and I am in the process of moving everything to Lustre. However, the general structure is there.

[review-notebook-app[bot]]

Check out this pull request on 

Review Jupyter notebook visual diffs & provide feedback on notebooks.

---

Powered by ReviewNB

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:23Z ----------------------------------------------------------------

Obviously need to fill out the CRITERIA.

I'm thinking it could be something like:

1. Amplitude errors on baselines within the H1C configuration, and within the H1C frequency range of 100-200 MHz, and for the Gaussian, Airy and HERA beams, to be < 1%, except for the uniform beam when sources are close to the horizon (as this is not realistic).
2. Phase errors (under the same restrictions) indicative of source-position errors of < 15' (the approximation resolution of H1C). This is relevant for power spectra -- calibration in general requires higher accuracy, but for H1C Validation, calibration is performed with sky models from RIMEz, negating this requirement.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:24Z ----------------------------------------------------------------

I think we can say it has passed, if you go back to the positions before converting to CIRS.

I think you can also suggest a follow-up action that it would be useful for pyuvsim (and simulators in general) to be able to output source alt/az for each of the simulation times, to make for simpler comparisons.

Also mention that the source position errors found here are fine for power spectra in the validation effort, but will potentially lead to calibration errors if used for that purpose in future IDRs.

Also mention that too-low lmax primarily affects uniform beam (because of the inherent step function in time). Setting it higher resolves problems, but also increases compute time significantly.

lwhitler commented on 2020-06-29T18:43:41Z ----------------------------------------------------------------

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

steven-murray commented on 2020-06-30T14:07:21Z ----------------------------------------------------------------

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

steven-murray commented on 2020-07-06T16:54:40Z ----------------------------------------------------------------

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:24Z ----------------------------------------------------------------

You are implicitly using pyuvdata and h5py, so I'd keep them in the versions.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:25Z ----------------------------------------------------------------

I think the prepare_rimez_data.py file should go in the repo itself alongside this notebook.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

I'd put all .py files here in the repo.

steven-murray commented on 2020-07-06T16:58:03Z ----------------------------------------------------------------

Looks like this has been done.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA).

steven-murray commented on 2020-07-06T16:58:33Z ----------------------------------------------------------------

Great!

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:24Z ----------------------------------------------------------------

I think we can say it has passed, if you go back to the positions before converting to CIRS.

I think you can also suggest a follow-up action that it would be useful for pyuvsim (and simulators in general) to be able to output source alt/az for each of the simulation times, to make for simpler comparisons.

Also mention that the source position errors found here are fine for power spectra in the validation effort, but will potentially lead to calibration errors if used for that purpose in future IDRs.

Also mention that too-low lmax primarily affects uniform beam (because of the inherent step function in time). Setting it higher resolves problems, but also increases compute time significantly.

lwhitler commented on 2020-06-29T18:43:41Z ----------------------------------------------------------------

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

steven-murray commented on 2020-06-30T14:07:21Z ----------------------------------------------------------------

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

steven-murray commented on 2020-07-06T16:54:40Z ----------------------------------------------------------------

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:24Z ----------------------------------------------------------------

You are implicitly using pyuvdata and h5py, so I'd keep them in the versions.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:25Z ----------------------------------------------------------------

I think the prepare_rimez_data.py file should go in the repo itself alongside this notebook.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

I'd put all .py files here in the repo.

steven-murray commented on 2020-07-06T16:58:03Z ----------------------------------------------------------------

Looks like this has been done.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA).

steven-murray commented on 2020-07-06T16:58:33Z ----------------------------------------------------------------

Great!

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:24Z ----------------------------------------------------------------

You are implicitly using pyuvdata and h5py, so I'd keep them in the versions.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:25Z ----------------------------------------------------------------

I think the prepare_rimez_data.py file should go in the repo itself alongside this notebook.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

I'd put all .py files here in the repo.

steven-murray commented on 2020-07-06T16:58:03Z ----------------------------------------------------------------

Looks like this has been done.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA).

steven-murray commented on 2020-07-06T16:58:33Z ----------------------------------------------------------------

Great!

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:25Z ----------------------------------------------------------------

I think the prepare_rimez_data.py file should go in the repo itself alongside this notebook.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

I'd put all .py files here in the repo.

steven-murray commented on 2020-07-06T16:58:03Z ----------------------------------------------------------------

Looks like this has been done.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA).

steven-murray commented on 2020-07-06T16:58:33Z ----------------------------------------------------------------

Great!

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

I'd put all .py files here in the repo.

steven-murray commented on 2020-07-06T16:58:03Z ----------------------------------------------------------------

Looks like this has been done.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA).

steven-murray commented on 2020-07-06T16:58:33Z ----------------------------------------------------------------

Great!

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:26Z ----------------------------------------------------------------

Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA).

steven-murray commented on 2020-07-06T16:58:33Z ----------------------------------------------------------------

Great!

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:27Z ----------------------------------------------------------------

Can probably suggest why: pyuvsim is doing a single calculation per-source (just a factor multiplied by an exponential). RIMEz is doing a whole FFT essentially, which means its result is at ~machine precision.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:28Z ----------------------------------------------------------------

Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation?

lwhitler commented on 2020-07-03T16:59:31Z ----------------------------------------------------------------

Not actually sure. Possibly?

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:29Z ----------------------------------------------------------------

I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors).

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:30Z ----------------------------------------------------------------

Probably should say what the sim is here (uniform beam)

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-06-29T14:38:31Z ----------------------------------------------------------------

... resulting in a cutoff at the frequency at which the sources are not adequately resolved.

lwhitler commented 4 years ago

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

lwhitler commented 4 years ago

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Looks like this has been done.

---

View entire conversation on ReviewNB

steven-murray commented 4 years ago

Great!

---

View entire conversation on ReviewNB

review-notebook-app[bot] commented 4 years ago

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[lwhitler]

Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)?

---

View entire conversation on ReviewNB

[steven-murray]

I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later).

---

View entire conversation on ReviewNB

[lwhitler]

Not actually sure. Possibly?

---

View entire conversation on ReviewNB

[steven-murray]

OK, so this has been resolved -- apparently CIRS doesn't work for this version of RIMEz and we should definitely go back.

---

View entire conversation on ReviewNB

[steven-murray]

Looks like this has been done.

---

View entire conversation on ReviewNB

[steven-murray]

Great!

---

View entire conversation on ReviewNB

[review-notebook-app[bot]]

View / edit / reply to this conversation on ReviewNB

steven-murray commented on 2020-07-06T17:08:22Z ----------------------------------------------------------------

I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time.

zacharymartinot commented 3 years ago

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

zacharymartinot commented 3 years ago

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[zacharymartinot]

Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling.

As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook.

Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference.

The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done.

[zacharymartinot]

Better/additional notes:

• The beam definitions input to RIMEz are not the same as the ones input to pyuvsim, except for the uniform beam.
• The harmonic method is a priori unsuited to calculating the result of a uniform beam

to be continued...