AttributeError: 'NoneType' object has no attribute 'data'

[AKHughes1994]

Hello,

I'm using a .yaml file to try and peel a bright source out of a FOV. The routine is based off OxKAT which uses CC to perform the peeling so, I'm just trying to transfer it over to QC. I've included my .yaml file (which was based on an example provided to me in the CC discussion board and was meant to mimic the OxKAT CC parset), and log report.

The process fails with the a common but uninformative error, I've plotted all of the relevant data columns (i.e., DATA, CORRECTED_DATA, MODEL_DATA, DIR1_DATA) and they all exist and are non-zero so I'm not sure why it's failing.

Any advice?

Cheers, qc_log.txt DD_bb.yaml.txt

Andrew

[IanHeywood]

Looks like something weird is going on here, seems to have a non-existent entry in the model list:

    in_b = [xds.get(ingredients[op_idx + 1]).data]
            │   │   │           └ 0
            │   │   └ ['MODEL_DATA', '', 'DIR1_DATA']
            │   └ <function Mapping.get at 0x7f654ea47790>
            └ <xarray.Dataset>
              Dimensions:     (row: 41769, chan: 64, uvw: 3, corr: 4, ant: 27)
              Coordinates:
                  ROWID       (row) int32 da...

AttributeError: 'NoneType' object has no attribute 'data'

CubiCal requires the +- syntax here:

MODEL_DATA+~DIR1_DATA:DIR1_DATA

but I think for QuartiCal it just needs ~ to do subtraction, so I'd try that.

[AKHughes1994]

@IanHeywood That works to prevent the error. Let me expand upon the question because the .yaml script isn't working as expected.

I'm trying to peel this bright source (following the routine from Oxkat). (I'm assuming) the dE calibration is working well for removing the side lobes of the bright source. You can see that the DI self-cal: Has residual sidelobes that DD self-cal corrects: Now, this is good (great even). But I'm trying to figure out why it's not removing the source (considering that the script peels out the DIR1 source with CC as a part of the Oxkat routine). Any insight?

Point of clarification as well, the input_model.recipe=MODEL_DATA~DIR1_DATA:DIR1_DATA where MODEL_DATA contains the full sky model and DIR1_DATA contains the model of only the problem source. When I set subtract_directions: [1] does the [1] denote MODEL_DATA~DIR1_DATA (i.e. indexing starting from 1), or does it correspond DIR1_DATA?

[bennahugo]

It is the same 0 indexing as in cubical. This looks to me like undersubtraction. Can you verify the fluxe before and after?

On Wed, Sep 6, 2023, 00:12 AKHughes1994 @.***> wrote:

@IanHeywood https://github.com/IanHeywood That works to prevent the error. Let me expand upon the question because the .yaml script isn't working as expected.

I'm trying to peel this bright source (following the routine from Oxkat). (I'm assuming) the dE calibration is working well for removing the side lobes of the bright source. You can see that the DI self-cal: [image: image] https://user-images.githubusercontent.com/49698839/265836501-6f69c5f3-0ab4-41b4-8ee6-42e7a0f3d0a8.png Has residual sidelobes that DD self-cal corrects: [image: image] https://user-images.githubusercontent.com/49698839/265836630-78cdb474-5eab-494a-bf9c-70897105ac51.png Now, this is good (great even). But I'm trying to figure out why it's not removing the source (considering that is the result with CC and Oxkat). Any insight?

Point of clarification as well, the input_model.recipe=MODEL_DATA~DIR1_DATA:DIR1_DATA where MODEL_DATA contains the full sky model and DIR1_DATA contains the model of only the problem source. When I set subtract_directions: [1] does the [1] denote MODEL_DATA~DIR1_DATA (i.e. indexing starting from 1), or does it correspond DIR1_DATA?

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/QuartiCal/issues/295#issuecomment-1707372508, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6RKIPGX2JEKRMSTSKTXY6PTLANCNFSM6AAAAAA4H6CEPE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[JSKenyon]

As @bennahugo said, the models are zero indexed. As for some part of the source being left in the map, it is not unusual. If you made your solution intervals smaller, I would expect the subtraction to improve. If you have an equivalent result/config with CubiCal, would you mind posting it here? It is important to note that QC and CC are different in many ways so it isn't easy to isolate a specific cause for a change in the results. I am happy to take a look though. One thing I spotted is that you are using complex terms for both G and dE. This isn't a problem per se, but it means that in the absence of a polarized model, there may be a danger of moving polarized flux into stokes I.

[AKHughes1994]

@bennahugo was correct, and it was me misreading. The source went from 1.04e-1 -> 1.04e-3 (which is similar to what was happening in CC). I was misreading the exponent, and interpreting is as reducing the side lobes without subtracting any emission from the bright source.

Can I ask a more fundamental question before closing this issue, What is the difference between corrected_residual and corrected_data, and when do we want to use one or the other? I see the examples in the documentation use both of them.

[bennahugo]

Corrected data can only be formed from the direction independent gains, whereas corrected residuals are formed with dd corrections. It is not possible, in visibility space, to truely apply direction dependent solutions, without resorting to rending restored sources into place (see e.g. tigger-restore). However this is just painting and is not really physically meaningful.

If you need corrected images of the source with residuals I think the only way at present is the ddfacet/killms route.

On Wed, Sep 6, 2023, 21:51 AKHughes1994 @.***> wrote:

@bennahugo https://github.com/bennahugo was correct, and it was me misreading. The source went from 1.04e-1 -> 1.04e-3 (which is similar to what was happening in CC). I was misreading the exponent.

Can I ask a more fundamental question before closing this issue, What is the difference between corrected_residual and corrected_data, and when do we want to use one or the other? I see the examples in the documentation use both of them.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/QuartiCal/issues/295#issuecomment-1709006267, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6WHFNI73ZOYKZ65CE3XZDH4RANCNFSM6AAAAAA4H6CEPE . You are receiving this because you were mentioned.Message ID: @.***>

[o-smirnov]

corrected_data is the original data, with the inverse of DI gain terms applied. Normally you'd use that before you do any DD solutions.

corrected_residual is data minus DI-and-DD-corrupted [subset of] model, with the inverse of DI gain terms applied. The subtract directions parameter controls which subset is subtracted. So that's what you would use if you want to peel away some source(s).

[JSKenyon]

Hope that @o-smirnov and @bennahugo managed to clear that up for you @AKHughes1994. If you need any further assistance, just let me/us know.

[JSKenyon]

@AKHughes1994 can we close this issue?

[AKHughes1994]

Yes sorry! I should have closed it.