Problems with changes in PCA for adimsdi between vip 9.7 and 9.11

[maddalenareggiani]

I am using PCA in ADI+SDI mode (both "single" and "double" mode) with VIP 9.11. First of all, I am getting different results with PCA adimsdi from what I used to get with VIP 9.7, even if using the same dataset and parameter choice. Although I am using the same number of ncomp as before, the final PCA images look sharper, but there are also many negative values that weren't there before. This doesn't happen when using simple ADI.

In addition, I am getting the following warnings when running PCA in adimsdi mode: -python3.7-scikit-learn-0.21.2/lib/python3.7/site-packages/sklearn/preprocessing/data.py:172: UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features. warnings.warn("Numerical issues were encountered " -python3.7-scikit-learn-0.21.2/lib/python3.7/site-packages/sklearn/preprocessing/data.py:189: UserWarning: Numerical issues were encountered when scaling the data and might not be solved. The standard deviation of the data is probably very close to 0. warnings.warn("Numerical issues were encountered " I am not sure if this is what is causing the problem.

Finally, the outcome of the new pca is also affecting the results of the contrast curve function. As attachment you can find the contrast curves I obtained for the same dataset and same parameters for vip9.7 and vip9.11. The curves obtained with vip9.11 always show the same behaviour, regardless of the dataset I use.

Therefore, I am now wondering which one is the correct outcome of pca (and version of pca to use), and, if it is the one from vip 9.11, whether there is anything else that has been changed in vip_hci.pca.pca that is breaking the contrast curve function.

Thanks! Maddalena

[carlos-gg]

Hi Maddalena, sorry for the late reply. I have started a new job (nothing to do with HCI) and the past few weeks have been hectic.

The question "what is the correct output" doesn't have a straightforward answer. The ADI+SDI mode for PCA was developed while I was in Grenoble/IPAG. These functionalities of VIP are mostly undocumented. During development, the validation was done on a couple of SPHERE IFS datasets and as far as I remember, there weren't significant changes in the output that might explain the "broken contrast curve" with VIP 9.11.

The error/warning from Scikit-learn, related to the pixel scaling (temporal/spatial mean or standardization), might give us a clue about what happened, but might be that with a newer version of sklearn (used in VIP 9.11), new warnings and verbosity was introduced.

Could you please tell me:

• Is this only happening with PCA in admsdi mode? First thing, we want to make sure the contrast curve works for other algos and PCA modes.
• Could you share the outputs of the PCA (in admsdi mode and with the same parameter choice) for the two versions of VIP?

That will help debug this issue. Thanks.

[maddalenareggiani]

Hi Carlos,

good luck with the new job! I hope you enjoy the new place, too.

The question "what is the correct output" doesn't have a straightforward answer. The ADI+SDI mode for PCA was developed while I was in Grenoble/IPAG. These functionalities of VIP are mostly undocumented. During development, the validation was done on a couple of SPHERE IFS datasets and as far as I remember, there weren't significant changes in the output that might explain the "broken contrast curve" with VIP 9.11.

The error/warning from Scikit-learn, related to the pixel scaling (temporal/spatial mean or standardization), might give us a clue about what happened, but might be that with a newer version of sklearn (used in VIP 9.11), new warnings and verbosity was introduced.

Could you please tell me:

Is this only happening with PCA in admsdi mode? First thing, we want to make sure the contrast curve works for other algos and PCA modes. Yes, it is only happening with PCA in admsdi mode. And from what I have tried, it is really the PCA function that is returning different results, not the contrast curve function itself.

Could you share the outputs of the PCA (in admsdi mode and with the same parameter choice) for the two versions of VIP?

Attached you can find the two PCA outcomes with VIP 9.07 (test_pca_py27) and with VIP 9.11 (test_pca_py3).

Let me know if you need anything else.

Thanks a lot, Maddalena

That will help debug this issue. Thanks.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/vortex-exoplanet/VIP/issues/412?email_source=notifications&email_token=ADMAVB3CWXO4CFEQYLP5UPTQSO5HDA5CNFSM4JEUIRLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDLKTBQ#issuecomment-550939014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMAVB5LNO4C4OLRB34HJXDQSO5HDANCNFSM4JEUIRLA.

[carlos-gg]

Hi Maddalena, thanks. It seems the plots didn't get attached to your message. Might have happened because you replied via email?

[VChristiaens]

Closing the issue as this should now be solved in pull request #421