Filling missing data

[jddingemanse]

Automizing different filling missing data techniques.

[jddingemanse]

@IsraelGebresilassie , you have uploaded the code file with an example of three filling missing data techniques (Estimation of Missing Data.py). When I run the example code of the Inverse distance method (IDW), I get, for example, for Parb values of 0.35-1.00. If I understand it correctly, these values should be used to fill missing data for the Arba Minch column.

Possible wrong IDW calculation Can you double check the mathematics of IDW and how they are implemented in the code, because I think the values should be higher. At this moment, for example, based on Mirab (8.0), Chano (10) and Shara (9), together with distances to Arba Minch of 10, 12 and 14, respectively, gives an estimation of 0.767. That would seem incorrect.

[IsraelGebresilasieKimo]

Ok, I will check.

With Regards,

---

Israel Gebresilasie (MSc in Climate Change and Development)

Lecturer, Faculty of Meteorology and Hydrology, Water Technology Institute, Arba Minch University

*Arba Minch Ethiopia *** Phone: +251 927833912*

*Alternate email: @. @.>*

On Thu, Dec 29, 2022 at 12:19 AM jddingemanse @.***> wrote:

@IsraelGebresilassie https://github.com/IsraelGebresilassie , you have uploaded the code file with an example of three filling missing data techniques (Estimation of Missing Data.py https://github.com/jddingemanse/awtiCodeDev/blob/904e7fa0548bf6b1ad8546262e5d634ed0c72639/code/Estimation%20of%20Missing%20%20Data.py). When I run the example code of the Inverse distance method (IDW), I get, for example, for Parb values of 0.35-1.00. If I understand it correctly, these values should be used to fill missing data for the Arba Minch column.

Possible wrong IDW calculation Can you double check the mathematics of IDW and how they are implemented in the code, because I think the values should be higher. At this moment, for example, based on Mirab (8.0), Chano (10) and Shara (9), together with distances to Arba Minch of 10, 12 and 14, respectively, gives an estimation of 0.767. That would seem incorrect.

— Reply to this email directly, view it on GitHub https://github.com/jddingemanse/awtiCodeDev/issues/8#issuecomment-1367148048, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZN2QS5AFZKR743OHF5YHIDWPVCS5ANCNFSM6AAAAAATDKJKYA . You are receiving this because you were mentioned.Message ID: @.***>

[IsraelGebresilasieKimo]

here is the calculation (8/10^2)+(10/12^2)+(9/14^2) for numerator = 0.195 approximately (1/10+1/12+1/14) for Dominator = 0.255 approximately then 0.195/0.255= 0.765 approximately

[jddingemanse]

here is the calculation (8/10^2)+(10/12^2)+(9/14^2) for numerator = 0.195 approximately (1/10+1/12+1/14) for Dominator = 0.255 approximately then 0.195/0.255= 0.765 approximately

Good that you checked. But now about the implication; it feels incorrect that values 8, 9 and 10 result in an estimated 0.765. It should be somewhere between 8 and 10, with the station closest to Arba Minch having the most weight in the eventual outcome.

Is the 0.765 maybe a step in between, that is needed to calculate the final actual precipitation estimation?

• Can you share the full equations such as you also showed in your slides?

edit: found online equations

I found below equations, are they correct?

Based on that the numerator and dominator should be:

• (1/10)8+(1/12)10+(1/14)*9 = 2.28
• (1/10) + (1/12) + (1/14) = 0.255

This results in 2.28/0.255 = 8.9. That sounds more logical. If that is correct, then I suggest the following code change for the IDW part: current code

# a code to calculate missing data via inverse distance method
Parb=(Data.Mirab.replace(np.nan,0)/(DAM)**2+Data.Chano.replace(np.nan,0)/(DAC)**2+Data.Shara.replace(np.nan,0)/(DAS)**2)/(1/DAM+1/DAC+1/DAS)
Pmirab=( Data.Arbaminch.replace(np.nan,0)/(DAM)**2+Data.Chano.replace(np.nan,0)/(DCM)**2+Data.Shara.replace(np.nan,0)/(DMS)**2)/(1/DAM+1/DCM+1/DMS)
PCHANO=( Data.Arbaminch.replace(np.nan,0)/(DAC)**2+Data.Mirab.replace(np.nan,0)/(DCM)**2+Data.Shara.replace(np.nan,0)/(DCS)**2)/(1/DAC+1/DCM+1/DCS)
PSHARA=(Data.Arbaminch.replace(np.nan,0)/(DAS)**2+Data.Mirab.replace(np.nan,0)/(DMS)**2+Data.Chano.replace(np.nan,0)/(DCS)**2)/(1/DAS+1/DMS+1/DCS)

proposed new code

Parb=(Data.Mirab.replace(np.nan,0)/(DAM)+Data.Chano.replace(np.nan,0)/(DAC)+Data.Shara.replace(np.nan,0)/(DAS))/(1/DAM+1/DAC+1/DAS)
Pmirab=( Data.Arbaminch.replace(np.nan,0)/(DAM)+Data.Chano.replace(np.nan,0)/(DCM)+Data.Shara.replace(np.nan,0)/(DMS))/(1/DAM+1/DCM+1/DMS)
PCHANO=( Data.Arbaminch.replace(np.nan,0)/(DAC)+Data.Mirab.replace(np.nan,0)/(DCM)+Data.Shara.replace(np.nan,0)/(DCS))/(1/DAC+1/DCM+1/DCS)
PSHARA=(Data.Arbaminch.replace(np.nan,0)/(DAS)+Data.Mirab.replace(np.nan,0)/(DMS)+Data.Chano.replace(np.nan,0)/(DCS))/(1/DAS+1/DMS+1/DCS)

[IsraelGebresilasieKimo]

Thank you Jan for your interesting suggestion based on the real interpretations but before applying the change we have to discuses. her is my formula and let me check it again.

[jddingemanse]

Thank you Jan for your interesting suggestion based on the real interpretations but before applying the change we have to discuses. her is my formula and let me check it again.

Ah, if that is the formula, then, of the following ...

(8/10^2)+(10/12^2)+(9/14^2) for numerator = 0.195 approximately (1/10+1/12+1/14) for Dominator = 0.255 approximately

the calculation of the denominator is incorrect. It should be instead:

• 1/(10^2) + 1/(12^2) + 1/(14^2) = 0.015

Also, recalculating the numerator leads to 0.126 So final answer would be 0.126/0.015=8.4. That answer sounds quite reasonable.

Then, I propose the following code:

Parb=(Data.Mirab.replace(np.nan,0)/(DAM**2)+Data.Chano.replace(np.nan,0)/(DAC**2)+Data.Shara.replace(np.nan,0)/(DAS**2))/(1/(DAM**2)+1/(DAC**2)+1/(DAS**2))
Pmirab=( Data.Arbaminch.replace(np.nan,0)/(DAM**2)+Data.Chano.replace(np.nan,0)/(DCM**2)+Data.Shara.replace(np.nan,0)/(DMS**2))/(1/(DAM**2)+1/(DCM**2)+1/(DMS**2))
PCHANO=( Data.Arbaminch.replace(np.nan,0)/(DAC**2)+Data.Mirab.replace(np.nan,0)/(DCM**2)+Data.Shara.replace(np.nan,0)/(DCS**2))/(1/(DAC**2)+1/(DCM**2)+1/(DCS**2))
PSHARA=(Data.Arbaminch.replace(np.nan,0)/(DAS**2)+Data.Mirab.replace(np.nan,0)/(DMS**2)+Data.Chano.replace(np.nan,0)/(DCS**2))/(1/(DAS**2)+1/(DMS**2)+1/(DCS**2))

[IsraelGebresilasieKimo]

Thank you so much, Jan but in your calculation, I think there is typing error for the numerator and you are correct for the denominator I forgot the power of two (squaring). for the rest ok for me thank you again for your correction. keep going.....

[jddingemanse]

Thank you so much, Jan but in your calculation, I think there is typing error for the numerator and you are correct for the denominator I forgot the power of two (squaring). for the rest ok for me thank you again for your correction. keep going.....

Indeed, I made a mistake, even for some reason in both the numerator and the denominator. To fully say it correctly:

• 8/(10^2)+10/(12^2)+9/(14^2) = 0.195
• 1/(10^2)+1/(12^2)+1/(14^2) = 0.022
• 0.195/0.022 = 8.86

After you have changed your code file, I will close this issue. Any other items related to filling missing data we can do under a discussion, or we can create a new and more specific issue for that.

[IsraelGebresilasieKimo]

Now the code is modified.

[jddingemanse]

Perfect; I hereby close the issue. If there is any remark regarding filling missing data code, please start a discussion, or open a new issue.