Output differences compared to provided example data

[EliT1626]

This is NOT a bug issue, but I didn't know where else to post something like this. I was wondering if anyone else using this AMAZING Python program had run into issues with data output differing, sometimes significantly, from the provided example data.

I am attaching two files, both of which are PNGs of PDSI plots from August of 2012. In the plot where I used my own data (plot 1) there is a noticeably high amount of areas where the PDSI values are much lower than in the provided example data (plot 2). These sorts of issues persist during other times as well. I was curious if this could be related to something on my end, or if the program inherently struggles with higher resolution data.

The data used in this case is almost exclusively 0.25 degree monthly ERA reanalysis data. The PET data was created from 0.25 degree monthly temperature data. The precip and AWC data are also 0.25 degree monthly data.

Another question I had was does the calibration period matter for the regular PDSI calculations? If I only run the program for 10 years worth of data will I get different output data compared to 40 years of data? Thanks.

[monocongo]

Thanks for bringing this up, @EliT1626

Alas things have gone a little of the rails in recent months (my day job has overwhelmed me, and this project is essentially a one-man show/labor of love) -- the example results in climate-indices-examples are likely out of sync.

The Palmer provided in this package is experimental -- I did my best to provide a Python translation of the various Palmers in use at NOAA but there are some discrepancies that were never ironed out and the few experts in this field who can make decisions on the scientific validity never had the time to fully vet the results of that code. My intention is to better document this so people won't use the Palmers (PDSI, PHDI, etc.) for proper climate science work or papers, as well as to excise the NCO parts since NCO doesn't work as advertised on Windows, (this is 80% of the issues reported).

I'm not sure if the above gives much help, but hopefully some insight into where things stand. Sorry I can't be of more immediate assistance!

[EliT1626]

Thanks for the super quick response. I appreciate the honesty and effort you have put into this. Do you have any suggestions for any alternatives at this time? Fixing/rewriting something like you have developed is outside of my plans, but am still looking for some quasi-realtime PDSI measurements.

[Amarounek]

@EliT1626 Well I understand you... Recently, I was calculating SPI from AgERA5 and it seems to me that something is wrong.

I had a lot of memory issues during merging files and later I've discovered that some AgERA5 files are corrupted (I think years 2015 - 2018).
Copernicus wrote me after some correspodence this:

Dear ERA5 user, We are writing to you because you may have downloaded some corrupted ERA5 data from the Climate Data Store (CDS). Some data was corrupted in the following ERA5 CDS catalogue entries (datasets): ERA5 hourly data on single levels from 1979 to present ERA5 hourly data on pressure levels from 1979 to present 361 corrupted fields, out of a total of 3.1 billion ERA5 fields, have been identified and replaced. This corruption problem was resolved on 2021-04-14. Users that have downloaded any of these affected fields prior to 2021-04-15, are recommended to re-download them. For reference, this issue has been recorded as a "Resolved issue" in the ERA5: data documentation. We apologise for any inconvenience this may have caused you. If you have any questions, you may contact us.

So there can be some issues. I was also checking the differences between AgERA5 SPI results and SPI results originated from IMERG and there were definitely some huge differences (however in times when it was broken).