Error using precip variability parameter file for TRMM day

acordonez commented 1 year ago

I attempted to use this parameter file to generate metrics for the daily TRMM data. I only changed the pmpdir variable. The precip variability driver threw the following error:

(pcmdi_metrics_3.1) -bash-4.2$ variability_across_timescales_PS_driver.py -p variability_across_timescales_PS_day_params_TRMM.py /p/user_pub/PCMDIobs/obs4MIPs/NASA-GSFC/TRMM-3B42v-7/day/pr/1x1/latest *.nc [2001, 2019] 3650 1825 ~/pmp_results/graphics/precip/obs/v20230718/variability_across_timescales ~/pmp_results/diagnostic_results/precip/obs/v20230718/variability_across_timescales ~/pmp_results/metrics_results/precip/obs/v20230718/variability_across_timescales TRMM-3B42v-7 ['/p/user_pub/PCMDIobs/obs4MIPs/NASA-GSFC/TRMM-3B42v-7/day/pr/1x1/latest/pr_day_TRMM-3B42v-7_PCMDIFROGS_1x1_19980101-20191230.nc'] TRMM-3B42v-7 gregorian syr, eyr: 2001 2019 2001 Complete regridding from (365, 180, 360) to (365, 90, 180) 2001 (365, 90, 180) 2002 Complete regridding from (365, 180, 360) to (365, 90, 180) 2002 (730, 90, 180) 2003 Complete regridding from (365, 180, 360) to (365, 90, 180) 2003 (1095, 90, 180) 2004 Complete regridding from (366, 180, 360) to (366, 90, 180) 2004 (1461, 90, 180) 2005 Complete regridding from (365, 180, 360) to (365, 90, 180) 2005 (1826, 90, 180) 2006 Complete regridding from (365, 180, 360) to (365, 90, 180) 2006 (2191, 90, 180) 2007 Complete regridding from (365, 180, 360) to (365, 90, 180) 2007 (2556, 90, 180) 2008 Complete regridding from (366, 180, 360) to (366, 90, 180) 2008 (2922, 90, 180) 2009 Complete regridding from (365, 180, 360) to (365, 90, 180) 2009 (3287, 90, 180) 2010 Complete regridding from (365, 180, 360) to (365, 90, 180) 2010 (3652, 90, 180) 2011 Complete regridding from (365, 180, 360) to (365, 90, 180) 2011 (4017, 90, 180) 2012 Complete regridding from (366, 180, 360) to (366, 90, 180) 2012 (4383, 90, 180) 2013 Complete regridding from (365, 180, 360) to (365, 90, 180) 2013 (4748, 90, 180) 2014 Complete regridding from (365, 180, 360) to (365, 90, 180) 2014 (5113, 90, 180) 2015 Complete regridding from (365, 180, 360) to (365, 90, 180) 2015 (5478, 90, 180) 2016 Complete regridding from (366, 180, 360) to (366, 90, 180) 2016 (5844, 90, 180) 2017 Complete regridding from (365, 180, 360) to (365, 90, 180) 2017 (6209, 90, 180) 2018 Complete regridding from (365, 180, 360) to (365, 90, 180) 2018 (6574, 90, 180) 2019 Complete regridding from (364, 180, 360) to (364, 90, 180) 2019 (6938, 90, 180) Traceback (most recent call last): File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_3.1/bin/variability_across_timescales_PS_driver.py", line 59, in <module> precip_variability_across_timescale( File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_3.1/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 57, in precip_variability_across_timescale clim, anom = ClimAnom(drg, ntd, syr, eyr, cal) File "/home/ordonez4/miniconda3/envs/pcmdi_metrics_3.1/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 150, in ClimAnom dseg[iyr] = yrtmpseg ValueError: could not broadcast input array from shape (364,1,90,180) into shape (366,1,90,180)

lee1043 commented 1 year ago

Ana is running precip metrics for downscaled dataset. @msahn Any chance if this error is looking familiar to you?

acordonez commented 1 year ago

@msahn I've also gotten this error for a model dataset. Is it a calendar issue? Edit: This is also for the "day" frequency. The driver has been working for me for "3hr" frequency. Traceback (most recent call last): File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/bin/variability_across_timescales_PS_driver.py", line 60, in <module> precip_variability_across_timescale( File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 57, in precip_variability_across_timescale clim, anom = ClimAnom(drg, ntd, syr, eyr, cal) File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 166, in ClimAnom dseg[iyr] = yrtmpseg ValueError: could not broadcast input array from shape (366,1,90,180) into shape (365,1,90,180)

msahn commented 1 year ago

Ana is running precip metrics for downscaled dataset. @msahn Any chance if this error is looking familiar to you?

It seems that the number of days for 2019 is "364" in the output log. The number of days for each year should be "365" or "366". If the data for 2019 is incomplete, the analysis period should be 2001-2018.

acordonez commented 1 year ago

Thanks @msahn. As far as I know it should be complete, but I'll try running it without the last year and let you know if that works.

msahn commented 1 year ago

@msahn I've also gotten this error for a model dataset. Is it a calendar issue? Edit: This is also for the "day" frequency. The driver has been working for me for "3hr" frequency. Traceback (most recent call last): File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/bin/variability_across_timescales_PS_driver.py", line 60, in <module> precip_variability_across_timescale( File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 57, in precip_variability_across_timescale clim, anom = ClimAnom(drg, ntd, syr, eyr, cal) File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 166, in ClimAnom dseg[iyr] = yrtmpseg ValueError: could not broadcast input array from shape (366,1,90,180) into shape (365,1,90,180)

Yes, I also think it is a calendar issue. Could you provide an entire output log message, like the TRMM case?

acordonez commented 1 year ago

@msahn Here's the whole log:

(pcmdi_metrics) aordonez@perlmutter:login23:~/pmp_param> variability_across_timescales_PS_driver.py -p variability_modes_param.py 
/global/homes/a/aordonez/pmp_param/
pr_day_ACCESS-CM2_historical_r1i1p1f1_19500101-20141231.nc
[1950, 2014]
3650 1825
/pscratch/sd/a/aordonez/pmp_data/variability_modes/downscaled/LOCA2/ACCESS-CM2/r1i1p1f1/historical/
/pscratch/sd/a/aordonez/pmp_data/variability_modes/downscaled/LOCA2/ACCESS-CM2/r1i1p1f1/historical/
/pscratch/sd/a/aordonez/pmp_data/variability_modes/downscaled/LOCA2/ACCESS-CM2/r1i1p1f1/historical/
['/global/homes/a/aordonez/pmp_param/pr_day_ACCESS-CM2_historical_r1i1p1f1_19500101-20141231.nc']
ACCESS-CM2.r1i1p1f1
['/global/homes/a/aordonez/pmp_param/pr_day_ACCESS-CM2_historical_r1i1p1f1_19500101-20141231.nc']
ACCESS-CM2.r1i1p1f1 standard
syr, eyr: 1950 2014
1950
Complete regridding from (365, 474, 944) to (365, 90, 180)
1950 (365, 90, 180)
1951
Complete regridding from (365, 474, 944) to (365, 90, 180)
1951 (730, 90, 180)
1952
Complete regridding from (366, 474, 944) to (366, 90, 180)
1952 (1096, 90, 180)
1953
Complete regridding from (365, 474, 944) to (365, 90, 180)
1953 (1461, 90, 180)
1954
Complete regridding from (365, 474, 944) to (365, 90, 180)
1954 (1826, 90, 180)
1955
Complete regridding from (365, 474, 944) to (365, 90, 180)
1955 (2191, 90, 180)
1956
Complete regridding from (366, 474, 944) to (366, 90, 180)
1956 (2557, 90, 180)
1957
Complete regridding from (365, 474, 944) to (365, 90, 180)
1957 (2922, 90, 180)
1958
Complete regridding from (365, 474, 944) to (365, 90, 180)
1958 (3287, 90, 180)
1959
Complete regridding from (365, 474, 944) to (365, 90, 180)
1959 (3652, 90, 180)
1960
Complete regridding from (366, 474, 944) to (366, 90, 180)
1960 (4018, 90, 180)
1961
Complete regridding from (365, 474, 944) to (365, 90, 180)
1961 (4383, 90, 180)
1962
Complete regridding from (365, 474, 944) to (365, 90, 180)
1962 (4748, 90, 180)
1963
Complete regridding from (365, 474, 944) to (365, 90, 180)
1963 (5113, 90, 180)
1964
Complete regridding from (366, 474, 944) to (366, 90, 180)
1964 (5479, 90, 180)
1965
Complete regridding from (365, 474, 944) to (365, 90, 180)
1965 (5844, 90, 180)
1966
Complete regridding from (365, 474, 944) to (365, 90, 180)
1966 (6209, 90, 180)
1967
Complete regridding from (365, 474, 944) to (365, 90, 180)
1967 (6574, 90, 180)
1968
Complete regridding from (366, 474, 944) to (366, 90, 180)
1968 (6940, 90, 180)
1969
Complete regridding from (365, 474, 944) to (365, 90, 180)
1969 (7305, 90, 180)
1970
Complete regridding from (365, 474, 944) to (365, 90, 180)
1970 (7670, 90, 180)
1971
Complete regridding from (365, 474, 944) to (365, 90, 180)
1971 (8035, 90, 180)
1972
Complete regridding from (366, 474, 944) to (366, 90, 180)
1972 (8401, 90, 180)
1973
Complete regridding from (365, 474, 944) to (365, 90, 180)
1973 (8766, 90, 180)
1974
Complete regridding from (365, 474, 944) to (365, 90, 180)
1974 (9131, 90, 180)
1975
Complete regridding from (365, 474, 944) to (365, 90, 180)
1975 (9496, 90, 180)
1976
Complete regridding from (366, 474, 944) to (366, 90, 180)
1976 (9862, 90, 180)
1977
Complete regridding from (365, 474, 944) to (365, 90, 180)
1977 (10227, 90, 180)
1978
Complete regridding from (365, 474, 944) to (365, 90, 180)
1978 (10592, 90, 180)
1979
Complete regridding from (365, 474, 944) to (365, 90, 180)
1979 (10957, 90, 180)
1980
Complete regridding from (366, 474, 944) to (366, 90, 180)
1980 (11323, 90, 180)
1981
Complete regridding from (365, 474, 944) to (365, 90, 180)
1981 (11688, 90, 180)
1982
Complete regridding from (365, 474, 944) to (365, 90, 180)
1982 (12053, 90, 180)
1983
Complete regridding from (365, 474, 944) to (365, 90, 180)
1983 (12418, 90, 180)
1984
Complete regridding from (366, 474, 944) to (366, 90, 180)
1984 (12784, 90, 180)
1985
Complete regridding from (365, 474, 944) to (365, 90, 180)
1985 (13149, 90, 180)
1986
Complete regridding from (365, 474, 944) to (365, 90, 180)
1986 (13514, 90, 180)
1987
Complete regridding from (365, 474, 944) to (365, 90, 180)
1987 (13879, 90, 180)
1988
Complete regridding from (366, 474, 944) to (366, 90, 180)
1988 (14245, 90, 180)
1989
Complete regridding from (365, 474, 944) to (365, 90, 180)
1989 (14610, 90, 180)
1990
Complete regridding from (365, 474, 944) to (365, 90, 180)
1990 (14975, 90, 180)
1991
Complete regridding from (365, 474, 944) to (365, 90, 180)
1991 (15340, 90, 180)
1992
Complete regridding from (366, 474, 944) to (366, 90, 180)
1992 (15706, 90, 180)
1993
Complete regridding from (365, 474, 944) to (365, 90, 180)
1993 (16071, 90, 180)
1994
Complete regridding from (365, 474, 944) to (365, 90, 180)
1994 (16436, 90, 180)
1995
Complete regridding from (365, 474, 944) to (365, 90, 180)
1995 (16801, 90, 180)
1996
Complete regridding from (366, 474, 944) to (366, 90, 180)
1996 (17167, 90, 180)
1997
Complete regridding from (365, 474, 944) to (365, 90, 180)
1997 (17532, 90, 180)
1998
Complete regridding from (365, 474, 944) to (365, 90, 180)
1998 (17897, 90, 180)
1999
Complete regridding from (365, 474, 944) to (365, 90, 180)
1999 (18262, 90, 180)
2000
Complete regridding from (366, 474, 944) to (366, 90, 180)
2000 (18628, 90, 180)
2001
Complete regridding from (365, 474, 944) to (365, 90, 180)
2001 (18993, 90, 180)
2002
Complete regridding from (365, 474, 944) to (365, 90, 180)
2002 (19358, 90, 180)
2003
Complete regridding from (365, 474, 944) to (365, 90, 180)
2003 (19723, 90, 180)
2004
Complete regridding from (366, 474, 944) to (366, 90, 180)
2004 (20089, 90, 180)
2005
Complete regridding from (365, 474, 944) to (365, 90, 180)
2005 (20454, 90, 180)
2006
Complete regridding from (365, 474, 944) to (365, 90, 180)
2006 (20819, 90, 180)
2007
Complete regridding from (365, 474, 944) to (365, 90, 180)
2007 (21184, 90, 180)
2008
Complete regridding from (366, 474, 944) to (366, 90, 180)
2008 (21550, 90, 180)
2009
Complete regridding from (365, 474, 944) to (365, 90, 180)
2009 (21915, 90, 180)
2010
Complete regridding from (365, 474, 944) to (365, 90, 180)
2010 (22280, 90, 180)
2011
Complete regridding from (365, 474, 944) to (365, 90, 180)
2011 (22645, 90, 180)
2012
Complete regridding from (366, 474, 944) to (366, 90, 180)
2012 (23011, 90, 180)
2013
Complete regridding from (365, 474, 944) to (365, 90, 180)
2013 (23376, 90, 180)
2014
Complete regridding from (365, 474, 944) to (365, 90, 180)
2014 (23741, 90, 180)
Traceback (most recent call last):
  File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/bin/variability_across_timescales_PS_driver.py", line 60, in <module>
    precip_variability_across_timescale(
  File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 57, in precip_variability_across_timescale
    clim, anom = ClimAnom(drg, ntd, syr, eyr, cal)
  File "/global/homes/a/aordonez/miniconda3/envs/pcmdi_metrics/lib/python3.10/site-packages/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py", line 166, in ClimAnom
    dseg[iyr] = yrtmpseg
ValueError: could not broadcast input array from shape (366,1,90,180) into shape (365,1,90,180)

msahn commented 1 year ago

Thank you. I think this issue is coming from the defined calendar name. "ACCESS-CM2.r1i1p1f1" data uses "standard" as a calendar name, but it is not defined in the metric code.

Could you help modify the code as below? From https://github.com/PCMDI/pcmdi_metrics/blob/01474bef5ce306488aa3469632190748a7a456fe/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py#L133 to if "gregorian" or "standard" in cal:

msahn commented 1 year ago

https://github.com/PCMDI/pcmdi_metrics/blob/01474bef5ce306488aa3469632190748a7a456fe/pcmdi_metrics/precip_variability/lib/lib_variability_across_timescales.py#L174

This line also should be modified.

acordonez commented 1 year ago

@msahn Those code changes fixed this issue for me. Thanks for your help

lee1043 commented 1 year ago

@msahn Thank you for your help!

msahn commented 1 year ago

@acordonez @lee1043 You're welcome. I am happy to be able to help with this issue.

PCMDI / pcmdi_metrics

Error using precip variability parameter file for TRMM day #961