search

JGCRI / RCMIP5

R scripts for processing CMIP5 data
Other
32 stars 10 forks source link

Error in loadEnsemble #132

Closed cahartin closed 9 years ago

cahartin commented 9 years ago

Error in loadEnsemble(variable, model, experiment, ensemble, domain, path = path, : Duplicate values in this file's time data

dissic MPI-ESM-MR historical

Below is a print out of this model. Note that the files are not complete and that we are missing a few years in the files.

cahartin commented 9 years ago

Wed May 27 14:02:03 2015 Starting new job [1] "Getting file info..." Models for this experiment I see: MPI-ESM-MR Number of files: 1525 [1] "================================================" [1] "dissic" [1] "historical" [1] "F:/output_oceans/dissic_historical.csv" [1] "MPI-ESM-MR" [1] "================================================" Number of files for this model, variable, experiment I see: 4 Wed May 27 14:02:15 2015 Year sequence is: 1850 1852 1854 1856 1858 1860 1862 1864 1866 1868 1870 1872 1874 1876 1878 1880 1882 1884 1886 1888 1890 1892 1894 1896 1898 1900 1902 1904 1906 1908 1910 1912 1914 1916 1918 1920 1922 1924 1926 1928 1930 1932 1934 1936 1938 1940 1942 1944 1946 1948 1950 1952 1954 1956 1958 1960 1962 1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 [1] "--------------------------- new yearseq" [1] 1850 Wed May 27 14:02:15 2015 Attempting load of dissic MPI-ESM-MR historical Oyr F:/CMIP5_HISTORICAL 1850 Averaging ensembles: r1i1p1 Loading F:/CMIP5_HISTORICAL/dissic/dissic_Oyr_MPI-ESM-MR_historical_r1i1p1_1860-1869.nc

All done! Processed: 0 of 1 successfully [1] "done with job"

bpbond commented 9 years ago

I don't see that error message ("Duplicate values in this file's time data") anywhere in the source code, although there is a line

assert_that(!any(duplicated(thisTimeRaw)))

in loadEnsemble:230. What packageVersion("RCMIP5") is being used here?

cahartin commented 9 years ago

version: 1.1.9010

ktoddbrown commented 9 years ago

Can you load the file in netcdf?

Edit: Or rather with the ncdf package.

ktoddbrown commented 9 years ago

@cahartin Is this still an issue?

cahartin commented 9 years ago

I got sidetracked. let me take a look this afternoon.

cahartin commented 9 years ago

Yes, the file loads when only using ncdf library to load. but does not load using loadCMIP5

bpbond commented 9 years ago

Corinne, can you fx the file to me?

ktoddbrown commented 9 years ago

ESGF and I are having words today. I don't think I can get this downloaded to debug it right now. I'll give it another try on Monday if the issue is still open.

cahartin commented 9 years ago

@bpbond its transferring now.

bpbond commented 9 years ago

Hey @cahartin thanks for the file, but I can't reproduce this error–this file loads fine for me. Can you make sure you're fully updated and then provide exact sequence of calls (or the script itself)?

cahartin commented 9 years ago

packageVersion("RCMIP5") [1] ‘1.1.9020’ library(RCMIP5)

d<-loadCMIP5(variable="dissic", model="MPI-ESM-MR", experiment="historical", domain="Oyr", yearRange=c(1850,1851), ZRange=c(0,10), path="F:/CMIP5_HISTORICAL/dissic", verbose=T) Averaging ensembles: r1i1p1 Loading F:/CMIP5_HISTORICAL/dissic/dissic_Oyr_MPI-ESM-MR_historical_r1i1p1_1860-1869.nc

ktoddbrown commented 9 years ago

Are you sure that 1850-1851 are in the years covered by the model?

cahartin commented 9 years ago

@ktoddbrown good call, but yea it happens on all years I try.

cahartin commented 9 years ago

i also tried it with no yearRange or ZRange specified, and I end up with the same error.

bpbond commented 9 years ago

Thanks for the excellent reproducible example!

Still very weird. I'm executing the exact same loadCMIP5 as you are, and I get correct behavior:

Averaging ensembles: r1i1p1 
Loading required namespace: ncdf4
Loading /Users/d3x290/Data//CMIP5/dissic_Oyr_MPI-ESM-MR_historical_r1i1p1_1860-1869.nc 
- dissic dimension names: i j lev time 
- loading only Z values 1 - 1 
- skipping file because not in yearRange
Warning message:
In loadCMIP5(variable = "dissic", model = "MPI-ESM-MR", experiment = "historical",  :
  No ensembles were loaded: dissic MPI-ESM-MR historical
ktoddbrown commented 9 years ago

I downloaded 1860-1869 and am also unable to replicate as well.

cahartin commented 9 years ago

weird. I wonder if it has something to do with windows??

when I load the file using the ncdf package the Year values come back not as I would have expected. It may just be a bogus model! ncid<-open.ncdf("F:/CMIP5_HISTORICAL/dissic/dissic_Oyr_MPI-ESM-MR_historical_r1i1p1_1860-1869.nc")

year<-get.var.ncdf(ncid, "time") year [1] 3836.0 4201.5 4566.5 4931.5 5297.0 5662.5 6027.5 6392.5 6758.0 7123.5

bpbond commented 9 years ago

No, I think it's a real error, but it seems to be occurring when the code tries to load dissic_Oyr_MPI-ESM-MR_historical_r1i1p1_1960-1969.nc. Right? That's probably the one we need to test against.

cahartin commented 9 years ago

Bogus time data in the 1960-1969 file. Do we want a better error message?

ncid<-open.ncdf("F:/CMIP5_HISTORICAL/dissic/dissic_Oyr_MPI-ESM-MR_historical_r1i1p1_1960-1969.nc")

year<-get.var.ncdf(ncid, "time") year [1] 40360.0 40725.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

ktoddbrown commented 9 years ago

We could fall back on reconstructing it from the file names... at least for yr and mon frequency this should be fairly straight forward.

bpbond commented 9 years ago

I believe we determined this was a bug in the file, not our problem. Closing.