Closed RowanFealy closed 4 years ago
Follow up on previous post
ncdump of original file netcdf G:/ERA-Interim/Data1/interim_daily_201401.nc { dimensions: longitude = 67; latitude = 61; time = UNLIMITED; // (124 currently) variables: double t2m(time=124, latitude=61, longitude=67); :scale_factor = 0.0010899350256532252; // double :add_offset = 267.3471494294598; // double :_FillValue = -32767S; // short :missing_value = -32767S; // short :units = "K"; :long_name = "2 metre temperature";
float longitude(longitude=67);
:units = "degrees_east";
:long_name = "longitude";
:_CoordinateAxisType = "Lon";
float latitude(latitude=61);
:units = "degrees_north";
:long_name = "latitude";
:_CoordinateAxisType = "Lat";
int time(time=124);
:units = "hours since 1900-01-01 00:00:00.0";
:long_name = "time";
:calendar = "gregorian";
:_CoordinateAxisType = "Time";
// global attributes: :Conventions = "CF-1.6"; :history = "2020-04-04 14:58:42 GMT by grib_to_netcdf-2.15.0: grib_to_netcdf /data/data05/scratch/0c/5d/_mars-atls17-a82bacafb5c306db76464bc7e824bb75-e2CKAd.grib -o /data/data02/scratch/51/b6/_grib2netcdf-atls19-a82bacafb5c306db76464bc7e824bb75-yvSrEu.nc -utime"; :_CoordSysBuilder = "ucar.nc2.dataset.conv.CF1Convention"; }
ncdump of file are writing out from toolUI netcdf G:/ERA-Interim/Data/interim_daily_201401.sub.nc { dimensions: time = UNLIMITED; // (124 currently) latitude = 61; longitude = 67; variables: double t2m(time=124, latitude=61, longitude=67); :_FillValue = -32767.0; // double :missing_value = -32767S; // short :units = "K"; :long_name = "2 metre temperature"; :coordinates = "time latitude longitude ";
int time(time=124);
:units = "hours since 1900-01-01 00:00:00.0";
:long_name = "time";
:calendar = "gregorian";
:_CoordinateAxisType = "Time";
float latitude(latitude=61);
:units = "degrees_north";
:long_name = "latitude";
:_CoordinateAxisType = "Lat";
:standard_name = "latitude";
float longitude(longitude=67);
:units = "degrees_east";
:long_name = "longitude";
:_CoordinateAxisType = "Lon";
:standard_name = "longitude";
// global attributes: :Conventions = "CF-1.6"; :history = "Fri Apr 17 14:55:04 2020: ncpdq --unpack G:\ERA-Interim\Data1\interim_daily_201401.nc G:\ERA-Interim\Data\interim_daily_201401.nc\n2020-04-04 14:58:42 GMT by grib_to_netcdf-2.15.0: grib_to_netcdf /data/data05/scratch/0c/5d/_mars-atls17-a82bacafb5c306db76464bc7e824bb75-e2CKAd.grib -o /data/data02/scratch/51/b6/_grib2netcdf-atls19-a82bacafb5c306db76464bc7e824bb75-yvSrEu.nc -utime"; :nco_openmp_thread_number = 1; // int :History = "Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)\nOriginal Dataset = G:/ERA-Interim/Data/interim_daily_201401.nc; Translation Date = 2020-04-17T15:07:02.594Z"; :geospatial_lat_min = 30.0; // double :geospatial_lat_max = 75.0; // double :geospatial_lon_min = -15.0; // double :geospatial_lon_max = 34.5; // double :_CoordSysBuilder = "ucar.nc2.dataset.conv.CF1Convention"; }
Finally, I added the missing Axis information to the lat and lon (via ncatted), but still get same error
@RowanFealy can you provide 2 original sample files?
Dear Rowan, it is difficult without the files but in your references I don't see any problem with the file. I have taken a quick look to the code and I think that the problem is located in the loop of the line 146-162, I think that the function doesn't find the file due to the directory structure. Do you have the files in a subdirectory with the name of the variable? I mean, something like G:/ERA-Interim/Data/t2m/interim_daily_201401.sub.nc or including the name of the variable in the namefile in order to be able to locate the file according to the code. See this line: varfile <- grep(vars[ind[i]], lf, value = TRUE) It is only a first idea so I am not sure if it will work... Best regards. Sixto
Hi Sixto Thank you for responding – I greatly appreciate it.
No, I tried to alter the directory path to see if that was an issue – I have sent a link with the files to the reply address in this email.
Regards Rowan
From: Sixto Herrera García notifications@github.com Sent: Friday 17 April 2020 20:26 To: SantanderMetGroup/loadeR loadeR@noreply.github.com Cc: Rowan Fealy Rowan.Fealy@mu.ie; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [SantanderMetGroup/loadeR] Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException (#64)
Warning
This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.
Dear Rowan, it is difficult without the files but in your references I don't see any problem with the file. I have taken a quick look to the code and I think that the problem is located in the loop of the line 146-162, I think that the function doesn't find the file due to the directory structure. Do you have the files in a subdirectory with the name of the variable? I mean, something like G:/ERA-Interim/Data/t2m/interim_daily_201401.sub.nc or including the name of the variable in the namefile in order to be able to locate the file according to the code. See this line: varfile <- grep(vars[ind[i]], lf, value = TRUE) It is only a first idea so I am not sure if it will work... Best regards. Sixto
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/SantanderMetGroup/loadeR/issues/64#issuecomment-615424358, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APHIWLQDWG7PXTCQKFPYPM3RNCUMHANCNFSM4MK343EA.
Apologies for resending – I got a bounced email message on the last one
Hi Antonio Thank you for the really quick response – I will attach the original file (downloaded from ECMWF) and the file I wrote out of toolsUI.
The file names without sub are the original files. I have shared a one drive link to notifications@github.commailto:notifications@github.com address. Please let me know if I can do anything on myside to resolve this
Thanks again R
From: Antonio S. Cofiño notifications@github.com Sent: Friday 17 April 2020 20:21 To: SantanderMetGroup/loadeR loadeR@noreply.github.com Cc: Rowan Fealy Rowan.Fealy@mu.ie; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [SantanderMetGroup/loadeR] Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException (#64)
Warning
This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.
@RowanFealyhttps://github.com/RowanFealy do you have 2 original sample files?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/SantanderMetGroup/loadeR/issues/64#issuecomment-615421975, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APHIWLR2LXYUC5FDOOIKY33RNCT2FANCNFSM4MK343EA.
Hi Rowan, I have replicated your structure with some files of ERA5 downloaded from the same server and the problem seems to be the one I said before about the name of the directory or the files to let the function to find the corresponding files. The first call to the function was done with the raw data structure and the second one introducing the name of the variable in the namefile: fg10_ECMWF_ERA5_197901_SFC_049.128.nc As you can see in the second time it works perfectly. Cheers makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:42:37] Creating dataset from 2 files [2020-04-18 10:42:37] Scanning file 1 out of 1 [2020-04-18 10:42:37] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException
makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:44:02] Creating dataset from 2 files [2020-04-18 10:44:02] Scanning file 1 out of 1 [2020-04-18 10:44:02] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... [2020-04-18 10:44:02] Dimension length defined [2020-04-18 10:44:02] NcML file "/home/sixto/Documentos/DATA/ERA-Interim.ncml" created from 2 files corresponding to 1 variables Use 'dataInventory' to obtain a description of the dataset
Thank you! I misread the previous response you gave and renamed the folder structure, rather than adding the variable name to the file name.
Guys – I gotta say – the response has been super quick and fantastic - I struggled with this for most of yesterday. I should also say the package is great.
Much appreciated Rowan
From: Sixto Herrera García notifications@github.com Sent: Saturday 18 April 2020 09:48 To: SantanderMetGroup/loadeR loadeR@noreply.github.com Cc: Rowan Fealy Rowan.Fealy@mu.ie; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [SantanderMetGroup/loadeR] Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException (#64)
Warning
This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.
Hi Rowan, I have replicated your structure with some files of ERA5 downloaded from the same server and the problem seems to be the one I said before about the name of the directory or the files to let the function to find the corresponding files. The first call to the function was done with the raw data structure and the second one introducing the name of the variable in the namefile: fg10_ECMWF_ERA5_197901_SFC_049.128.nc As you can see in the second time it works perfectly. Cheers makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:42:37] Creating dataset from 2 files [2020-04-18 10:42:37] Scanning file 1 out of 1 [2020-04-18 10:42:37] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException
makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:44:02] Creating dataset from 2 files [2020-04-18 10:44:02] Scanning file 1 out of 1 [2020-04-18 10:44:02] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... [2020-04-18 10:44:02] Dimension length defined [2020-04-18 10:44:02] NcML file "/home/sixto/Documentos/DATA/ERA-Interim.ncml" created from 2 files corresponding to 1 variables Use 'dataInventory' to obtain a description of the dataset
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/SantanderMetGroup/loadeR/issues/64#issuecomment-615781000, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APHIWLRQIMYV5SHGJGZDUEDRNFSM7ANCNFSM4MK343EA.
Hi I am trying to use the makeAggregatedDataset with ERA-Interim data that I downloaded from ECMWF in netcdf. I get the following error [2020-04-17 16:53:22] Creating dataset from 2 files [2020-04-17 16:53:22] Scanning file 1 out of 1 [2020-04-17 16:53:22] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException
I had a read of the previous related thread. Using the toolsUI I can see that the grid appears to the correctly read using the FeatureTypes > Grids option. However, I noticed that the files (t2m) had a scale and offset feature - so I used the toolUI to write out a new netcdf file that was unpacked.
The coordinates do not contain an Axis coordinate - not sure how to add this to the file. But I would have presumed that the CF compliant files from toolUI would be consistent in format.
Using dataInventory on the file works fine - reads it no problem
I also get the following from
I have also updated to the latest release of loadeR
Any help would be gratefully appreciated. R