SantanderMetGroup / loadeR

A climate4R package for data access
https://github.com/SantanderMetGroup/climate4R
GNU General Public License v3.0
13 stars 13 forks source link

Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException #64

Closed RowanFealy closed 4 years ago

RowanFealy commented 4 years ago

Hi I am trying to use the makeAggregatedDataset with ERA-Interim data that I downloaded from ECMWF in netcdf. I get the following error [2020-04-17 16:53:22] Creating dataset from 2 files [2020-04-17 16:53:22] Scanning file 1 out of 1 [2020-04-17 16:53:22] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException

I had a read of the previous related thread. Using the toolsUI I can see that the grid appears to the correctly read using the FeatureTypes > Grids option. However, I noticed that the files (t2m) had a scale and offset feature - so I used the toolUI to write out a new netcdf file that was unpacked.

The coordinates do not contain an Axis coordinate - not sure how to add this to the file. But I would have presumed that the CF compliant files from toolUI would be consistent in format.

Using dataInventory on the file works fine - reads it no problem

I also get the following from

gds <- openDataset("G:/ERA-Interim/Data/interim_daily_201401.sub.nc") grid <- gds$findGridByShortName('t2m') grid [1] "Java-Object{t2m}"

I have also updated to the latest release of loadeR

Any help would be gratefully appreciated. R

RowanFealy commented 4 years ago

Follow up on previous post

ncdump of original file netcdf G:/ERA-Interim/Data1/interim_daily_201401.nc { dimensions: longitude = 67; latitude = 61; time = UNLIMITED; // (124 currently) variables: double t2m(time=124, latitude=61, longitude=67); :scale_factor = 0.0010899350256532252; // double :add_offset = 267.3471494294598; // double :_FillValue = -32767S; // short :missing_value = -32767S; // short :units = "K"; :long_name = "2 metre temperature";

float longitude(longitude=67);
  :units = "degrees_east";
  :long_name = "longitude";
  :_CoordinateAxisType = "Lon";

float latitude(latitude=61);
  :units = "degrees_north";
  :long_name = "latitude";
  :_CoordinateAxisType = "Lat";

int time(time=124);
  :units = "hours since 1900-01-01 00:00:00.0";
  :long_name = "time";
  :calendar = "gregorian";
  :_CoordinateAxisType = "Time";

// global attributes: :Conventions = "CF-1.6"; :history = "2020-04-04 14:58:42 GMT by grib_to_netcdf-2.15.0: grib_to_netcdf /data/data05/scratch/0c/5d/_mars-atls17-a82bacafb5c306db76464bc7e824bb75-e2CKAd.grib -o /data/data02/scratch/51/b6/_grib2netcdf-atls19-a82bacafb5c306db76464bc7e824bb75-yvSrEu.nc -utime"; :_CoordSysBuilder = "ucar.nc2.dataset.conv.CF1Convention"; }

ncdump of file are writing out from toolUI netcdf G:/ERA-Interim/Data/interim_daily_201401.sub.nc { dimensions: time = UNLIMITED; // (124 currently) latitude = 61; longitude = 67; variables: double t2m(time=124, latitude=61, longitude=67); :_FillValue = -32767.0; // double :missing_value = -32767S; // short :units = "K"; :long_name = "2 metre temperature"; :coordinates = "time latitude longitude ";

int time(time=124);
  :units = "hours since 1900-01-01 00:00:00.0";
  :long_name = "time";
  :calendar = "gregorian";
  :_CoordinateAxisType = "Time";

float latitude(latitude=61);
  :units = "degrees_north";
  :long_name = "latitude";
  :_CoordinateAxisType = "Lat";
  :standard_name = "latitude";

float longitude(longitude=67);
  :units = "degrees_east";
  :long_name = "longitude";
  :_CoordinateAxisType = "Lon";
  :standard_name = "longitude";

// global attributes: :Conventions = "CF-1.6"; :history = "Fri Apr 17 14:55:04 2020: ncpdq --unpack G:\ERA-Interim\Data1\interim_daily_201401.nc G:\ERA-Interim\Data\interim_daily_201401.nc\n2020-04-04 14:58:42 GMT by grib_to_netcdf-2.15.0: grib_to_netcdf /data/data05/scratch/0c/5d/_mars-atls17-a82bacafb5c306db76464bc7e824bb75-e2CKAd.grib -o /data/data02/scratch/51/b6/_grib2netcdf-atls19-a82bacafb5c306db76464bc7e824bb75-yvSrEu.nc -utime"; :nco_openmp_thread_number = 1; // int :History = "Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)\nOriginal Dataset = G:/ERA-Interim/Data/interim_daily_201401.nc; Translation Date = 2020-04-17T15:07:02.594Z"; :geospatial_lat_min = 30.0; // double :geospatial_lat_max = 75.0; // double :geospatial_lon_min = -15.0; // double :geospatial_lon_max = 34.5; // double :_CoordSysBuilder = "ucar.nc2.dataset.conv.CF1Convention"; }

RowanFealy commented 4 years ago

Finally, I added the missing Axis information to the lat and lon (via ncatted), but still get same error

cofinoa commented 4 years ago

@RowanFealy can you provide 2 original sample files?

sixtohg commented 4 years ago

Dear Rowan, it is difficult without the files but in your references I don't see any problem with the file. I have taken a quick look to the code and I think that the problem is located in the loop of the line 146-162, I think that the function doesn't find the file due to the directory structure. Do you have the files in a subdirectory with the name of the variable? I mean, something like G:/ERA-Interim/Data/t2m/interim_daily_201401.sub.nc or including the name of the variable in the namefile in order to be able to locate the file according to the code. See this line: varfile <- grep(vars[ind[i]], lf, value = TRUE) It is only a first idea so I am not sure if it will work... Best regards. Sixto

RowanFealy commented 4 years ago

Hi Sixto Thank you for responding – I greatly appreciate it.

No, I tried to alter the directory path to see if that was an issue – I have sent a link with the files to the reply address in this email.

Regards Rowan

From: Sixto Herrera García notifications@github.com Sent: Friday 17 April 2020 20:26 To: SantanderMetGroup/loadeR loadeR@noreply.github.com Cc: Rowan Fealy Rowan.Fealy@mu.ie; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [SantanderMetGroup/loadeR] Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException (#64)

Warning

This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.

Dear Rowan, it is difficult without the files but in your references I don't see any problem with the file. I have taken a quick look to the code and I think that the problem is located in the loop of the line 146-162, I think that the function doesn't find the file due to the directory structure. Do you have the files in a subdirectory with the name of the variable? I mean, something like G:/ERA-Interim/Data/t2m/interim_daily_201401.sub.nc or including the name of the variable in the namefile in order to be able to locate the file according to the code. See this line: varfile <- grep(vars[ind[i]], lf, value = TRUE) It is only a first idea so I am not sure if it will work... Best regards. Sixto

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/SantanderMetGroup/loadeR/issues/64#issuecomment-615424358, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APHIWLQDWG7PXTCQKFPYPM3RNCUMHANCNFSM4MK343EA.

RowanFealy commented 4 years ago

Apologies for resending – I got a bounced email message on the last one

Hi Antonio Thank you for the really quick response – I will attach the original file (downloaded from ECMWF) and the file I wrote out of toolsUI.

The file names without sub are the original files. I have shared a one drive link to notifications@github.commailto:notifications@github.com address. Please let me know if I can do anything on myside to resolve this

Thanks again R

From: Antonio S. Cofiño notifications@github.com Sent: Friday 17 April 2020 20:21 To: SantanderMetGroup/loadeR loadeR@noreply.github.com Cc: Rowan Fealy Rowan.Fealy@mu.ie; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [SantanderMetGroup/loadeR] Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException (#64)

Warning

This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.

@RowanFealyhttps://github.com/RowanFealy do you have 2 original sample files?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/SantanderMetGroup/loadeR/issues/64#issuecomment-615421975, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APHIWLR2LXYUC5FDOOIKY33RNCT2FANCNFSM4MK343EA.

sixtohg commented 4 years ago

Hi Rowan, I have replicated your structure with some files of ERA5 downloaded from the same server and the problem seems to be the one I said before about the name of the directory or the files to let the function to find the corresponding files. The first call to the function was done with the raw data structure and the second one introducing the name of the variable in the namefile: fg10_ECMWF_ERA5_197901_SFC_049.128.nc As you can see in the second time it works perfectly. Cheers makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:42:37] Creating dataset from 2 files [2020-04-18 10:42:37] Scanning file 1 out of 1 [2020-04-18 10:42:37] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException

makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:44:02] Creating dataset from 2 files [2020-04-18 10:44:02] Scanning file 1 out of 1 [2020-04-18 10:44:02] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... [2020-04-18 10:44:02] Dimension length defined [2020-04-18 10:44:02] NcML file "/home/sixto/Documentos/DATA/ERA-Interim.ncml" created from 2 files corresponding to 1 variables Use 'dataInventory' to obtain a description of the dataset

RowanFealy commented 4 years ago

Thank you! I misread the previous response you gave and renamed the folder structure, rather than adding the variable name to the file name.

Guys – I gotta say – the response has been super quick and fantastic - I struggled with this for most of yesterday. I should also say the package is great.

Much appreciated Rowan

From: Sixto Herrera García notifications@github.com Sent: Saturday 18 April 2020 09:48 To: SantanderMetGroup/loadeR loadeR@noreply.github.com Cc: Rowan Fealy Rowan.Fealy@mu.ie; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [SantanderMetGroup/loadeR] Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException (#64)

Warning

This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.

Hi Rowan, I have replicated your structure with some files of ERA5 downloaded from the same server and the problem seems to be the one I said before about the name of the directory or the files to let the function to find the corresponding files. The first call to the function was done with the raw data structure and the second one introducing the name of the variable in the namefile: fg10_ECMWF_ERA5_197901_SFC_049.128.nc As you can see in the second time it works perfectly. Cheers makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:42:37] Creating dataset from 2 files [2020-04-18 10:42:37] Scanning file 1 out of 1 [2020-04-18 10:42:37] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... Error in .jnew(.class.to.jclass[cm], a) : java.lang.NullPointerException

makeAggregatedDataset("/home/sixto/Documentos/DATA/ERA-Interim/Data","/home/sixto/Documentos/DATA/ERA-Interim.ncml") [2020-04-18 10:44:02] Creating dataset from 2 files [2020-04-18 10:44:02] Scanning file 1 out of 1 [2020-04-18 10:44:02] Defining aggregating dimension length This process may be slow but will significantly speed-up data retrieval... [2020-04-18 10:44:02] Dimension length defined [2020-04-18 10:44:02] NcML file "/home/sixto/Documentos/DATA/ERA-Interim.ncml" created from 2 files corresponding to 1 variables Use 'dataInventory' to obtain a description of the dataset

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/SantanderMetGroup/loadeR/issues/64#issuecomment-615781000, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APHIWLRQIMYV5SHGJGZDUEDRNFSM7ANCNFSM4MK343EA.