Handling large dataset from S3 fails.

Unidata / netcdf-java

The Unidata netcdf-java library

https://docs.unidata.ucar.edu/netcdf-java/current/userguide/index.html

BSD 3-Clause "New" or "Revised" License

146 stars 70 forks source link

Handling large dataset from S3 fails. #433

Open hyoklee opened 4 years ago

hyoklee commented 4 years ago

cdm-core-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT / TDS 5.0.0 beta8
Linux ip-172-31-25-188 5.3.0-1032-aws #34~18.04.2-Ubuntu SMP Fri Jul 24 10:06:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "11.0.8" 2020-07-14 OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1) OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)

Issue:

java.lang.StackOverflowError: null
ucar.unidata.io.RemoteRandomAccessFile.readFromCache(Remote RandomAccessFile.java:107

Steps to reproduce:
1. Run Ubuntu EC2 on AWS us-west-2.
2. Run TDS and create a catalog for Terra Fusion Data Sampler s3://terrafusiondatasampler/P108/TERRA_BF_L1B_O10204_20011118010522_F000_V001.h5
3. Try to read a large dataset (e.g., /MODIS/granule_2001322_0110/_1KM/Geolocation/Longitude via OPeNDAP web interface or pydap/netCDF4-python.
4. Try to export or dump the above dataset using toolsUI.

lesserwhirls commented 4 years ago

I've been trying to reproduce this using the latest code, but so far I have not had any luck. The location where the overflow is happening indicates that we are likely in an infinite regression situation, which is not good (I'm going to make a change to better handle that case).

Would you be able to test with your configurations using a snapshot version of the TDS? https://artifacts.unidata.ucar.edu/repository/unidata-all/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20200924.230921-19.war (don't forget to rename it to thredds##5.0.0-20200924.war, or something similar).

hyoklee commented 4 years ago

I tried the snapshot and I got:

Error {
    code = 500;
    message = "String index out of range: -1";
};

lesserwhirls commented 3 years ago

Ok, I was able to reproduce this with a few other datasets and it's now fixed (will be making a release of netCDF-Java today). Sorry it took so long! The fix should be available in this SNAPSHOT of the TDS, as well as the latest TDS Docker image of 5.0-SNAPSHOT.

hyoklee commented 3 years ago

@lesserwhirls , I tested your patch and it seems better. However, I can't dump ASCII value with the following error:

Request Too Large: 175.91168 Mbytes, max=50.0

I adjusted the limit in threddsConfig.xml but it doesn't seem to be honored:

  <Opendap>
    <ascLimit>50000</ascLimit>
    <binLimit>5000000</binLimit>
    <serverVersion>opendap/3.7</serverVersion>
  </Opendap>

Also, Panoply can't plot MODIS 250m longitude dataset with 8120x5416 dimensions. It throws "Connection Reset" message and kills the Tomcat server. Thus, I'm still doubtful about the fix.

dopplershift commented 3 years ago

I htink you need to adjust asclimit like:

 <Opendap>
   <ascLimit>250000</ascLimit>
   <binLimit>5000000</binLimit>
   <serverVersion>opendap/3.7</serverVersion>
 </Opendap>