Closed danhamill closed 3 years ago
Dan, are you clipping the data on import? I've seen this before when high-resolution data is being imported unclipped. This runs up the memory pretty quickly. I thought the batch importer was using a relatively safe batch size of 1 during import, but maybe something is happing differently when accessing via the API as opposed to the UI. You could try controlling the batch size of your data by only passing a few files (~5-10) at a time to the BatchImporter
then increasing the batch size as memory permits. Let me know how it goes.
I am clipping the data to a HUC4 watershed. I tried reducing the input grids to one and keep getting the same error when I try and pass 2 or more RTMA or MRMS grids. Here is my jython script:
from mil.army.usace.hec.vortex.io import BatchImporter
from mil.army.usace.hec.vortex import Options
from mil.army.usace.hec.vortex.geo import WktFactory
from glob import glob
import os
shps ={'NORTH PLATTE':r"C:\workspace\Post-Wildfire\shp\Watersheds\huc4_northPlatte_ALB.shp",
'SOUTH PLATTE':r"C:\workspace\Post-Wildfire\shp\Watersheds\huc4_southPlatte_ALB.shp",
'WILLAMETTE':r"C:\workspace\Post-Wildfire\shp\Watersheds\huc4_willamette_ALB.shp"}
prism = {16:r"G:\PRISM\WY_2016_PRISM",
17:r"G:\PRISM\WY_2017_PRISM",
18:r"G:\PRISM\WY_2018_PRISM",
19:r"G:\PRISM\WY_2019_PRISM",
20:r"G:\PRISM\WY_2020_PRISM"}
mrms = {16:r"G:\MRMS\precip\2016",
17:r"G:\MRMS\precip\2017",
18:r"G:\MRMS\precip\2018",
19:r"G:\MRMS\precip\2019",
20:r"G:\MRMS\precip\2020"}
rtma = {16:r'G:\RTMA\Temp\2016',
17:r'G:\RTMA\Temp\2017',
18:r'G:\RTMA\Temp\2018',
19:r'G:\RTMA\Temp\2019',
20:r'G:\RTMA\Temp\2020'}
var_names = {'RTMA':'Temperature_height_above_ground',
'MRMS':'GaugeCorrQPE01H_altitude_above_msl',
'PRISM':'ppt'}
exts = {'RTMA':'*.grib2',
'MRMS':'*.grib2',
'PRISM':'*_*_stable_*_*.bil',}
for wy in [16,17,18,19,20]:
print 'Now Processing Water Year ' + str(wy)
for basin in ['NORTH PLATTE','SOUTH PLATTE', 'WILLAMETTE']:
print 'Now Processing basin ' + basin
clip_shp = shps[basin]
if os.path.exists(clip_shp):
print 'Found Shapefile'
for ds, files in zip(['PRISM','MRMS','RTMA'][1:2], [prism,mrms,rtma][1:2]):
print 'Now processing ' + ds + ' files'
dss_file = r"C:\workspace\Post-Wildfire\dss" + os.sep + ds + os.sep + '_'.join([basin, 'WY'+str(wy), ds])+ ".dss"
if os.path.exists(dss_file):
print 'Found DSS File'
in_files = glob(files[wy] + os.sep + exts[ds])
print 'Found' + ' ' + str(len(in_files)) + ' Files'
variables = [var_names[ds]]
for i in in_files:
geo_options = Options.create()
geo_options.add('pathToShp', clip_shp)
geo_options.add('targetCellSize', '2000')
geo_options.add('targetWkt', WktFactory.shg())
geo_options.add('resamplingMethod', 'bilinear')
destination = dss_file
write_options = Options.create()
write_options.add('partF', ds)
write_options.add('partA', 'SHG')
write_options.add('partB', basin)
myImport = BatchImporter.builder().inFiles([i]).variables(variables).geoOptions(geo_options).destination(destination).writeOptions(write_options).build()
myImport.process()
Is geo_options.add('resamplingMethod', 'Bilinear')
correct?
I was able to successfully pass multiple PRISM bil files with out having any memory issues. So maybe it has something to do with how the grib2 files are being read?
geo_options.add('resamplingMethod', 'Bilinear')
is correct.
Are you able to reproduce the out of memory error with a really simple test case? I.e. distill your script to a single file import. If you're able to isolate the issue and reproduce that'll help me troubleshoot.
Sure. See attached. If you process all five grids in data
I get the memory error shown above. If you only process two of the grids in data
you get a resampler error:
ucar.nc2.grib.collection.Grib2Iosp SEVERE Failed to readData
ucar.nc2.dt.grid.GeoGrid SEVERE GeoGrid.getdataSlice() on variable GaugeCorrQPE01H_altitude_above_msl dataset C:/workspace/git_clones/vortex_scripting/multi_grib/data/MRMS_GaugeCorr_QPE_01H_00.00_20170930-120000.grib2
mil.army.usace.hec.vortex.io.NetcdfDataReader SEVERE javax.imageio.IIOException: Caught exception during read:
OS Name is Windows 10
-----DSS---ZOPEN: Existing File Opened, File: C:\workspace\Post-Wildfire\dss\MRMS\NORTH PLATTE_WY18_MRMS.dss
Unit: 3; DSS Versions - Software: 6-YN, File: 6-YN, Library 7-HK
-----DSS---ZWRITE Unit 3; Vers. 2: /SHG/NORTH PLATTE/PRECIPITATION/30SEP2017:1200/30SEP2017:1300/MRMS/
Traceback (most recent call last):
File "C:\workspace\git_clones\vortex_scripting\multi_grib\import_grib.py", line 29, in <module>
myImport.process()
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at mil.army.usace.hec.vortex.io.BatchImporter.process(BatchImporter.java:85)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
java.lang.IllegalArgumen
Thanks for the test case. I was able to run the script by upping the memory spec for the JVM. I used -J-Xmx2g
, to specify 2 gigabytes for the JVM, for example:
C:\Programs\jython2.7.2\bin\jython.exe -J-Xmx2g -Djava.library.path=%VORTEX_HOME%\bin;%VORTEX_HOME%\bin\gdal .\import_grib.py
Hopefully that works for you!
Yes That worked.
I think it would be helpful to update the wiki. The the JVM memory spec flag -J-Xmx2g
and the resampling argument geo_options.add('resamplingMethod', 'Bilinear')
for the jython script. I don't think you can submit a PR for wiki pages.
Agree! I'll make that update. Thanks!
Tom,
I am developing a jython workflow to process data for several basins and I am running into
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError
. Have you seen this before?If I assign the constructed
BatchImporter.builder()
tomy_import
from a jython terminal and runmy_import.process()
, I see this:If you execute the code from a python file, it throws a the
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError
and exits the program before any of theZWRITE
commands.