Multiprocessing error from jython script.

[danhamill]

Tom,

I am developing a jython workflow to process data for several basins and I am running into java.lang.OutOfMemoryError: java.lang.OutOfMemoryError. Have you seen this before?

If I assign the constructed BatchImporter.builder() to my_import from a jython terminal and run my_import.process(), I see this:

>>> my_import.process()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:603)
        at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
        at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
        at mil.army.usace.hec.vortex.io.BatchImporter.process(BatchImporter.java:85)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError
>>> [529.788s][warning][gc,alloc] ForkJoinPool.commonPool-worker-23: Retried waiting for GCLocker too often allocating 2385952 words
 -----DSS---ZWRITE Unit    3; Vers.    3:  /SHG/NORTH PLATTE/PRECIPITATION/04SEP2016:1200/05SEP2016:1200/PRISM/
 -----DSS---ZWRITE Unit    3; Vers.    2:  /SHG/NORTH PLATTE/PRECIPITATION/28OCT2015:1200/29OCT2015:1200/PRISM/
 -----DSS---ZWRITE Unit    3; Vers.    3:  /SHG/NORTH PLATTE/PRECIPITATION/28APR2016:1200/29APR2016:1200/PRISM/
 -----DSS---ZWRITE Unit    3; Vers.    3:  /SHG/NORTH PLATTE/PRECIPITATION/26MAY2016:1200/27MAY2016:1200/PRISM/
 -----DSS---ZWRITE Unit    3; Vers.    2:  /SHG/NORTH PLATTE/PRECIPITATION/29OCT2015:1200/30OCT2015:1200/PRISM/
 -----DSS---ZWRITE Unit    3; Vers.    3:  /SHG/NORTH PLATTE/PRECIPITATION/29APR2016:1200/30APR2016:1200/PRISM/

If you execute the code from a python file, it throws a the java.lang.OutOfMemoryError: java.lang.OutOfMemoryError and exits the program before any of the ZWRITE commands.

[tombrauer]

Dan, are you clipping the data on import? I've seen this before when high-resolution data is being imported unclipped. This runs up the memory pretty quickly. I thought the batch importer was using a relatively safe batch size of 1 during import, but maybe something is happing differently when accessing via the API as opposed to the UI. You could try controlling the batch size of your data by only passing a few files (~5-10) at a time to the BatchImporter then increasing the batch size as memory permits. Let me know how it goes.

[danhamill]

I am clipping the data to a HUC4 watershed. I tried reducing the input grids to one and keep getting the same error when I try and pass 2 or more RTMA or MRMS grids. Here is my jython script:

from mil.army.usace.hec.vortex.io import BatchImporter
from mil.army.usace.hec.vortex import Options
from mil.army.usace.hec.vortex.geo import WktFactory
from glob import glob
import os

shps ={'NORTH PLATTE':r"C:\workspace\Post-Wildfire\shp\Watersheds\huc4_northPlatte_ALB.shp",
        'SOUTH PLATTE':r"C:\workspace\Post-Wildfire\shp\Watersheds\huc4_southPlatte_ALB.shp",
        'WILLAMETTE':r"C:\workspace\Post-Wildfire\shp\Watersheds\huc4_willamette_ALB.shp"}

prism = {16:r"G:\PRISM\WY_2016_PRISM",
        17:r"G:\PRISM\WY_2017_PRISM",
        18:r"G:\PRISM\WY_2018_PRISM",
        19:r"G:\PRISM\WY_2019_PRISM",
        20:r"G:\PRISM\WY_2020_PRISM"}

mrms = {16:r"G:\MRMS\precip\2016",
        17:r"G:\MRMS\precip\2017",
        18:r"G:\MRMS\precip\2018",
        19:r"G:\MRMS\precip\2019",
        20:r"G:\MRMS\precip\2020"}

rtma = {16:r'G:\RTMA\Temp\2016',
        17:r'G:\RTMA\Temp\2017',
        18:r'G:\RTMA\Temp\2018',
        19:r'G:\RTMA\Temp\2019',
        20:r'G:\RTMA\Temp\2020'}

var_names = {'RTMA':'Temperature_height_above_ground',
             'MRMS':'GaugeCorrQPE01H_altitude_above_msl',
             'PRISM':'ppt'}

exts = {'RTMA':'*.grib2',
        'MRMS':'*.grib2',
        'PRISM':'*_*_stable_*_*.bil',}

for wy in [16,17,18,19,20]:
    print 'Now Processing Water Year ' + str(wy)
    for basin in ['NORTH PLATTE','SOUTH PLATTE', 'WILLAMETTE']:
        print 'Now Processing basin ' + basin
        clip_shp = shps[basin]
        if os.path.exists(clip_shp):
            print 'Found Shapefile'
        for ds, files in zip(['PRISM','MRMS','RTMA'][1:2], [prism,mrms,rtma][1:2]):
            print 'Now processing ' + ds + ' files'
            dss_file = r"C:\workspace\Post-Wildfire\dss" + os.sep + ds + os.sep + '_'.join([basin, 'WY'+str(wy), ds])+ ".dss"
            if os.path.exists(dss_file):
                print 'Found DSS File'
            in_files = glob(files[wy] + os.sep + exts[ds])
            print 'Found' + ' ' + str(len(in_files)) + ' Files'
            variables = [var_names[ds]]
            for i in in_files:
                geo_options = Options.create()
                geo_options.add('pathToShp', clip_shp)
                geo_options.add('targetCellSize', '2000')
                geo_options.add('targetWkt', WktFactory.shg())
                geo_options.add('resamplingMethod', 'bilinear')

                destination = dss_file

                write_options = Options.create()
                write_options.add('partF', ds)
                write_options.add('partA', 'SHG')
                write_options.add('partB', basin)
                myImport = BatchImporter.builder().inFiles([i]).variables(variables).geoOptions(geo_options).destination(destination).writeOptions(write_options).build()

                myImport.process()

Is geo_options.add('resamplingMethod', 'Bilinear') correct?

I was able to successfully pass multiple PRISM bil files with out having any memory issues. So maybe it has something to do with how the grib2 files are being read?

[tombrauer]

geo_options.add('resamplingMethod', 'Bilinear') is correct.

Are you able to reproduce the out of memory error with a really simple test case? I.e. distill your script to a single file import. If you're able to isolate the issue and reproduce that'll help me troubleshoot.

[danhamill]

Sure. See attached. If you process all five grids in data I get the memory error shown above. If you only process two of the grids in data you get a resampler error:

ucar.nc2.grib.collection.Grib2Iosp SEVERE Failed to readData
ucar.nc2.dt.grid.GeoGrid SEVERE GeoGrid.getdataSlice() on variable GaugeCorrQPE01H_altitude_above_msl dataset C:/workspace/git_clones/vortex_scripting/multi_grib/data/MRMS_GaugeCorr_QPE_01H_00.00_20170930-120000.grib2
mil.army.usace.hec.vortex.io.NetcdfDataReader SEVERE javax.imageio.IIOException: Caught exception during read:

OS Name is Windows 10
    -----DSS---ZOPEN:  Existing File Opened,  File: C:\workspace\Post-Wildfire\dss\MRMS\NORTH PLATTE_WY18_MRMS.dss
                       Unit:    3;  DSS Versions - Software: 6-YN, File: 6-YN,  Library 7-HK
 -----DSS---ZWRITE Unit    3; Vers.    2:  /SHG/NORTH PLATTE/PRECIPITATION/30SEP2017:1200/30SEP2017:1300/MRMS/
Traceback (most recent call last):
  File "C:\workspace\git_clones\vortex_scripting\multi_grib\import_grib.py", line 29, in <module>
    myImport.process()
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
        at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
        at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
        at mil.army.usace.hec.vortex.io.BatchImporter.process(BatchImporter.java:85)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
java.lang.IllegalArgumen

multi_grib.zip

[tombrauer]

Thanks for the test case. I was able to run the script by upping the memory spec for the JVM. I used -J-Xmx2g, to specify 2 gigabytes for the JVM, for example:

C:\Programs\jython2.7.2\bin\jython.exe -J-Xmx2g -Djava.library.path=%VORTEX_HOME%\bin;%VORTEX_HOME%\bin\gdal .\import_grib.py

Hopefully that works for you!

[danhamill]

Yes That worked.

[danhamill]

I think it would be helpful to update the wiki. The the JVM memory spec flag -J-Xmx2g and the resampling argument geo_options.add('resamplingMethod', 'Bilinear') for the jython script. I don't think you can submit a PR for wiki pages.

[tombrauer]

Agree! I'll make that update. Thanks!