whitespace after configFile=filename in batch script causes verdi.sh -batch to ignore the config file

[lizadams]

Describe the bug config files are ignored when there are blank spaces at the end of the definition in the batch txt file

To Reproduce Steps to reproduce the behavior on atmos

1. Go to test directory on atmos

   cd /home/eadams02/dkj_test

2. Review the verdi batch script:

cat verdi_2.1.5_batch.csh

Output:

#!/bin/csh -f

xvfb_run /home/eadams02/VERDI_2.1.5/verdi.sh -batch $cwd/tileplot_png_with_extra_scpace_at_end_of_cfg_file.txt
xvfb_run /home/eadams02/VERDI_2.1.5/verdi.sh -batch $cwd/tileplot_png.txt

3. run the verdi batch script:

   ./verdi_2.1.5_batch.csh

4. Use ls -lrt * to list the filesizes of the png files created

ls -lrt *

5. The smaller file sizes are named *extra_space.png and are produced by the script that ignores the config file due to two empty spaces at the end of the following line: configFile=/Users/lizadams/downloads/nolte_test/11bin.cfg in the tileplot_png_with_extra_scpace_at_end_of_cfg_file.txt file

6. When running verdi in batch mode, it gives the following output, it would be good to remove this message.

Jan 24, 2024 5:12:28 PM org.geotools.map.MapContent finalize
SEVERE: Call MapContent dispose() to prevent memory leaks

Expected behavior

1. I would expect the batch script parser to ignore blank spaces at the end of the definition of the configFile
2. Daiwen suggested that VERDI should let the user know if the batch script is using the configFile or not, rather than just ignoring the configFile, say if the path was specified wrong, or if the name was wrong, or if there are blank spaces that cause the configFile to be skipped when creating the plot.
3. Please remove the SEVERE message from org.geotools.map.MapContent finalize, if possible. See this message when running in batch script mode, but do not see this using the verdi command line.

PM25_OA_Ref-2015.png (plot created using the config file)

PM25_OA_Ref-2015_extra_space.png (plot created ignoring the config file)

Screenshots See additional plots, batch *.txt files, and verdi run scripts (both batch and cli) available in the google folder found at this link or on the directory /home/eadams02/dkj_test or /home/eadams02/nolte_test on atmos.

cd  /home/eadams02/dkj_test 

or

cd /home/eadams02/nolte_test

https://drive.google.com/drive/u/1/folders/1u_FJ0bL3T9IGBFArhomuK60ViGBMcP1w

OS (please complete the following information): Linux and Mac

Additional context Chris Nolte first reported the issue with using config files when using the verdi batch mode.

[lizadams]

Tested latest build from Tony. The config file with two spaces after the configFile definition still causes a failure, but now VERDI gives an error message, and the plots are not created.

./verdi_batch.csh 2024.01.25 15:14:47.791 [main] ERROR anl.verdi.commandline.BatchScriptHandler - Error in BatchScriptHandler.processTasks: Task is null. Cannot invoke "anl.verdi.plot.config.PlotConfiguration.getConfigFileName()" because "config" is null BatchScriptHandler: Error in BatchScriptHandler.processTasks: Task is null. Cannot invoke "anl.verdi.plot.config.PlotConfiguration.getConfigFileName()" because "config" is null java.lang.NullPointerException: Cannot invoke "anl.verdi.plot.config.PlotConfiguration.getConfigFileName()" because "config" is null at anl.verdi.plot.gui.FastTilePlot.configure(FastTilePlot.java:2667) at anl.verdi.commandline.task.TilePlotTask.run(TilePlotTask.java:148) at anl.verdi.commandline.BatchScriptHandler.processTasks(BatchScriptHandler.java:146) at anl.verdi.commandline.BatchScriptHandler.run(BatchScriptHandler.java:125) at anl.verdi.core.VerdiPlugin.run(VerdiPlugin.java:150) at saf.core.runtime.CorePlugin.runApplicationRunnable(CorePlugin.java:93) at saf.core.runtime.CorePlugin.run(CorePlugin.java:48) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at saf.core.runtime.Boot.run(Boot.java:178) at saf.core.runtime.Boot.main(Boot.java:388)

However, it gives this error whether there are two spaces after the configFile or not. It isn't reading the configFile, even if it points to a valid configFile.

[lizadams]

Using the following command line script, all three versions of VERDI that I am testing work to read the provided configFile and correctly generates the tile plot using the colorMap and legendBins used in the config file.

#!/bin/csh -f

cd /Users/lizadams/Downloads/VERDI_2.1.5_01_03/
./verdi.command -configFile /Users/lizadams/downloads/nolte_test/11bin.cfg -f /Users/lizadams/downloads/nolte_test/yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_Ref_2015 -s "PM25[1]" -f  /Users/lizadams/downloads/nolte_test/yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_2015_2015 -s "PM25[2]" -s "PM25[2]-PM25[1]" -g tile -saveImage "png" /Users/lizadams/Downloads/nolte_test/PM25_Ref-2015_01-03gui -quit

cd /Users/lizadams/Downloads/VERDI_2.1.5_01_24/
./verdi.command -configFile /Users/lizadams/downloads/nolte_test/11bin.cfg -f /Users/lizadams/downloads/nolte_test/yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_Ref_2015 -s "PM25[1]" -f  /Users/lizadams/downloads/nolte_test/yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_2015_2015 -s "PM25[2]" -s "PM25[2]-PM25[1]" -g tile -saveImage "png" /Users/lizadams/Downloads/nolte_test/PM25_Ref-2015_01-24gui -quit

cd /Users/lizadams/Downloads/VERDI_2.1.5_01_30/
./verdi.command -configFile /Users/lizadams/downloads/nolte_test/11bin.cfg -f /Users/lizadams/downloads/nolte_test/yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_Ref_2015 -s "PM25[1]" -f  /Users/lizadams/downloads/nolte_test/yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_2015_2015 -s "PM25[2]" -s "PM25[2]-PM25[1]" -g tile -saveImage "png" /Users/lizadams/Downloads/nolte_test/PM25_Ref-2015_01-30gui -quit

Using the same config file using batch command line, fails for all except for the 01-03 build. Tony tried to fix the issue of spaces causing VERDI not to read and use the configFile for the 01-24 and 01-30 builds, but this fix does not work. An error is created for all of the versions except for the 01-03 build, but that build just ignores the configFile if there are blank spaces after the definition of the configFile. The configFile used for this test does not have any blank spaces, so VERDI should be able to read it, and produce plots that match the above CLI option.

#!/bin/csh -f

/Users/lizadams/downloads/VERDI_2.1.5_01_03/verdi.command -batch /Users/lizadams/Downloads/nolte_test/tileplot_png.txt
ls -lrt *.png
mv PM25_Ref-2015.png PM25_Ref-2015-01-03.png
mv PM25_OA_Ref-2015.png  PM25_OA_Ref-2015-01-03.png

/Users/lizadams/downloads/VERDI_2.1.5_01_24/verdi.command -batch /Users/lizadams/Downloads/nolte_test/tileplot_png.txt
mv PM25_Ref-2015.png Ref-2015-01-24.png
mv PM25_OA_Ref-2015.png  PM25_OA_Ref-2015-01-24.png
ls -lrt *.png

/Users/lizadams/downloads/VERDI_2.1.5_01_24/verdi.command -batch /Users/lizadams/Downloads/nolte_test/tileplot_png.txt
mv PM25_Ref-2015.png Ref-2015-01-30.png
mv PM25_OA_Ref-2015.png  PM25_OA_Ref-2015-01-30.png
ls -lrt *.png

Contents of tileplot_png.txt

####################################################################################
# NOTE: Batch Scripting Language                                                   
#                                                                                  
#    * All parameter/value pairs should be inside one of the two blocks --
#        <Global/> or <Task/>
#    * Number of blocks is not limited
#    * Only one <Global/> block is recommended. <Global/> blocks should
#        contains different items if use multiple <Global/> blocks
#    * Parameter values in <Task/> blocks will override those in <Global/>
#    * Currently supported parameters (keys, case insensitive):
#        configFile    -- configuration file full path
#        f             -- dataset file path/name
#        dir           -- dataset file folder
#        pattern       -- dataset file name pattern
#        gtype         -- plot type (tile, line, bar, vector)
#        vector        -- vector plot variables
#        vectorTile    -- vector plot variables
#        s             -- variable name
#        ts            -- time step
#        titleString   -- plot title
#        subdomain     -- xmin ymin xmax ymax
#        subTitle1     -- plot subtitle one
#        subTitle2     -- plot subtitle two
#        saveImage     -- image file type (png, jpeg, eps, etc.)
#        imageFile     -- image file path/name
#        imageDir      -- image file folder
#        drawGridLines -- draw grid lines on the tile plot if 'yes'
#        imageWidth    -- image width
#        imageHeight   -- image height
#        unitString    -- units
#
#
####################################################################################

<Global>
  dir=/Users/lizadams/downloads/nolte_test
  configFile=/Users/lizadams/downloads/nolte_test/11bin.cfg
  f=yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_Ref_2015
  f=yearlyavg_aelmo_gmt_v54_cb6r3_ae7_aq_WR413_MYR_GCAM2CMAQ_12US1_2015_2015
  #pattern=*CCTM46*
  #s=[1]
  saveImage=png
  imageDir=/Users/lizadams/downloads/nolte_test
  #subdomain=20 10 75 50
</Global>

#######################
# Data files picked   #
# up from patterns    #
# specified for names #
#######################

<Task>
  s=PM25[1]-PM25[2]
  imageFile=PM25_Ref-2015
  gtype=tile
</Task>

<Task>
  s=PM25_OA[1]-PM25_OA[2]
  imageFile=PM25_OA_Ref-2015
  gtype=tile
</Task>

Contents of the 11bin.cfg

<PlotConfig>
        <property type="java.awt.Font" key="anl.verdi.plot.config.PlotConfiguration.domain_font" value="Lucida Grande-plain-13" />
        <property type="java.awt.Color" key="anl.verdi.plot.config.PlotConfiguration.obs_legend_color" value="-16777216" />
        <property type="java.awt.Font" key="anl.verdi.plot.config.PlotConfiguration.units_tick_font" value="Lucida Grande-plain-13" />
        <property type="java.awt.Color" key="anl.verdi.plot.config.PlotConfiguration.domain_color" value="-12566464" />
        <property type="java.awt.Color" key="anl.verdi.plot.config.PlotConfiguration.range_tick_color" value="-12566464" />
        <property type="java.awt.Color" key="anl.verdi.plot.config.PlotConfiguration.footer_line_2_color" value="-16777216" />
        <ColorMap intervalType="CUSTOM" scaleType="LINEAR" paletteType="DIVERGING" min="-2.5" max="2.5">
                <Format type="java.text.DecimalFormat" pattern="#0.00" />
                <Scale description="Color Scale">
                        <Step>-2.50</Step>
                  <Step>-2.00</Step>
                  <Step>-1.50</Step>
                  <Step>-0.50</Step>
                  <Step>-0.01</Step>
                  <Step> 0.01</Step>
                  <Step> 0.05</Step>
                  <Step> 0.50</Step>
                  <Step> 1.00</Step>
                  <Step> 1.50</Step>
                  <Step> 2.00</Step>
                  <Step> 2.50</Step>
                </Scale>
                <Palette description="Newton RGB (AVS)">
                <Color>-0010455</Color>
                <Color>-2615280</Color>
                <Color>-10218999</Color>
                <Color>-166217106</Color>
                <Color>-217239139</Color>
                <Color>-255255191</Color>
                <Color>-254224139</Color>
                <Color>-25317497</Color>
                <Color>-24410967</Color>
                <Color>-2154839</Color>
                <Color>-1650038</Color>
                </Palette>
        </ColorMap>

</PlotConfig>

Java Error message

2024.01.30 18:02:40.977 [main] ERROR anl.verdi.commandline.BatchScriptHandler - Error in BatchScriptHandler.processTasks: Task is null. Cannot invoke "anl.verdi.plot.config.PlotConfiguration.getConfigFileName()" because "config" is null
BatchScriptHandler: Error in BatchScriptHandler.processTasks: Task is null. Cannot invoke "anl.verdi.plot.config.PlotConfiguration.getConfigFileName()" because "config" is null
java.lang.NullPointerException: Cannot invoke "anl.verdi.plot.config.PlotConfiguration.getConfigFileName()" because "config" is null
    at anl.verdi.plot.gui.FastTilePlot.configure(FastTilePlot.java:2667)
    at anl.verdi.commandline.task.TilePlotTask.run(TilePlotTask.java:148)
    at anl.verdi.commandline.BatchScriptHandler.processTasks(BatchScriptHandler.java:146)
    at anl.verdi.commandline.BatchScriptHandler.run(BatchScriptHandler.java:125)
    at anl.verdi.core.VerdiPlugin.run(VerdiPlugin.java:150)
    at saf.core.runtime.CorePlugin.runApplicationRunnable(CorePlugin.java:93)
    at saf.core.runtime.CorePlugin.run(CorePlugin.java:48)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at saf.core.runtime.Boot.run(Boot.java:178)
    at saf.core.runtime.Boot.main(Boot.java:388)

[yadongxuEPA]

Tested VERDI_2.1.5_linux64_20240130.tar.gz on Atmos, tried both running the batch script from command window and directly open the batch text file in VERDI GUI, it did not work.
Below is the error message when ran the text file directly from VERDI GUI by File >>View Script Editor >>Run : (this error also showed up when used the text file without extra white space) Below is the error message from the command window:

[lizadams]

Tested latest build VERDI_2.1.5_mac_20240131.tar.gz on mac.

Confirmed that the verdi batch script is able to read the configuration file and create plots using the settings in the configuration File for the following cases.

1. batch script configuration file is specified as follows: configFile=/Users/lizadams/downloads/nolte_test/11bin.cfg
2. batch script configuration file is specified as above, but with two blank spaces after the name
3. batch script configuration file specifies a config file that uses hex colors

This issue is resolved.