Build's tests failed and code does not work, both due to NoValidGribException on valid GRIB2 files

[Timelessprod]

Hi.

I downloaded the actual source code (commit 0e5b8c3 and try to build it with gradle on Intellij. When doing it, all two thirdof the tests failed due to NoValidGribException. Here is the compilation output on Intellij console :

15:52:50: Executing task 'build'...

:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:jar UP-TO-DATE
:assemble UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test

mt.edu.um.cf2.jgribx.GribTest > testVersion STARTED

mt.edu.um.cf2.jgribx.GribTest > testVersion PASSED

mt.edu.um.cf2.jgribx.GribTest > testGrib1Gfs3 STARTED

mt.edu.um.cf2.jgribx.GribTest > testGrib1Gfs3 FAILED
    mt.edu.um.cf2.jgribx.NoValidGribException at GribTest.java:58

mt.edu.um.cf2.jgribx.GribTest > testGrib2Gfs3 STARTED

mt.edu.um.cf2.jgribx.GribTest > testGrib2Gfs3 FAILED
    mt.edu.um.cf2.jgribx.NoValidGribException at GribTest.java:163

3 tests completed, 2 failed
:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///C:/Users/amerat/Downloads/JGribX-master/JGribX-master/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
6 actionable tasks: 1 executed, 5 up-to-date
15:52:54: Task execution finished 'build'.

I thought it was due to corrupted GRIB files in your repository, but when I try with my own and with the JAR I built from the actual code (not your release), it fails too giving me this error :

mt.edu.um.cf2.jgribx.NoValidGribException: No valid GRIB records found.
    at mt.edu.um.cf2.jgribx.GribFile.<init>(GribFile.java:138)
    at mt.edu.um.cf2.jgribx.GribFile.<init>(GribFile.java:80)
    at mt.edu.um.cf2.jgribx.GribFile.<init>(GribFile.java:63)

The GRIB2 file I use is not corrupted, I can open it in Nasa's Panoply GUI Software without any error. Reading your GribFile.java:138, it looks like the code is seeing an empty record whereas it is not.

In my GRIB2 file, there's only one data Temperature_height_above_ground at only one timestamp 2021-10-12 14:00+0000 and height 2.00000 m, thus I wrote the code below.

Please note that the GRIB2 data is first acquired by a client API so that's why it's first held by a variable bytes which type is Array[Byte]. Before using your library I write the GRIB2 data to a file and then pass it's path to the GribFile constructor. This part of the code works fine, I can open the created file in Nasa's Panoply.

      import mt.edu.um.cf2.jgribx._
      import java.util.Calendar
      import java.util.GregorianCalendar

      val path: String = "mf-data.grb2"
      val file: File = new File(path)
      FileUtils.writeByteArrayToFile(file, bytes)

      val gribFile: GribFile = new GribFile(path)

      val date: Calendar = new GregorianCalendar(2021, Calendar.OCTOBER, 12, 14, 0, 0)
      val parameter: String = "TMP"
      val ltvid: String = "TGL:2"     // TGL = "specified height level above ground" & 2 = 2 meters
      val lat: Double = 50.7
      val lon: Double = 362.29

      val record: GribRecord = gribFile.getRecord(date, parameter, ltvid)
      val value: Double = record.getValue(lat, lon)
      println("Value at (" + lat + ", " + lon + ") : " + value + "\n")

Thanks.

[Timelessprod]

Using the release 0.4 JAR give same errors

[Timelessprod]

@spidru Here is the GRIB2 file I'm using : https://we.tl/t-l3xbpVrYUv Thanks

[spidru]

Hi @Timelessprod, thanks for sending the file. The issue is that the GRIB file seems to use Simple Packing in its DRS section, whereas at the moment only Complex Packing and Spatial Differencing is supported for GRIB2. I'll try to find some time to implement Simple Packing, shouldn't be too much work.

[Timelessprod]

Hi @Timelessprod, thanks for sending the file. The issue is that the GRIB file seems to use Simple Packing in its DRS section, whereas at the moment only Complex Packing and Spatial Differencing is supported for GRIB2. I'll try to find some time to implement Simple Packing, shouldn't be too much work.

@spidru Thanks for your reply and for adapting your library. Do you have an idea of when it would be ready ? So I can tell my manager when my product will be available.

Thank you very much.

[spidru]

Hi @Timelessprod, I'm also seeing that the file you shared also uses bitmap data, which is another thing that isn't supported at the moment. I'll try to get this done in a week or so, but it will depend on my availability.

[Timelessprod]

Ok thank you a lot !

[spidru]

Hi @Timelessprod, I've just pushed an initial version which include support for GRIB files specifying a bitmap as well as files using grid-point simple packing. Can you try it out?

https://83-92336207-gh.circle-artifacts.com/0/build/libs/JGribX.jar

[Timelessprod]

Hi @spidru. I'm not at work now but I'll try it tomorrow. Thanks for your time and fast reaction !

[Timelessprod]

Hi @spidru, I've update my JAR with your new one but it uses a more recent Java version than the one I can use at work (JDK 8 for Scala 1.12). Thus the code crash with this error :

java.lang.UnsupportedClassVersionError: mt/edu/um/cf2/jgribx/GribFile has been compiled by a more recent version of the Java Runtime (class file version 53.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at com.databricks.backend.daemon.driver.ClassLoaders$ReplWrappingClassLoader.loadClass(ClassLoaders.scala:65)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)

[spidru]

Hi @Timelessprod, the project is currently being built with JDK 9 by default. Here's a version compiled with JDK 8 that should work for you.

https://93-92336207-gh.circle-artifacts.com/0/build/libs/JGribX.jar

[Timelessprod]

Thank you ! I'll try it now

[Timelessprod]

After trying it yesterday and this morning, I'm having errors with it when using Spark dataframes as it seems that some functions of the library are not serializable. When I try to call the show() method on my dataframe after filling it with GRIB data, I get this stack trace :

org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:416)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:406)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2604)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$1(RDD.scala:893)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:165)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:125)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:395)
    at org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:892)
    at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:727)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:200)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$3(SparkPlan.scala:252)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:165)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:248)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:192)
    at org.apache.spark.sql.execution.UnionExec.$anonfun$doExecute$5(basicPhysicalOperators.scala:684)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.map(TraversableLike.scala:238)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
    at org.apache.spark.sql.execution.UnionExec.doExecute(basicPhysicalOperators.scala:684)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:200)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$3(SparkPlan.scala:252)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:165)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:248)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:192)
    at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:79)
    at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:88)
    at org.apache.spark.sql.execution.ResultCacheManager.getOrComputeResult(ResultCacheManager.scala:508)
    at org.apache.spark.sql.execution.CollectLimitExec.executeCollectResult(limit.scala:58)
    at org.apache.spark.sql.Dataset.collectResult(Dataset.scala:2994)
    at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3717)
    at org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2718)
    at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3709)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:116)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:249)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:101)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:845)
    at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:199)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3707)
    at org.apache.spark.sql.Dataset.head(Dataset.scala:2718)
    at org.apache.spark.sql.Dataset.take(Dataset.scala:2925)
    at org.apache.spark.sql.Dataset.getRows(Dataset.scala:307)
    at org.apache.spark.sql.Dataset.showString(Dataset.scala:344)
    at org.apache.spark.sql.Dataset.show(Dataset.scala:840)
    at org.apache.spark.sql.Dataset.show(Dataset.scala:799)

If I try not to print the table, then the dataframe won't be filled with data as Spark use lazy evaluation. So maybe it would requier you to refine almost all of the lib which I don't want you to do, you've helped me enough and I don't want you to have to rewrite your whoe library only for my usecase. However, I also get some NoValidGrbException on some GRIB files such as this one https://we.tl/t-exk4aytqYs. Such files give me this stack trace :

mt.edu.um.cf2.jgribx.NoValidGribException: No valid GRIB records found.
    at mt.edu.um.cf2.jgribx.GribFile.<init>(GribFile.java:138)
    at mt.edu.um.cf2.jgribx.GribFile.<init>(GribFile.java:80)
    at mt.edu.um.cf2.jgribx.GribFile.<init>(GribFile.java:63)

Maybe it's something to try to fix, isn't it ?

Thanks for all the support you've gave me this far.

[spidru]

Hmm not quite sure how I would go about that serializable issue. If it's a blocking issue let me know and I'll have to look into it in some detail.

In the meantime, I'll find some time this week to have a look at your new GRIB file. Most probably there's something which is not yet supported. You can actually check that using the JGribX CLI as follows: java -jar JGribX.jar -i inputfile.grb2 -l 4.

[spidru]

Hi @Timelessprod, apologies for the long delay. Is this issue still relevant? If so, could you please re-share the GRIB file giving you a NoValidGribException?

[Timelessprod]

Hi @spidru I'm sorry to tell you that I have no longer access to this file as it was on my professional computer and that my internship has ended 5 months ago. I'm sorry for the inconvenience.

[spidru]

No problem at all. In that case, I'll close this issue since there is no further action to be done.