ucar.nc2.filter.UnknownFilterException: Unknown filter - v5.6.0

jacobwentzib commented 2 weeks ago

Versions impacted by the bug

5.6.0 - Latest

Environment: UCAR Library Version: cdm-core 5.6.0 Java Version: 21

What went wrong?

Recently upgraded to 5.6.0 and see the following exception being thrown where the ServiceLoader is unable to load Filters properly.

https://github.com/Unidata/netcdf-java/blob/42186c77b9d424aeb45e333be80593c7a03c608e/cdm/core/src/main/java/ucar/nc2/filter/Filters.java#L80

Once we downgrade back to 5.5.3 everything works properly.

Has there been any changes to classpath or anything else that could be causes this issue? We are seeing this isolated to jar executions but IDE development with target classes runs properly.

We have created a custom ServiceLoader within our application and can see the resources being found.

Relevant stack trace

Being invoked from: Variable.class:

    public Array read(Section section) throws IOException, InvalidRangeException {
        return section == null ? this._read() : this._read(Section.fill(section, this.shape));
    }

Error throw from above link: java.io.IOException: ucar.nc2.filter.UnknownFilterException: Unknown filter: no filter found with id 2

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed? Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

[X] I agree to follow the UCAR/Unidata Code of Conduct

tdrwenski commented 2 weeks ago

Can you add the full error stack trace?
Are any FilterProviders getting loaded (yours or NetCDF-java's)? It looks like the error is about "id 2" which should be the Shuffle filter.

jacobwentzib commented 2 weeks ago

NetCDF-java's FilterProviders.

Here is the full trace: java.io.IOException: ucar.nc2.filter.UnknownFilterException: Unknown filter: no filter found with id 2 at ucar.nc2.internal.iosp.hdf5.H5tiledLayoutBB.(H5tiledLayoutBB.java:90) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:226) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:204) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.Variable.reallyRead(Variable.java:797) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.Variable._read(Variable.java:736) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.Variable._read(Variable.java:754) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.Variable.read(Variable.java:600) ~[cdm-core-5.6.0.jar!/:5.6.0] at .GoesrDecoder.extractDataFromVariable(GoesrDecoder.java:376) ~[goesr-0.0.0.jar!/:0.0.0] at .GoesrDecoder.extractDataAsync(GoesrDecoder.java:320) ~[goesr-0.0.0.jar!/:0.0.0] at .GoesrDecoder.decode(GoesrDecoder.java:145) ~[goesr-0.0.0.jar!/:0.0.0] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379) ~[spring-tx-6.1.11.jar!/:6.1.11] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-6.1.11.jar!/:6.1.11] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:113) ~[spring-aop-6.1.11.jar!/:6.1.11] at org.springframework.util.concurrent.FutureUtils.lambda$toSupplier$0(FutureUtils.java:74) ~[spring-core-6.1.11.jar!/:6.1.11] at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[na:na] at java.base/java.lang.VirtualThread.run(VirtualThread.java:311) ~[na:na] Caused by: ucar.nc2.filter.UnknownFilterException: Unknown filter: no filter found with id 2 at ucar.nc2.filter.Filters.getFilter(Filters.java:80) ~[cdm-core-5.6.0.jar!/:5.6.0] at ucar.nc2.internal.iosp.hdf5.H5tiledLayoutBB.(H5tiledLayoutBB.java:88) ~[cdm-core-5.6.0.jar!/:5.6.0] ... 24 common frames omitted 

tdrwenski commented 2 weeks ago

Thanks for sending that. I am not aware of any changes between version 5.5.3 and 5.6.0 that may have caused this issue. And I am not yet able to reproduce this behavior. Just to be sure I am testing the same situation:

is it correct you are using the artifact cdm-core-5.6.0.jar?
Can you check that you have the netcdf-java filters in that jar? By doing e.g. jar tf cdm-core-5.6.0.jar | grep META-INF/services/ucar.nc2.filter.FilterProvider
is your custom Filter implemented similar to the example here and registered via a META-INF/services/ucar.nc2.filter.FilterProvider file?

jacobwentzib commented 1 week ago

Correct we are utilizing: cdm-core-5.6.0.jar
Directory is Empty in META-INF/services for the dependency.
We do not have any custom Filter we are just trying to leverage existing.

tdrwenski commented 1 week ago

Thanks for checking that. I don't know how it's possible that your cdm-core-5.6.0.jar is missing those filters. Are you getting your artifacts from our artifacts server: https://artifacts.unidata.ucar.edu/repository/unidata-all/? Can you try replacing that dependency? For instance, by downloading it manually from here?

jacobwentzib commented 1 week ago

the pom.xml:

  <properties>
    <edu.ucar.version>5.6.0</edu.ucar.version>
  </properties>

  ...
    <dependency>
      <groupId>edu.ucar</groupId>
      <artifactId>cdm-core</artifactId>
      <version>${edu.ucar.version}</version>
      <exclusions>
        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    ... 
      <repositories>
        <repository>
          <id>unidata-all</id>
          <name>Unidata All</name>
          <url>https://artifacts.unidata.ucar.edu/repository/unidata-all/</url>
        </repository>
        <repository>
          <id>ucar-unidata</id>
          <url>https://repo.osgeo.org/repository/release/</url>
        </repository>
      <repositories>

Expanding the jar I see the services in the META-INF of the cdm-core jar.

It seems like the SPI is not working when bundled. As mentioned previously this works when ran in IDE with all jars on class path but when we create our jar it gets these issues where it cannot find the Filters.

We do not want to shade our jar to navigate around this.

jacobwentzib commented 1 day ago

Bump on this.

Unidata / netcdf-java