In the geocode function RemoveThermalNoise step failed with a null pointer exception when a shapefile is specified

[antoine-collet]

Hello,

I am having trouble geocoding scenes when specifying a shapefile in order to reduce processing time. There seems to be a problem with the RemoveThermalNoise step (which failed with an exception). It is to note the function is running smoothly if I do not specify the site shapefile.

• which operating system are you using?
  Ubuntu 18.4

• which environment is pyroSAR running in?
  A dockerised environement

• which version of pyroSAR are you using?
  Installed via pip: pyroSAR==0.11

• which function of pyroSAR did you call with which parameters?
  geocode(infile=scene, outdir='./experiment_v1/subset', tr=10, scaling='dB', shapefile='shape.shp')

• if applicable, which version of SNAP or GAMMA are you using in pyroSAR? SNAP version 7.0.0

• the full error message

  
  Executing processing graph
  java.lang.NullPointerException
  done.
  INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
  INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
  INFO: org.hsqldb.persist.Logger: dataFileCache open start
  org.esa.snap.core.gpf.graph.GraphException: [NodeId: ThermalNoiseRemoval] java.lang.NullPointerException
  at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
  at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
  at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
  at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
  at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:91)
  at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:64)
  at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
  at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
  at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
  at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
  at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
  at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
  at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
  at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
  at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
  at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
  at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
  at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
  at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)
  Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
  at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:415)
  at org.esa.s1tbx.calibration.gpf.Sentinel1RemoveThermalNoiseOp.initialize(Sentinel1RemoveThermalNoiseOp.java:165)
  at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
  at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
  at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
  at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
  ... 28 more
  Caused by: java.lang.NullPointerException
  at org.esa.s1tbx.calibration.gpf.Sentinel1RemoveThermalNoiseOp.getProductType(Sentinel1RemoveThermalNoiseOp.java:179)
  at org.esa.s1tbx.calibration.gpf.Sentinel1RemoveThermalNoiseOp.initialize(Sentinel1RemoveThermalNoiseOp.java:140)
  ... 32 more

Error: [NodeId: ThermalNoiseRemoval] java.lang.NullPointerException

[S1B_IW_GRDH_1SDV_20190501T180408_20190501T180433_016054_01E318_F627_tmp1.xml] failed with return code 1 failed: experiment_v1/subset/S1BIW_A_20190501T180408_NR_Orb_TF_TC_dB/S1BIW_A_20190501T180408_NR_Orb_TF_TC_dB_proc.xml

[johntruckenbrodt]

Hi @antoine-collet, thanks a lot for reporting this! I gave it a try myself and am running in the same error. However, running the same workflow in the GUI works fine so it might be specific to GPT. There is some discussion in the STEP forum about something similar here. This discussion resulted in this issue. I think there is unfortunately nothing we can do right now except wait for this issue to be fixed. The workflow itself should work though, I have processed lots of images like this. Cheers, John

[antoine-collet]

Hello @johntruckenbrodt, thanks for the quick reply! I'll be patient then :) Antoine

[vfp1]

Thank you very much for reporting this @antoine-collet, I was running crazy with this error. Will wait for the issue fix. However, I am surprised that there is not proper versioning on the .sh installers, so that we can easily rollback to previous versions. I have one version in local installed few months ago that works flawlessly, but a version installed just yesterday from this url http://step.esa.int/downloads/7.0/installers/esa-snap_sentinel_unix_7_0.sh is causing this error. I am running pipelines in a Docker without GUI access, hence the problem.

[johntruckenbrodt]

Yep, that's been bothering me for a long time. Instead of new features I would love to see some better software packaging. There is not even a proper way to automatically update from the command line, only a workaround written by a former colleague of mine described here.

[vfp1]

Yeah, thanks for the link @johntruckenbrodt, and again, your library is just amazing! Super glad to be working with it! I am working on a fix to subset using external tools (rasterio) so that I can bypass it, let me know if it is of your interest to add it as a workaround. Apparently a similar issue with snap subset happened back in 2018, so I just want to cover my back.

[johntruckenbrodt]

Hi @vfp1 really great you enjoy working with pyroSAR, thanks mate! It sure would be convenient to be able to subset S1 scenes in Python. I wonder though whether it's that easy. Subsetting the tif files should be relatively straight forward but all the metadata files like calibration and noise vectors might have to be adjusted as well. Seems like this is the file containing the relevant SNAP code.
It'd be great if you keep me posted on your findings.

[vfp1]

Thanks! Will keep you posted with this.

[vfp1]

Dear @johntruckenbrodt, is it possible to add more nodes to pyroSAR? I have been trying this for a while, I think one solution will be to add the Land-Sea Mask with vector masking capabilities.

[johntruckenbrodt]

Hi @vfp1, do you mean adding nodes to pyroSAR in general or to the geocode function? pyroSAR has support for all the nodes listed here. If there are some missing I can easily add them. If you want to add new nodes to the geocode workflow, you can run the function with argument test=True so that only the workflow XML is created but not executed. Then you could modify this XML in the SNAP API.

[clausmichele]

I think that my pull request https://github.com/johntruckenbrodt/pyroSAR/pull/123 solves this issue, I've just tried with the sample referred in #122 with removeS1ThermalNoise=True and it worked.

[dilipsai1]

I am still having this issue.Is there any workaround for this?

[johntruckenbrodt]

Wow! Such a long discussion and no fix in sight. Thanks a lot @clausmichele for providing this embarrassingly easy solution!!! I'll merge the PR in a minute.