pyroSAR appearing to look for a tmp file before it's created

davedavemckay commented 3 years ago

which operating system are you using?
RHEL
which environment is pyroSAR running in?
user-level installation of Python 3.9.2
which version of pyroSAR are you using?
clone of the GitHub repository
which function of pyroSAR did you call with which parameters?
geocode
if applicable, which version of SNAP or GAMMA are you using in pyroSAR? SNAP 8.0, Sentinel-1 Toolbox 8.0.4
the full error message [Errno 2] No such file or directory: '[...]/T30VUJ/2021/03/29/pyrosar/S1B__IW___D_20210329T063721_NR_Orb_Cal_TF_TC_dB/tmp/S1B_IW_GRDH_1SDV_20210329T063721_20210329T063746_026226_03213E_AA5A_tmp3.dim'

geocode call: geocode(scene, outdir='%s' % (path2pyrosar), t_srs=crs_epsg, tr=resolution, scaling='dB', polarizations='all', geocoding_type='Range-Doppler', removeS1BorderNoise=True, removeS1BorderNoiseMethod='pyroSAR', removeS1ThermalNoise=True, terrainFlattening=terrainFlattening, externalDEMFile=dem_file, externalDEMNoDataValue=dem_ndv, externalDEMApplyEGM=True, speckleFilter=speckleFilter, refarea='gamma0', shapefile=granule_shp, alignToStandardGrid = True, standardGridOriginX = origin[0], standardGridOriginY = origin[1])

johntruckenbrodt commented 3 years ago

Hi @davedavemckay, thanks for the issue. I was not able to reproduce this but the error should no longer occur with commit 0e5e62b. Could you please check again? @maawoo, could you also check since you got a similar error?

maawoo commented 3 years ago

Now my process fails during Apply-Orbit-File even though POEORB files were downloaded manually at some point beforehand. Wasn't the problem regarding the qc.sentinel1.eo.esa.int API resolved as stated in #131? Or am I missing something?

Start processing of scenes for tile '32TMT':
['C:\\Users\\du23yow\\Documents\\COPA\\data\\S1B_IW_GRDH_1SDV_20210601T171435_20210601T171500_027166_033EB1_C785.zip', 'C:\\Users\\du23yow\\Documents\\COPA\\data\\S1B_IW_GRDH_1SDV_20210601T171500_20210601T171525_027166_033EB1_161A.zip']
executing node sequences..
ThermalNoiseRemoval
ThermalNoiseRemoval
SliceAssembly
Remove-GRD-Border-Noise
Apply-Orbit-File
Executing processing graph
Exception calling QC Rest API:  qc.sentinel1.eo.esa.int
Exception calling QC Rest API:  qc.sentinel1.eo.esa.int
qc.sentinel1.eo.esa.int
 done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.3.1 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: qc.sentinel1.eo.esa.int
org.esa.snap.core.gpf.graph.GraphException: [NodeId: Apply-Orbit-File] qc.sentinel1.eo.esa.int
        at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
        at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
        at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
        at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
        at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
        at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
        at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
        at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
        at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
        at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
        at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
        at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
        at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:94)
        at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:25)
Caused by: org.esa.snap.core.gpf.OperatorException: qc.sentinel1.eo.esa.int
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp.initialize(ApplyOrbitFileOp.java:215)
        at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
        at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
        at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
        at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
        ... 29 more
Caused by: java.net.UnknownHostException: qc.sentinel1.eo.esa.int
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
        at java.net.InetAddress.getAllByName(InetAddress.java:1193)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.esa.s1tbx.io.orbits.sentinel1.SentinelPODOrbitFile.downloadFromQCRestAPI(SentinelPODOrbitFile.java:173)
        at org.esa.s1tbx.io.orbits.sentinel1.SentinelPODOrbitFile.retrieveOrbitFile(SentinelPODOrbitFile.java:112)
        at org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp.updateOrbits(ApplyOrbitFileOp.java:269)
        at org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp.initialize(ApplyOrbitFileOp.java:203)
        ... 33 more

Error: [NodeId: Apply-Orbit-File] qc.sentinel1.eo.esa.int

[S1B_IW_GRDH_1SDV_20210601T171435_20210601T171500_027166_033EB1_C785_tmp4.xml] failed with return code 1
failed: C:\Users\du23yow\Documents\COPA\data\proc_out\32TMT\S1B__IW___A_20210601T171435_NR_Asm_Orb_Cal_TF_TC_proc.xml

maawoo commented 3 years ago

Now my process fails during Apply-Orbit-File even though POEORB files were downloaded manually at some point beforehand. Wasn't the problem regarding the qc.sentinel1.eo.esa.int API resolved as stated in #131? Or am I missing something?

Resolved after updating SNAP 👍🏻

johntruckenbrodt commented 3 years ago

Looks like this is solved. Thanks for your contributions.

jthetzel commented 3 years ago

I also encountered this error and had trouble figuring out how to update SNAP in a Dockerfile. In case others are in a similar situation, adding the following as a RUN command after installing SNAP resolved the error.

snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do     
    echo "$line";     
    [ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java"; 
done; 
exit 0

The pkill workaround is needed because the SNAP update-all command hangs after updating ( https://senbox.atlassian.net/wiki/spaces/SNAP/pages/30539785/Update+SNAP+from+the+command+line ). The exit 0 is needed because the workaround was returning a non-zero status causing the Docker build to terminate.

johntruckenbrodt commented 3 years ago

Thanks @jthetzel!

johntruckenbrodt / pyroSAR

pyroSAR appearing to look for a tmp file before it's created #153