remi-braun
commented
3 years ago Hello,
It seems something went wrong with Snap 🙁
What is your Snap version ? GPT doesn't seem to recognize the -q option and this is an error I ´ve never seen before
Closed paron2407 closed 2 years ago
remi-braun
commented
3 years ago Hello,
It seems something went wrong with Snap 🙁
What is your Snap version ? GPT doesn't seem to recognize the -q option and this is an error I ´ve never seen before
paron2407
commented
3 years ago Hi,
I have really problems to make this running. Do you have a quick guide to transform to geo coordinates of sentinel 1 SAR data?
Kind regards,
remi-braun
commented
3 years ago Hi,
I need to know your Snap version in order to investigate what is going on 🙂
remi-braun
commented
3 years ago Hi,
I have really problems to make this running. Do you have a quick guide to transform to geo coordinates of sentinel 1 SAR data?
Kind regards,
This is done by default when loading a band 😃 EOReader always works in projected coordinates But EOReader needs ESA's SNAP software to achieve it This is where the problem is Have you intalled it ?
paron2407
commented
3 years ago Hi First thanks for the package, as it solves 80 % of my problems up to now and is able to solve 99.999%. However I have some simple questions to set your environment up. Therefore some simple questions: My configuration is SNAP (newest version as of today) (It could be that I use python 3.7, which version do you use?)
Simple questions 1.) SNAP gpt Your documentation says: "Sentinel-3 and SAR products need SNAP gpt to be geocoded. Ensure that you have the folder containing your gpt executable in your PATH. How do I get it? Do I need to extract it from SNAP and does it work in all python versions. 2.) DEM Bands: Your Documentation says: "These bands need a valid worldwide DEM path positioned thanks to the environment variable EOREADER_DEM_PATH. You can use both a local path e.g. /mnt/dataserver/dems/srtm_30_v4/index.vrt or \dataserver\DEMS\srtm_30_v4\index.vrt or a URL pointing to a web resources hosted on a S3 compatible storage e.g. https://s3.storage.com/dem-bucket/srtm_cog.tif (not available on Windows for now)." How can I get a index.vrt file (DEM path file)
My current problem is that my application should be fully automatic without any user interaction in between, therefore some manual work does not benefit the project.
paron2407
commented
3 years ago Ok some further information. I have now managed to install snappy with python version 3.6, as described in the ESA tutorial. However when I now want to install eoreader, I cannot install it as it requires python 3.7. Error message: Specifications:
Your python: python=3.6
remi-braun
commented
3 years ago Glad to hear that ! To reassure you, EOReader is used by us in fully automatic processing chains, it is designed to work in that way 🙂
Concerning your problems:
gpt is delivered with Snap. Just localize it on disk (ie which gpt on Linux) and add its folder to the PATH of your computer. If the gpt command already works in your shell you already have it. remi-braun
commented
3 years ago Ok some further information. I have now managed to install snappy with python version 3.6, as described in the ESA tutorial. However when I now want to install eoreader, I cannot install it as it requires python 3.7.
Error message:
Specifications:
- eoreader -> python[version='>=3.7']
Your python: python=3.6
You don't need snappy, the python implementation of GPT is crap We use the graph and command line directly 😉
paron2407
commented
3 years ago Hi, Thanks for the help. Yes snappy is really crappy. It runs now, but it is not the best SAR post processing tool(performance vice). However I managed it also now to come a bit further with eoreader and tried to open a Sentinel 1 SAR image from Copernicus OpenHub. With the following code: path = "/media/michael/Sicherheit/S1B_IW_GRDH_1SDV_20210731T171709_20210731T171734_028041_035850_C4F8.zip" eoreader = Reader() prod = eoreader.open(path) print(f"Acquisition datetime: {prod.datetime}") print(f"Condensed name: {prod.condensed_name}") from eoreader.env_vars import DEM_PATH os.environ[DEM_PATH] = os.path.join("SRTM_30m_v4") bands = [VV, HH, VV_DSPK, HH_DSPK, HILLSHADE, SLOPE] ok_bands = [band for band in bands if prod.has_band(band)] print(to_str(ok_bands)) band_dict = prod.load([VV])
I get then the following error message: ERROR 1: PROJ: proj_create_from_database: /home/michael/anaconda3/envs/opencv_tut/share/proj/proj.db lacks DATABASE.LAYOUT.VERSION.MAJOR / DATABASE.LAYOUT.VERSION.MINOR metadata. It comes from another PROJ installation. Traceback (most recent call last): File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 442, in get_band_paths exact_name=True, File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/sertit/files.py", line 986, in get_file_in_dir f"File with pattern {glob_pattern} not found in {directory}" FileNotFoundError: File with pattern *20210731T171709_S1_IW_GRD_VV.tif not found in /tmp/tmpcqy6gb35/tmp_20210731T171709_S1_IW_GRD
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "rasterio/_crs.pyx", line 285, in rasterio._crs._CRS.from_epsg File "rasterio/_err.pyx", line 192, in rasterio._err.exc_wrap_int rasterio._err.CPLE_AppDefinedError: PROJ: proj_create_from_database: /home/michael/anaconda3/envs/opencv_tut/share/proj/proj.db lacks DATABASE.LAYOUT.VERSION.MAJOR / DATABASE.LAYOUT.VERSION.MINOR metadata. It comes from another PROJ installation.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/michael/PycharmProjects/opencv_tut/ship.py", line 358, in
It seams as if rasterio still has a problem
remi-braun
commented
3 years ago This is a problem coming from your installation I think. How do you installed EOReader ? You installed rasterio and geopandas separatly ? Those libs both using GDAL may have incompatibilities. We are not familiar with the conda use as we direclty use pip on Docker
remi-braun
commented
3 years ago This may help you: #2094
paron2407
commented
3 years ago Mhh I have not uninstalled eoreader geopandas and rasterio. Afterwards I have just used pip3 eoreader and the failure is still the same. The question is, whether is is a problem with the EPSG code. Can I change/append it?
obj._crs = _CRS.from_epsg(code)File "rasterio/_crs.pyx", line 287, in rasterio._crs._CRS.from_epsg rasterio.errors.CRSError: The EPSG code is unknown. PROJ: proj_create_from_database: /home/michael/anaconda3/envs/opencv_tut/share/proj/proj.db lacks DATABASE.LAYOUT.VERSION.MAJOR / DATABASE.LAYOUT.VERSION.MINOR metadata. It comes from another PROJ installation.
remi-braun
commented
3 years ago Here the problem is not a real bug, this is "just" your Proj of your environment that cannot access its own projection database (proj.db) that is stored somewhere in your anaconda environment. It accesses another database from another proj that has not the same version. But activating your environment should link it dynamically 🤔 Have you an environment variable set from elsewhere on your computer ?
I would suggest you to recreate your env from scratch as these things can happen when you are modifying it too much (conda and environment variables are really not friends)
Note this is not an EOReader issue, more a common gdal x conda one 😩
paron2407
commented
3 years ago Hello,
Thanks for the hint. I have now set up a new environment and it worked, up to the point, where something crashed after approx. 20 % during loading of a band. :
Acquisition datetime: 2021-07-31 17:17:09 Condensed name: 20210731T171709_S1_IW_GRD ['VV', 'VV_DSPK', 'SLOPE'] Executing processing graph WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 31-JUL-2021 17:16:00.000000 Orbit files may be downloaded from https://scihub.copernicus.eu/gnss/odata/v1/ and placed in /home/michael/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1B/2021/07 WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Using Sentinel Restituted /home/michael/.snap/auxdata/Orbits/Sentinel-1/RESORB/S1B/2021/07/S1B_OPER_AUX_RESORB_OPOD_20210731T205830_V20210731T170218_20210731T201948.EOF.zip instead version = 3.31 ....10%....20%Killed Traceback (most recent call last): File "/home/michael/PycharmProjects/OpenSeaOverSight/venv/lib/python3.8/site-packages/eoreader/products/sar/sar_product.py", line 439, in get_band_paths band_paths[band] = files.get_file_in_dir( File "/home/michael/PycharmProjects/OpenSeaOverSight/venv/lib/python3.8/site-packages/sertit/files.py", line 985, in get_file_in_dir raise FileNotFoundError( FileNotFoundError: File with pattern *20210731T171709_S1_IW_GRD_VV_DSPK.tif not found in /tmp/tmpkrrv5ei9/tmp_20210731T171709_S1_IW_GRD
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/michael/PycharmProjects/OpenSeaOverSight/venv/lib/python3.8/site-packages/eoreader/products/sar/sar_product.py", line 439, in get_band_paths band_paths[band] = files.get_file_in_dir( File "/home/michael/PycharmProjects/OpenSeaOverSight/venv/lib/python3.8/site-packages/sertit/files.py", line 985, in get_file_in_dir raise FileNotFoundError( FileNotFoundError: File with pattern *20210731T171709_S1_IW_GRD_VV.tif not found in /tmp/tmpkrrv5ei9/tmp_20210731T171709_S1_IW_GRD
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/michael/PycharmProjects/OpenSeaOverSight/venv/lib/python3.8/site-packages/eoreader/products/sar/sar_product.py", line 787, in _pre_process_sar misc.run_cli(cmd_list) File "/home/michael/PycharmProjects/OpenSeaOverSight/venv/lib/python3.8/site-packages/sertit/misc.py", line 354, in run_cli raise RuntimeError(f"Exe {cmd[0]} has failed.") RuntimeError: Exe gpt has failed.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/michael/PycharmProjects/OpenSeaOverSight/main.py", line 31, in
remi-braun
commented
3 years ago This is very very weird... it says "Killed" as if the process shut itself down...
Can you relaunch the command when monitoring the memory ? Does it saturate ? I've never seen this Snap bug even on computer with little computational power 🙁
How many CPU and RAM do you have ?
paron2407
commented
3 years ago Hi,
I have 16 gb RAM and have now extended the swap to 8 gb. Core: Intel i7 4 x 2.6. I have managed to come a bit further, but still it crashes. Is it nominal, that 24 gb of RAM is needed for a 700 mb SAR image?
Acquisition datetime: 2021-07-31 17:17:09
Condensed name: 20210731T171709_S1_IW_GRD
['VV', 'VV_DSPK', 'SLOPE']
Executing processing graph
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 31-JUL-2021 17:16:00.000000
Orbit files may be downloaded from https://scihub.copernicus.eu/gnss/odata/v1/
and placed in /home/michael/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1B/2021/07
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Using Sentinel Restituted /home/michael/.snap/auxdata/Orbits/Sentinel-1/RESORB/S1B/2021/07/S1B_OPER_AUX_RESORB_OPOD_20210731T205830_V20210731T170218_20210731T201948.EOF.zip instead
version = 3.31
....10%....20%....30%....40%....50%....60%....70%....80%....90% done.
More than one file corresponding to the pattern .img has been found here /tmp/tmpixg25jq3/20210731T171709_S1_IW_GRD.data. Only the first item will be returned.
More than one file corresponding to the pattern .img has been found here /tmp/tmpixg25jq3/20210731T171709_S1_IW_GRD.data. Only the first item will be returned.
Executing processing graph
SEVERE: com.sun.xml.bind.v2.util.XmlFactory: null
org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
at org.apache.xerces.parsers.AbstractSAXParser.setFeature(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.setFeatures(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.
first_line_time metadata value is null last_line_time metadata value is null ....10%....20%..
remi-braun
commented
3 years ago Ok this time the first process finishes (orthorectifying the SAR data) but Snap still crashes
I'm no expert with Snap configuration, so I don't really know what is going on here.
I've never had such crashes with Snap, but it seems it's a problem on their side. I think it has sth to do with the memory (SAR processing in memory intensive) Can you try it on a better computer to see if this is the problem ? I must say I never tried to orthorectify SAR data with less than 32Go of memory
remi-braun
commented
3 years ago Oh I found it, see this solution This should fix it 🙂
paron2407
commented
3 years ago Hello,
thanks for the effort. Unfortunately this does not change the situation. Is there no option from eoreader available to read more efficient. I have a 700 mb file and need about 30 gb of ram to process it. There must be something wrong. When I do the same operations in the SNAP gui it works good, but this does not help me unfortunately.
Maybe I can overpass some functions: essentially I need to:
remi-braun
commented
3 years ago In EOReader you can only load projected bands so the function load returns an orthorectified VV band. This is why the computation when loading is heavy. The only line you need to read the orthorectified VV band as an xarray is prod.load(VV)
However you are right, needing 32Go to ortho a SAR image seems too much. But I've never tested with less memory as I do not have access to light computer like that. Note that we use Snap directly, so if this doesn't work I would say it is a problem with Snap rather than EOReader.
If you set the log to DEBUG, you will see the command line that is ran by gpt, so you could test it in your shell if you want to run it outside of EOReader
However, the bug shown by last traceback you sent is resolved by the link I gave you, as I corrected that issue on my computer.
I am really surprised that using Snap is so hard for you, this should be quite straightforward. Have you modified any preferences on your computer ?
remi-braun
commented
2 years ago Closing due to no response. Please re-open or comment if you have more information about this issue. Thanks!
I have an issue after trying to opening a sentinel 1 image: Code: l5_path = r"/media/michael/Sicherheit/S1B_IW_GRDH_1SDV_20210731T171709_20210731T171734_028041_035850_C4F8.zip" os.environ[DEM_PATH] = os.path.join("/media/michael/Sicherheit", "s1b-iw-grd-vv-20210731t171709-20210731t171734-028041-035850-001.tiff") eoreader = Reader() l5_prod = eoreader.open(l5_path) band_dict = l5_prod.load([VV])
Failure: gpt: opção inválida: -q Traceback (most recent call last): File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 442, in get_band_paths exact_name=True, File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/sertit/files.py", line 986, in get_file_in_dir f"File with pattern {glob_pattern} not found in {directory}" FileNotFoundError: File with pattern *20210731T171709_S1_IW_GRD_VV.tif not found in /tmp/tmpl5_4fb7c/tmp_20210731T171709_S1_IW_GRD
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 787, in _pre_process_sar misc.run_cli(cmd_list) File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/sertit/misc.py", line 354, in run_cli raise RuntimeError(f"Exe {cmd[0]} has failed.") RuntimeError: Exe gpt has failed.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/michael/PycharmProjects/opencv_tut/ship.py", line 255, in
band_dict = l5_prod.load([VV])
File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/product.py", line 652, in load
band_dict = self._load(bands, resolution, size)
File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 680, in _load
bands = self._load_bands(band_list, resolution=resolution, size=size)
File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 622, in _load_bands
band_paths = self.get_band_paths(bands, resolution)
File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 451, in get_band_paths
all_band_paths = self._pre_process_sar(resolution)
File "/home/michael/anaconda3/envs/opencv_tut/lib/python3.7/site-packages/eoreader/products/sar/sar_product.py", line 789, in _pre_process_sar
raise RuntimeError("Something went wrong with SNAP!") from ex
RuntimeError: Something went wrong with SNAP!