AgroMonitor API

[geofbaum]

Not sure if it is a current issue with AgroMonitor or OpenWeatherMap in general but I keep on getting a time out error when trying to download satellite imagery as in the example in the docs.

https://pyowm.readthedocs.io/en/latest/usage-examples-v2/agro-api-usage-examples.html#operations-summary

It seems that I am capable of getting one image without the error though when I then go to get the stats or view the image it's blank as well. Checking the length of the returned results shows that there should be 5 images available for the time frame that I searched in my polygon.

Tested on Win7 in a JupyterLab environment, Python 3.6, also sent a request to OWM help desk info just to check if there's an issue on their end.

[csparpa]

Hi, please keep the issue up to date with the response from OWM helpdesk. thanks!

[geofbaum]

@csparpa Hi, I sent them a copy of my code using just the regular request code. I haven't heard back from them as of yet. However I can confirm that I am able to now download a .tiff or .png files from the API after parsing a regular request and extracting the embedded URL. The only difference is that I need to change my request to get each of the images as it

Range of Dates: Start: 1546750800 (2019-01-06) End: 1554782400 (2019-04-09)

This is the response I get from pyowm:

<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-02-01 00:00:00+00 by Landsat 8 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-02-10 00:00:00+00 by Landsat 8 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-02-17 00:00:00+00 by Landsat 8 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-02-26 00:00:00+00 by Landsat 8 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-03-14 00:00:00+00 by Landsat 8 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-02-10 00:00:00+00 by Sentinel-2 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-03-12 00:00:00+00 by Sentinel-2 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-03-27 00:00:00+00 by Sentinel-2 on polygon with id=5cab8a27d861700028090812>
<pyowm.agroapi10.imagery.MetaGeoTiffImage - GEOTIFF ndvi image acquired at 2019-04-01 00:00:00+00 by Sentinel-2 on polygon with id=5cab8a27d861700028090812>

From a direct url request from the API endpoint I get the correct corresponding dates to the dates that are in the list above, so I'm not sure why the download through pyowm is not working but I am able to loop through my request and get the images directly through my request.

If I hear something back from OWM I will update this with their response.

[csparpa]

@geofbaum it would be far quicker for me to understand what's going on if you shared the Python code of yours that performs the failing request... would you mind posting it here?

Also please make sure to embed details about your actual Polygon coordinates cause I need to replicate it in order to obtain the same images as in your case

Thanks

[geofbaum]

No problem worries, I'll attach the code below. Also just to add, the response from OWM was that they don't think it's an issue on their end as if I print out the urls for the images that pyowm would be requesting, I can download them just as I can download them from my request with just the requests lib.

The polygon I tested this with is the following one: polygon = GeoPolygon([[[-73.62634,45.18234],[-73.60685,45.17112],[-73.60496,45.17284],[-73.60754,45.18385],[-73.6217,45.19214],[-73.62419,45.18945],[-73.62634,45.18234]]])

pol_id =  'id of the above polygon'
acq_from = 1546750800                # 06 January 2019
acq_to = 1551157200                 # 26 February 2019
img_type = ImageTypeEnum.GEOTIFF     # the image format type
preset = PresetEnum.NDVI    # the preset
sat = SatelliteEnum.LANDSAT_8.symbol # the satellite

results = mgr.search_satellite_imagery(pol_id, acq_from, acq_to, img_type=img_type, preset=preset, acquired_by=sat)

satellite_images = [mgr.download_satellite_image(result) for result in results]

It times out during the download process, there should be four images I believe for the range of dates, and from adding in a statement into the http_client.py file it looks like at least on my end that it's timing out after the third of the four images.

If I increase the range of the acquisition dates, to today's date 1554868800 there is an additional image available from Landsat8 and if I try to directly download it from the results list with sat_img_last = mgr.download_satellite_image(results[4]) I also get a timeout error for that one as well.

Let me know if I can give you any more info.

[csparpa]

@geofbaum thanks for the code, I've been able to debug it

Two issues here: one is related to the timeout, the other on to the blank downloaded files.

Timeout

PyOWM has a configuration module with all the config values for it, including the timeout for HTTP requests. This is currently set to: 2 seconds. Which looks, honestly, a bit too small for downloading satellite images... Therefore I've thought to increase it to 5 seconds as the default for the future PyOWM v3

You can nevertheless manually configure the timeout value for your local PyOWM installation: just locate the pyowm.weatherapi25.configuration25 module and tweak the API_AVAILABILITY_TIMEOUT value to whatever you like - eg. 5 seconds.

Or, alternatively, just clone the PyOWM develop branch and use it...

So this is explained and possibly fixed

Blank files

This is actually puzzling.

I've tried to manually download (as you youself did) the four GeoTIFF images resulting from your query. I've tried first via web browser and then via wget:

wget http://api.agromonitoring.com/data/1.0/0225c538c00/5caf769bd86170003e0908dd?appid=xxxxxxxxxx -O 0.tif
wget http://api.agromonitoring.com/data/1.0/0225c5f6980/5caf769bd86170003e0908dd?appid=xxxxxxxxxx -O 1.tif
wget http://api.agromonitoring.com/data/1.0/0225c68a400/5caf769bd86170003e0908dd?appid=xxxxxxxxxx -O 2.tif
wget http://api.agromonitoring.com/data/1.0/0225c748180/5caf769bd86170003e0908dd?appid=xxxxxxxxxx -O 3.tif

Same outcome: blank tiff files

When I try to download the very same images using PyOWM:

# polygon is built with: GeoPolygon([[[-73.62634,45.18234],[-73.60685,45.17112],[-73.60496,45.17284],[-73.60754,45.18385],[-73.6217,45.19214],[-73.62419,45.18945],[-73.62634,45.18234]]])

pol_id =  polygon.id
acq_from = 1546750800                # 06 January 2019
acq_to = 1551157200                 # 26 February 2019
img_type = ImageTypeEnum.GEOTIFF     # the image format type
preset = PresetEnum.NDVI    # the preset
sat = SatelliteEnum.LANDSAT_8.symbol # the satellite

results = mgr.search_satellite_imagery(pol_id, acq_from, acq_to, img_type=img_type, preset=preset, acquired_by=sat)
satellite_images = [mgr.download_satellite_image(result) for result in results]
for index, img in enumerate(satellite_images):
    img.persist('/home/claudio/i_{}.tif'.format(index))

I keep on getting blank tiff files

Can you confirm this behaviour?

[csparpa]

I keep on getting blank tiff files

This is not really true, Claudio :)

@geofbaum I've installed the exiftools package on Linux and here is what it tells us for one the images that I've downloaded with PyOWM:

$ exiftool i_0.tif 
ExifTool Version Number         : 11.35
File Name                       : i_0.tif
Directory                       : .
File Size                       : 85 kB
File Modification Date/Time     : 2019:04:11 21:51:05+02:00
File Access Date/Time           : 2019:04:11 21:56:02+02:00
File Inode Change Date/Time     : 2019:04:11 21:51:05+02:00
File Permissions                : rw-rw-r--
File Type                       : TIFF
File Type Extension             : tif
MIME Type                       : image/tiff
Exif Byte Order                 : Little-endian (Intel, II)
Image Width                     : 125
Image Height                    : 173
Bits Per Sample                 : 32
Compression                     : Uncompressed
Photometric Interpretation      : BlackIsZero
Strip Offsets                   : (Binary data 62 bytes, use -b option to extract)
Samples Per Pixel               : 1
Rows Per Strip                  : 16
Strip Byte Counts               : (Binary data 54 bytes, use -b option to extract)
Planar Configuration            : Chunky
Sample Format                   : Float
Pixel Scale                     : 19.1092570712941 19.1092570712941 0
Model Tie Point                 : 0 0 0 -8196055.90416338 5651811.19917715 0
GDAL No Data                    : -9999
Geo Tiff Version                : 1.1.0
GT Model Type                   : Projected
GT Raster Type                  : Pixel Is Area
GT Citation                     : WGS 84 / Pseudo-Mercator
Geog Citation                   : WGS 84
Geog Angular Units              : Angular Degree
Projected CS Type               : Unknown (3857)
Proj Linear Units               : Linear Meter
Image Size                      : 125x173
Megapixels                      : 0.022

This means the img is not actually blank Nevertheless I'm having a hard time opening it in GIMP or other image viewers... I would like to try out with Quantum GIS or ArcGIS as well

[geofbaum]

The images are fine as I was able to open them in Paint, Windows Picture Viewer, and QGIS 3.6 and I changed the Timeout value to 5 and it seems to work now fine. However, I just tested it on my home laptop so it's not the same network or system I was working on the previous two days. I say that solely because I thought I had tested changing that yesterday and didn't get a difference. I will give it a try again tomorrow on the other system and see if that fixes it. If it does, I'll make sure to let you guys know as for now though I think the issue is fixed. Thanks for all the help!

[geofbaum]

Just to confirm, after changing the timeout delay to 5 on the system the issue originated on I am able to acquire the images. Thank again!