search

asfadmin / Discovery-asf_search

BSD 3-Clause "New" or "Revised" License
124 stars 43 forks source link

[Bug] Why do I get duplicate products in my response of OPERA RTC-S1 data? #305

Open cmarshak opened 2 months ago

cmarshak commented 2 months ago

So, I am searching OPERA RTC-S1 data (see snippet below) and I am getting rows with the same product repeated twice or more. Am I doing something wrong (or even slightly dumb) or is there an actual issue?

from rasterio.crs import CRS
import geopandas as gpd
import asf_search as asf
from shapely.geometry import box

aoi_bounds = [-72.008,-33.889,-70.049, -32.672]
aoi = box(*aoi_bounds)

resp = asf.geo_search(intersectsWith=aoi.wkt,
                      processingLevel='RTC',
                      polarization=['VV', 'VH'],
                      platform=asf.constants.SENTINEL1)
df = gpd.GeoDataFrame.from_features(resp.geojson(), crs=CRS.from_epsg(4326))
df.sort_values(by='fileID').reset_index(drop=True)
df.fileID.iloc[:2].tolist()

Gives me the following:

['OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0']

Partly sharing this in case my potential mistake helps others. It doesn't seem to be a duplicate product.

https://search.asf.alaska.edu/#/?searchType=List%20Search&searchList=OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0&resultsLoaded=true&granule=OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0

A full display of the records is here:

df.to_dict('records)[:2]

as

[{'geometry': <POLYGON ((-72.513 -33.613, -71.646 -33.397, -71.711 -33.214, -72.576 -33.42...>,
  'centerLat': -33.41339684033409,
  'centerLon': -72.11151852934287,
  'stopTime': '2024-06-20T23:36:52Z',
  'fileID': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
  'flightDirection': 'ASCENDING',
  'pathNumber': 91,
  'processingLevel': 'RTC',
  'url': 'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
  'startTime': '2024-06-20T23:36:49Z',
  'sceneName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
  'browse': ['https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
   'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
   'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png'],
  'platform': 'Sentinel-1A',
  'bytes': {'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5': {'bytes': 103992,
    'format': 'HDF5'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif': {'bytes': 6546248,
    'format': 'GeoTIFF'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif': {'bytes': 6477441,
    'format': 'GeoTIFF'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif': {'bytes': 51929,
    'format': 'GeoTIFF'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml': {'bytes': 161617,
    'format': 'XML'}},
  'md5sum': None,
  'granuleType': None,
  'orbit': 54413,
  'polarization': ['VV', 'VH'],
  'processingDate': '2024-06-21T03:26:22Z',
  'sensor': 'C-SAR',
  'groupID': 'S1A_IWDV_1063_1069_054413_091',
  'pgeVersion': '2.1.1',
  'operaBurstID': 'T091_195252_IW3',
  'validityStartDate': None,
  'subswath': 'IW3',
  'fileName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
  'beamModeType': 'IW',
  's3Urls': ['s3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png.md5',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png.md5',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif.md5'],
  'beamMode': 'IW',
  'additionalUrls': ['https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
   'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
   'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
   'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif'],
  'bistaticDelayCorrection': 'TRUE',
  'noiseCorrection': 'TRUE',
  'postProcessingFilter': 'FALSE'},
 {'geometry': <POLYGON ((-72.513 -33.613, -71.646 -33.397, -71.711 -33.214, -72.576 -33.42...>,
  'centerLat': -33.41339684033409,
  'centerLon': -72.11151852934287,
  'stopTime': '2024-06-20T23:36:52Z',
  'fileID': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
  'flightDirection': 'ASCENDING',
  'pathNumber': 91,
  'processingLevel': 'RTC',
  'url': 'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
  'startTime': '2024-06-20T23:36:49Z',
  'sceneName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0',
  'browse': ['https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png',
   'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png',
   'https://datapool.asf.alaska.edu/BROWSE/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png'],
  'platform': 'Sentinel-1A',
  'bytes': {'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5': {'bytes': 103992,
    'format': 'HDF5'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif': {'bytes': 6546248,
    'format': 'GeoTIFF'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif': {'bytes': 6477441,
    'format': 'GeoTIFF'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif': {'bytes': 51929,
    'format': 'GeoTIFF'},
   'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml': {'bytes': 161617,
    'format': 'XML'}},
  'md5sum': None,
  'granuleType': None,
  'orbit': 54413,
  'polarization': ['VV', 'VH'],
  'processingDate': '2024-06-21T03:26:22Z',
  'sensor': 'C-SAR',
  'groupID': 'S1A_IWDV_1063_1069_054413_091',
  'pgeVersion': '2.1.1',
  'operaBurstID': 'T091_195252_IW3',
  'validityStartDate': None,
  'subswath': 'IW3',
  'fileName': 'OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
  'beamModeType': 'IW',
  's3Urls': ['s3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE.png.md5',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_low-res.png.md5',
   's3://asf-cumulus-prod-opera-browse/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_BROWSE_thumbnail.png.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VH.tif.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif.md5',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif',
   's3://asf-cumulus-prod-opera-products/OPERA_L2_RTC-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif.md5'],
  'beamMode': 'IW',
  'additionalUrls': ['https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.h5',
   'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0.iso.xml',
   'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_VV.tif',
   'https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T091-195252-IW3_20240620T233649Z_20240621T025958Z_S1A_30_v1.0_mask.tif'],
  'bistaticDelayCorrection': 'TRUE',
  'noiseCorrection': 'TRUE',
  'postProcessingFilter': 'FALSE'}]
scottstanie commented 1 day ago

Adding that this is also been a challenge for getting CSLCs- right now my wrapper around asf_search has had to increase the complexity of the filtering I do a few times, and I'd expect normal users to have significant trouble at the moment