HTTP Error 500: Internal Server Error +Code for downloading images using coordinates, no shape format

[saraemp]

Hi, Thanks for creating a repo to download data from SARA. I ran the commands you listed in your usage example and I got an error (see below). According with the instructions in https://copernicus.nci.org.au/sara.client/#/help the url needs to have '_pretty' after 'search.json?' I changed that in saraclient_downloader.py but it still didn't work.

I was also wondering if you could please create and share a code to downloading images using coordinates, no shape format

Thanks!

[suhendra0812]

The issue is when passing the polygon file to GetData class. I forgot to add method to convert polygon file to WKT format, while geometry parameter in auscophub API is required in WKT format. Now, the issue is fixed.

Sorry for that mistake. I have no longer maintain this repo as now I recommend to use EODAG module which is capable to download satellite data from various providers. For more information, please follow here.

If you're interested, I'll give you the configuration for adding SARA Client in order to download satellite data from SARA using eodag.

This is the configuration in YAML file:

sara:
  description: |
    Geoscience Australia Sentinel hub endpoint to Sentinel-1, Sentinel-2 and Sentinel-3 data.
  roles:
    - host
  url: https://copernicus.nci.org.au/sara.client/
  search:
    type: QueryStringSearch
    api_endpoint: 'https://copernicus.nci.org.au/sara.server/1.0/api/collections/{collection}/search.json'
    pagination:
      next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
      total_items_nb_key_path: '$.properties.totalResults'
      # 2021/03/19: 500 is the max, no error if greater
      max_items_per_page: 500
    discover_metadata:
      auto_discovery: true
      metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
      search_param: '{metadata}={{{metadata}}}'
      metadata_path: '$.properties.*'
    metadata_mapping:
      # Opensearch resource identifier within the search engine context (in our case
      # within the context of the data provider)
      uid: '$.id'
      # OpenSearch Parameters for Collection Search (Table 3)
      productType:
        - productType
        - '$.properties.productType'
      platform: '$.properties.collection'
      platformSerialIdentifier:
        - platform
        - '$.properties.platform'
      instrument:
        - instrument
        - '$.properties.instrument'
      processingLevel:
        - processingLevel
        - '$.properties.processingLevel'

      # INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
      title: '$.properties.title'
      # topicCategory: '$.properties.topicCategory'
      keyword: '$.properties.keywords'
      abstract: '$.properties.description'
      resolution:
        - 'resolution'
        - '$.properties.resolution'
      organisationName:
        - 'organisationName'
        - '$.properties.organisationName'
      publicationDate: '$.properties.published'
      accessConstraint: '$.properties.license'

      # OpenSearch Parameters for Product Search (Table 5)
      parentIdentifier:
        - 'parentIdentifier'
        - '$.properties.parentIdentifier'
      acquisitionType: '$.properties.acquisitionType'
      orbitNumber:
        - 'orbitNumber'
        - '$.properties.orbitNumber'
      orbitDirection:
        - 'orbitDirection'
        - '$.properties.orbitDirection'
      swathIdentifier:
        - 'swath'
        - '$.properties.swath'
      cloudCover:
        - 'cloudCover=[0,{cloudCover}]'
        - '$.properties.cloudCover'
      snowCover:
        - 'snowCover=[0,{snowCover}]'
        - '$.properties.snowCover'
      # productVersion: '$.properties.version'
      # productQualityStatus: '$.properties.onlineQualityCheck'
      # processorName: '$.properties.processingName'
      # processingCenter: '$.properties.processingCenter'
      # creationDate: '$.properties.dhusIngestDate'
      modificationDate: '$.properties.updated'
      sensorMode:
        - 'sensorMode'
        - '$.properties.sensorMode'

      # OpenSearch Parameters for Acquistion Parameters Search (Table 6)
      startTimeFromAscendingNode:
        - startDate
        - '$.properties.startDate'
      completionTimeFromAscendingNode:
        - completionDate
        - '$.properties.completionDate'
      polarizationMode:
        - 'polarisation'
        - '$.properties.polarisation'

      # Custom parameters (not defined in the base document referenced above)
      id:
        - 'productIdentifier={id#remove_extension}'
        - '$.properties.productIdentifier'
      # The geographic extent of the product
      geometry:
        - 'geometry={geometry#to_rounded_wkt}'
        - '$.geometry'
      # The url of the quicklook
      quicklook: '$.properties.quicklook'
      # The url to download the product "as is" (literal or as a template to be completed either after the search result
      # is obtained from the provider or during the eodag download phase)
      downloadLink: '$.properties.services.download.url'
      # storageStatus: must be one of ONLINE, STAGING, OFFLINE
      # storageStatus: '{$.properties.storage.mode#get_group_name((?P<ONLINE>disk)|(?P<STAGING>staging)|(?P<OFFLINE>tape))}'

      # Additional metadata provided by the providers but that don't appear in the reference spec
      thumbnail: '$.properties.thumbnail'
  products:
    S1_SAR_OCN:
      productType: OCN
      collection: S1
    S1_SAR_GRD:
      productType: GRD
      collection: S1
    S1_SAR_SLC:
      productType: SLC
      collection: S1
    S2_MSI_L1C:
      collection: S2
      productType: S2MSI1C
    S2_MSI_L2A:
      collection: S2
      productType: S2MSI2A
    GENERIC_PRODUCT_TYPE:
      productType: '{productType}'
      collection: '{collection}'
  download:
    type: HTTPDownload
    base_uri: 'https://copernicus.nci.org.au/sara.server/1.0'
    extract: true
    archive_depth: 2
    order_enabled: true
    auth_error_code: 403
    dl_url_params:
      issuerId: sara
  auth:
    type: GenericAuth
    auth_uri: 'https://copernicus.nci.org.au/sara.server/1.0/api/user/connect'
    credentials:
      username: yourusername
      password: yourpassword

This is the way to add the config to eodag:

from eodag import EODataAccessGateway

dag = EODataAccessGateway()

with open("./provider.yaml") as f:
    dag.update_providers_config(f.read())

assert "sara" in dag.available_providers()

dag.set_preferred_provider("sara")

Regarding how to download, please follow the tutorial in the EODAG documentation.

Thanks

Suhendra

[saraemp]

Thanks Suhendra, this is very useful.

I know you no longer maintain this repo but I thought I should let you know that I got the following error after you made the changes in saraclient_downloader.py

File "", line 1, in File "/auscophub-1.2.1/saraclient_downloader.py", line 32, in class GetData: File "/auscophub-1.2.1/saraclient_downloader.py", line 33, in GetData def init(self, startdate, enddate, polygon: Path | str): TypeError: unsupported operand type(s) for |: 'type' and 'type'

[suhendra0812]

It is caused by typing issue when I used Python 3.11 the "|" character is only supported in 3.11. I removed the typing in the polygon parameter. It is fixed now perhaps.

[saraemp]

thanks