search

tnc-br / ddf-isoscapes

3 stars 0 forks source link

Missing PRD EE isoscapes #198

Closed benwulfe closed 2 months ago

benwulfe commented 4 months ago

Steps to Test:

Log into PRD timber id here: https://timberid.org/

Add sample and choose “Origin Verification: https://screenshot.googleplex.com/3Zh5SdqqfVAPLWy

Most of the fields dont matter. The ones that do: Lat, lon - ensure these come from a reference source such as https://drive.google.com/file/d/1TRiI0hWqHLqhhC0neklgXXLwEtIJMVxZ/view?usp=sharing

Hit Next and make sure there are five measurements: https://screenshot.googleplex.com/BqkBp4WYyjBohGC Next and then create sample: https://screenshot.googleplex.com/8XfmYNCXFHMTVUe

If you log into google cloud (see https://timberid.gitbook.io/timberid/architecture-of-timberid/gcp-guide/database-query-builder and choose ‘Cloud Functions’)

You can go directly to the correct cloud function here to see logs: https://pantheon.corp.google.com/functions/details/southamerica-east1/fraud-detection-update-sample-sc?env=gen2&authuser=0&hl=en&project=timberid-prd&tab=logs

And here to see source: https://pantheon.corp.google.com/functions/details/southamerica-east1/fraud-detection-update-sample-sc?env=gen2&authuser=0&hl=en&project=timberid-prd&tab=source

Do not update source code in Console because this comes from Github here: https://github.com/tnc-br/ddf-insights-analytics/tree/main/fraud-detection-update-sample

Specifically the “PRD” branch will auto deploy to production when you merge into that branch

Right now the logs say: https://screenshot.googleplex.com/Ahhux45bpBePKnY

Which means it never found an oxygen isoscape because it did not execute this code block: https://screenshot.googleplex.com/9H4MuNitobc4Tw6

kazob1998 commented 4 months ago

The identified issue has been confirmed.

Here is a proposed resolution:

Add Isoscapes to the EE Storage Path:

Initialize Parameters:

Add Exception Handler Block for Reading Isoscapes:

Reference: https://screenshot.googleplex.com/9HaSAZphREoHPhz

Code Implementation:

The code implementation involves initializing Params, fetching isoscapes, and handling exceptions while reading the isoscapes. Below is the code snippet for reference:


def __init__(self, lat, lon, oxygen_measurements, nitrogen_measurements, carbon_measurements):
        self.lat = lat
        self.lon = lon
        self.oxygen_measurements = oxygen_measurements
        self.nitrogen_measurements = nitrogen_measurements
        self.carbon_measurements = carbon_measurements

        # Initialize Earth Engine.
        credentials, project_id = google.auth.default()
        ee.Initialize(credentials)

        # fetching isoscapes
        def get_asset_list(parent_name) -> list:
          parent_asset = ee.data.getAsset(parent_name)
          parent_id = parent_asset['name']
          asset_list = []
          child_assets = ee.data.listAssets({'parent': parent_id})['assets']
          for child_asset in child_assets:
              child_id = child_asset['name']
              child_type = child_asset['type']
              if child_type in ['FOLDER', 'IMAGE_COLLECTION']:
                  # Recursively call the function to get child assets
                  asset_list.extend(get_asset_list(child_id))
              else:
                  asset_list.append(child_id)
          return asset_list

        self.oxygen_isoscape = None
        self.carbon_isoscape = None
        self.nitrogen_isoscape = None
        self.p_value_theshold = 0

        self.oxygen_isoscape_name = None
        self.oxygen_isoscape_date = None
        self.oxygen_isoscape_precision = None
        self.oxygen_isoscape_recall = None

        asset_list = get_asset_list(ISOSCAPES_EE_PATH)
        for asset in asset_list:
            if 'd18O_isoscape' in asset:
                try:
                    self.oxygen_isoscape = ee.Image(asset)
                    self.oxygen_isoscape_name = ee.data.getAsset(asset)['properties']['REFERENCE_ISOSCAPE_NAME']
                    self.oxygen_isoscape_date = ee.data.getAsset(asset)['properties']['DATE_TIME']
                    self.p_value_theshold = float(ee.data.getAsset(asset)['properties']['P_VALUE_THRESHOLD'])
                    self.oxygen_isoscape_precision = ee.data.getAsset(asset)['properties']['PRECISION']
                    self.oxygen_isoscape_recall = ee.data.getAsset(asset)['properties']['RECALL']

                    print(f'found d18O_isoscape in EE assets with properties: name {self.oxygen_isoscape_name} date {self.oxygen_isoscape_date} threshold {self.p_value_theshold} precision {self.oxygen_isoscape_precision} recall {self.oxygen_isoscape_recall}')
                except Exception as e:
                    print(f'caught {type(e)} while reading isoscape {asset}')
                    print(traceback.format_exc())   

            elif 'd13C_isoscape' in asset:
                self.carbon_isoscape = ee.Image(asset)
            elif 'd15N_isoscape' in asset:
                self.nitrogen_isoscape = ee.Image(asset)
        self.poi = ee.Geometry.Point(lon, lat)
benwulfe commented 4 months ago

Oh! So it isn't that the isoscape is missing, its that its not well formed or is missing one of the expected properties. That makes a lot of sense. You change looks good! Send me the PR and I will accept. At some point we should surface the error message back to firestore so its more easily accessed without grepping logs. but not a huge deal.

kazob1998 commented 4 months ago

The generated isoscapes should follow this format: {d18O/d13C/d15N}_isoscape. For instance, there are two d18O files (one for the mean and another for the variance). At this point, it's unclear which one should be selected as the isoscape and renamed. It seems likely that we may need to create a new isoscape for Oxygen. Alternatively, if we want to retain both, we'll need to modify the code to accommodate both files.

@VimWizardVersace can you please advise