search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

allow empty datacubes #649

Closed jdries closed 1 week ago

jdries commented 5 months ago

We have checks like this one: https://github.com/Open-EO/openeo-geotrellis-extensions/blob/68143219e6882dffd0d05a46aaf29e530c0b93a5/openeo-geotrellis/src/main/scala/org/openeo/geotrellis/layers/FileLayerProvider.scala#L974 To avoid empty datacubes. There are however use cases where this is not desirable, because the workflow can deal with the absence of data.

Stack trace in sentinelhub:

rg.openeo.geotrellissentinelhub.package$NoSuchFeaturesException: no features found for criteria: collection ID "sentinel-1-grd" 1 polygon(s) [2021-04-12T00:00:00+00:00, 2022-09-18T00:00:00+00:00) metadata properties {sat:orbit_state={eq=ASCENDING}, polarization={eq=DV}} at org.openeo.geotrellissentinelhub.PyramidFactory.datacube_seq(PyramidFactory.scala:414)

jdries commented 5 months ago

load_collection -> featureflags -> allow_empty_cube

soxofaan commented 5 months ago

@m-mohr do you think it makes sense to start a discussion about the concept of an empty datacube on the level of openeo-api or openeo-processes ?

To roughly sketch the use case we are trying to address here: user that loads S1 ASCENDING and DESCENDING separately and after that wants to pick the cube with most observations (e.g. with an if based on count of temporal labels). Sometimes it can happen that either of ASC or DESC has no observations at all for the spatio-temporal extent, which at the moment raises something like an "NoDataAvailable" error, failing the whole job. Some kind of empty cube (with zero observations or temporal labels), could be a workaround.

jdries commented 5 months ago

Update from the user: on our CDSE backend, the usecase actually does work. It indeed generates an empty datacube, but the processes used by that case did not crash on it.

One aspect that the spec could help with: what should we generate when we write an empty datacube to a file? Only stac metadata and no asset, or instead error the job? (We'll now error in most cases.)

m-mohr commented 5 months ago

Generally openEO doesn't disallow empty data cubes, we just felt it's more user-friendly to report no data in load_collection directly.

soxofaan commented 1 month ago

Another use case (related to https://github.com/Open-EO/openeo-geopyspark-driver/issues/785):

This currently is not possible because because the NoDataAvailable is thrown too eagerly from load_collection

EmileSonneveld commented 1 month ago

@Pratichhya , can you provide a minimal case that gives this error?

EmileSonneveld commented 1 month ago

Apearantly the graphs here are a good example already: https://github.com/Open-EO/openeo-geopyspark-driver/issues/785#issuecomment-2139048233

JeroenVerstraelen commented 3 weeks ago

Job option would be best. Then we don't need to adhere to the specification.

bossie commented 2 weeks ago

This already works on CDSE. @bossie

i.e. the SENTINEL1_GRD collection.

HansVRP commented 2 weeks ago

Based on a discussion with Dennis, we are in favor of running it on CDSE.

Yet, as the customer was attracted through Terrascope and has Terrascope credits we will need to evaluate how to handle it.

@Pratichhya Could we test cropsar1D on CDSE for these particular fields?

bossie commented 2 weeks ago

As discussed, the allow_empty_cube feature flag was already implemented so ideally this is just a matter of passing it as a parameter to the CropSAR process: TBC.

bossie commented 2 weeks ago

@Pratichhya, @HansVRP, locally I've managed to run the process graph with the 6 polygons succesfully:

```json { "process_graph": { "CropSAR1": { "process_id": "CropSAR", "arguments": { "biopar_type": "FCOVER", "date": [ "2023-01-01", "2023-02-01" ], "minimal_input_checks": true, "polygon": { "type": "FeatureCollection", "features": [ { "id": "0", "type": "Feature", "properties": { "*Field_Area": "60.7104355585474", "Area_": "60.7104355585474", "AssetType_": "Crop Year 2023", "Descriptor": "W-06-", "ENCLOSED_AREA": "60.607", "Errors": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.22650350455383, 44.279303122758215 ], [ -89.2265108460764, 44.27925677020906 ], [ -89.22653215200468, 44.27921495497037 ], [ -89.226565336766, 44.27918177020906 ], [ -89.22660715200468, 44.27916046428077 ], [ -89.22665350455384, 44.27915312275822 ], [ -89.226699857103, 44.27916046428077 ], [ -89.22674167234167, 44.27918177020906 ], [ -89.226774857103, 44.27921495497037 ], [ -89.22679616303128, 44.27925677020906 ], [ -89.22680350455384, 44.279303122758215 ], [ -89.22679616303128, 44.27934947530737 ], [ -89.226774857103, 44.27939129054606 ], [ -89.22674167234167, 44.27942447530737 ], [ -89.226699857103, 44.27944578123566 ], [ -89.22665350455384, 44.27945312275821 ], [ -89.22660715200468, 44.27944578123566 ], [ -89.226565336766, 44.27942447530737 ], [ -89.22653215200468, 44.27939129054606 ], [ -89.2265108460764, 44.27934947530737 ], [ -89.22650350455383, 44.279303122758215 ] ] ] } }, { "id": "1", "type": "Feature", "properties": { "*Field_Area": "35.0716392460504", "Area_": "35.0716392460504", "AssetType_": "Crop Year 2023", "Descriptor": "W-33-", "ENCLOSED_AREA": "35.012", "Errors": "41f0280f-651e-4f73-9cba-b0e4fc822970", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "41f0280f-651e-4f73-9cba-b0e4fc822970", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.20597244614544, 44.31106799892245 ], [ -89.205979787668, 44.31102164637329 ], [ -89.20600109359629, 44.3109798311346 ], [ -89.2060342783576, 44.31094664637329 ], [ -89.20607609359628, 44.310925340445 ], [ -89.20612244614544, 44.31091799892245 ], [ -89.2061687986946, 44.310925340445 ], [ -89.20621061393328, 44.31094664637329 ], [ -89.2062437986946, 44.3109798311346 ], [ -89.20626510462289, 44.31102164637329 ], [ -89.20627244614545, 44.31106799892245 ], [ -89.20626510462289, 44.3111143514716 ], [ -89.2062437986946, 44.31115616671029 ], [ -89.20621061393328, 44.311189351471604 ], [ -89.2061687986946, 44.31121065739989 ], [ -89.20612244614544, 44.311217998922444 ], [ -89.20607609359628, 44.31121065739989 ], [ -89.2060342783576, 44.311189351471604 ], [ -89.20600109359629, 44.31115616671029 ], [ -89.205979787668, 44.3111143514716 ], [ -89.20597244614544, 44.31106799892245 ] ] ] } }, { "id": "2", "type": "Feature", "properties": { "*Field_Area": "162.580630362381", "Area_": "162.580630362381", "AssetType_": "Crop Year 2023", "Descriptor": "W-08-", "ENCLOSED_AREA": "162.31", "Errors": "6045134c-f7ed-4f16-aa16-98f02750d2ff", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "6045134c-f7ed-4f16-aa16-98f02750d2ff", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.17208056415653, 44.20295170761009 ], [ -89.17208790567909, 44.202905355060935 ], [ -89.17210921160738, 44.202863539822246 ], [ -89.1721423963687, 44.20283035506093 ], [ -89.17218421160737, 44.20280904913265 ], [ -89.17223056415654, 44.20280170761009 ], [ -89.1722769167057, 44.20280904913265 ], [ -89.17231873194437, 44.20283035506093 ], [ -89.1723519167057, 44.202863539822246 ], [ -89.17237322263398, 44.202905355060935 ], [ -89.17238056415654, 44.20295170761009 ], [ -89.17237322263398, 44.202998060159246 ], [ -89.1723519167057, 44.203039875397934 ], [ -89.17231873194437, 44.20307306015925 ], [ -89.1722769167057, 44.203094366087534 ], [ -89.17223056415654, 44.20310170761009 ], [ -89.17218421160737, 44.203094366087534 ], [ -89.1721423963687, 44.20307306015925 ], [ -89.17210921160738, 44.203039875397934 ], [ -89.17208790567909, 44.202998060159246 ], [ -89.17208056415653, 44.20295170761009 ] ] ] } }, { "id": "3", "type": "Feature", "properties": { "*Field_Area": "27.4171938026853", "Area_": "27.4171938026853", "AssetType_": "Crop Year 2023", "Descriptor": "W-16-", "ENCLOSED_AREA": "27.371", "Errors": "75f86ffe-f88f-4445-9264-5addd09817ac", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "75f86ffe-f88f-4445-9264-5addd09817ac", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.21124489034247, 44.29969997229918 ], [ -89.21125223186503, 44.299653619750025 ], [ -89.21127353779332, 44.299611804511336 ], [ -89.21130672255464, 44.29957861975002 ], [ -89.21134853779331, 44.29955731382174 ], [ -89.21139489034248, 44.29954997229918 ], [ -89.21144124289164, 44.29955731382174 ], [ -89.21148305813031, 44.29957861975002 ], [ -89.21151624289163, 44.299611804511336 ], [ -89.21153754881992, 44.299653619750025 ], [ -89.21154489034248, 44.29969997229918 ], [ -89.21153754881992, 44.299746324848336 ], [ -89.21151624289163, 44.299788140087024 ], [ -89.21148305813031, 44.29982132484834 ], [ -89.21144124289164, 44.299842630776624 ], [ -89.21139489034248, 44.29984997229918 ], [ -89.21134853779331, 44.299842630776624 ], [ -89.21130672255464, 44.29982132484834 ], [ -89.21127353779332, 44.299788140087024 ], [ -89.21125223186503, 44.299746324848336 ], [ -89.21124489034247, 44.29969997229918 ] ] ] } }, { "id": "4", "type": "Feature", "properties": { "*Field_Area": "35.1545679670981", "Area_": "35.1545679670981", "AssetType_": "Crop Year 2023", "Descriptor": "W-15-", "ENCLOSED_AREA": "35.095", "Errors": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.2112191943176, 44.29602472823923 ], [ -89.21122653584015, 44.295978375690076 ], [ -89.21124784176844, 44.29593656045139 ], [ -89.21128102652976, 44.295903375690074 ], [ -89.21132284176844, 44.29588206976179 ], [ -89.2113691943176, 44.295874728239234 ], [ -89.21141554686676, 44.29588206976179 ], [ -89.21145736210543, 44.295903375690074 ], [ -89.21149054686676, 44.29593656045139 ], [ -89.21151185279504, 44.295978375690076 ], [ -89.2115191943176, 44.29602472823923 ], [ -89.21151185279504, 44.29607108078839 ], [ -89.21149054686676, 44.296112896027076 ], [ -89.21145736210543, 44.29614608078839 ], [ -89.21141554686676, 44.296167386716675 ], [ -89.2113691943176, 44.29617472823923 ], [ -89.21132284176844, 44.296167386716675 ], [ -89.21128102652976, 44.29614608078839 ], [ -89.21124784176844, 44.296112896027076 ], [ -89.21122653584015, 44.29607108078839 ], [ -89.2112191943176, 44.29602472823923 ] ] ] } }, { "id": "5", "type": "Feature", "properties": { "*Field_Area": "151.089663418471", "Area_": "151.089663418471", "AssetType_": "Crop Year 2023", "Descriptor": "W-04-", "ENCLOSED_AREA": "150.83", "Errors": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.21943779014806, 44.279709888570515 ], [ -89.21944513167062, 44.27966353602136 ], [ -89.21946643759891, 44.27962172078267 ], [ -89.21949962236023, 44.27958853602136 ], [ -89.2195414375989, 44.27956723009307 ], [ -89.21958779014807, 44.27955988857052 ], [ -89.21963414269723, 44.27956723009307 ], [ -89.2196759579359, 44.27958853602136 ], [ -89.21970914269723, 44.27962172078267 ], [ -89.21973044862551, 44.27966353602136 ], [ -89.21973779014807, 44.279709888570515 ], [ -89.21973044862551, 44.27975624111967 ], [ -89.21970914269723, 44.27979805635836 ], [ -89.2196759579359, 44.27983124111967 ], [ -89.21963414269723, 44.27985254704796 ], [ -89.21958779014807, 44.27985988857051 ], [ -89.2195414375989, 44.27985254704796 ], [ -89.21949962236023, 44.27983124111967 ], [ -89.21946643759891, 44.27979805635836 ], [ -89.21944513167062, 44.27975624111967 ], [ -89.21943779014806, 44.279709888570515 ] ] ] } } ] } }, "namespace": "vito" }, "saveresult1": { "process_id": "save_result", "arguments": { "data": { "from_node": "CropSAR1" }, "format": "JSON", "options": {} }, "result": true } } } ```

Does this look like a plausible result?

{
  "Field_0_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_1_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_2_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_3_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_4_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_5_cropSAR": {
    "2023-01-01": 0.5519999861717224,
    "2023-01-02": 0.5609999895095825,
    "2023-01-03": 0.5680000185966492,
    "2023-01-04": 0.574999988079071,
    "2023-01-05": 0.5830000042915344,
    "2023-01-06": 0.5899999737739563,
    "2023-01-07": 0.5960000157356262,
    "2023-01-08": 0.6029999852180481,
    "2023-01-09": 0.609000027179718,
    "2023-01-10": 0.6140000224113464,
    "2023-01-11": 0.6190000176429749,
    "2023-01-12": 0.6240000128746033,
    "2023-01-13": 0.628000020980835,
    "2023-01-14": 0.6330000162124634,
    "2023-01-15": 0.6370000243186951,
    "2023-01-16": 0.6399999856948853,
    "2023-01-17": 0.6430000066757202,
    "2023-01-18": 0.6460000276565552,
    "2023-01-19": 0.6489999890327454,
    "2023-01-20": 0.6510000228881836,
    "2023-01-21": 0.652999997138977,
    "2023-01-22": 0.656000018119812,
    "2023-01-23": 0.6579999923706055,
    "2023-01-24": 0.6600000262260437,
    "2023-01-25": 0.6620000004768372,
    "2023-01-26": 0.6639999747276306,
    "2023-01-27": 0.6660000085830688,
    "2023-01-28": 0.6669999957084656,
    "2023-01-29": 0.6690000295639038,
    "2023-01-30": 0.6710000038146973,
    "2023-01-31": 0.671999990940094,
    "2023-02-01": 0.6729999780654907
  }
}
HansVRP commented 2 weeks ago

cropSAR might actually give an output eventhough it receives an empty input cube. But it also might need a dedicated feature flag.

I guess we should be able to tie this flag in with the flag for allowing empty datacubes?

bossie commented 2 weeks ago

Yes, that is what this PR is about: https://git.vito.be/projects/APPL/repos/nextland/pull-requests/5/overview

This is an example process graph with the feature enabled:

```json { "process_graph": { "CropSAR1": { "process_id": "CropSAR", "arguments": { "biopar_type": "FCOVER", "date": [ "2023-01-01", "2023-02-01" ], "minimal_input_checks": true, "allow_empty_cubes": true, "polygon": { "type": "FeatureCollection", "features": [ { "id": "0", "type": "Feature", "properties": { "*Field_Area": "60.7104355585474", "Area_": "60.7104355585474", "AssetType_": "Crop Year 2023", "Descriptor": "W-06-", "ENCLOSED_AREA": "60.607", "Errors": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.22650350455383, 44.279303122758215 ], [ -89.2265108460764, 44.27925677020906 ], [ -89.22653215200468, 44.27921495497037 ], [ -89.226565336766, 44.27918177020906 ], [ -89.22660715200468, 44.27916046428077 ], [ -89.22665350455384, 44.27915312275822 ], [ -89.226699857103, 44.27916046428077 ], [ -89.22674167234167, 44.27918177020906 ], [ -89.226774857103, 44.27921495497037 ], [ -89.22679616303128, 44.27925677020906 ], [ -89.22680350455384, 44.279303122758215 ], [ -89.22679616303128, 44.27934947530737 ], [ -89.226774857103, 44.27939129054606 ], [ -89.22674167234167, 44.27942447530737 ], [ -89.226699857103, 44.27944578123566 ], [ -89.22665350455384, 44.27945312275821 ], [ -89.22660715200468, 44.27944578123566 ], [ -89.226565336766, 44.27942447530737 ], [ -89.22653215200468, 44.27939129054606 ], [ -89.2265108460764, 44.27934947530737 ], [ -89.22650350455383, 44.279303122758215 ] ] ] } }, { "id": "1", "type": "Feature", "properties": { "*Field_Area": "35.0716392460504", "Area_": "35.0716392460504", "AssetType_": "Crop Year 2023", "Descriptor": "W-33-", "ENCLOSED_AREA": "35.012", "Errors": "41f0280f-651e-4f73-9cba-b0e4fc822970", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "41f0280f-651e-4f73-9cba-b0e4fc822970", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.20597244614544, 44.31106799892245 ], [ -89.205979787668, 44.31102164637329 ], [ -89.20600109359629, 44.3109798311346 ], [ -89.2060342783576, 44.31094664637329 ], [ -89.20607609359628, 44.310925340445 ], [ -89.20612244614544, 44.31091799892245 ], [ -89.2061687986946, 44.310925340445 ], [ -89.20621061393328, 44.31094664637329 ], [ -89.2062437986946, 44.3109798311346 ], [ -89.20626510462289, 44.31102164637329 ], [ -89.20627244614545, 44.31106799892245 ], [ -89.20626510462289, 44.3111143514716 ], [ -89.2062437986946, 44.31115616671029 ], [ -89.20621061393328, 44.311189351471604 ], [ -89.2061687986946, 44.31121065739989 ], [ -89.20612244614544, 44.311217998922444 ], [ -89.20607609359628, 44.31121065739989 ], [ -89.2060342783576, 44.311189351471604 ], [ -89.20600109359629, 44.31115616671029 ], [ -89.205979787668, 44.3111143514716 ], [ -89.20597244614544, 44.31106799892245 ] ] ] } }, { "id": "2", "type": "Feature", "properties": { "*Field_Area": "162.580630362381", "Area_": "162.580630362381", "AssetType_": "Crop Year 2023", "Descriptor": "W-08-", "ENCLOSED_AREA": "162.31", "Errors": "6045134c-f7ed-4f16-aa16-98f02750d2ff", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "6045134c-f7ed-4f16-aa16-98f02750d2ff", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.17208056415653, 44.20295170761009 ], [ -89.17208790567909, 44.202905355060935 ], [ -89.17210921160738, 44.202863539822246 ], [ -89.1721423963687, 44.20283035506093 ], [ -89.17218421160737, 44.20280904913265 ], [ -89.17223056415654, 44.20280170761009 ], [ -89.1722769167057, 44.20280904913265 ], [ -89.17231873194437, 44.20283035506093 ], [ -89.1723519167057, 44.202863539822246 ], [ -89.17237322263398, 44.202905355060935 ], [ -89.17238056415654, 44.20295170761009 ], [ -89.17237322263398, 44.202998060159246 ], [ -89.1723519167057, 44.203039875397934 ], [ -89.17231873194437, 44.20307306015925 ], [ -89.1722769167057, 44.203094366087534 ], [ -89.17223056415654, 44.20310170761009 ], [ -89.17218421160737, 44.203094366087534 ], [ -89.1721423963687, 44.20307306015925 ], [ -89.17210921160738, 44.203039875397934 ], [ -89.17208790567909, 44.202998060159246 ], [ -89.17208056415653, 44.20295170761009 ] ] ] } }, { "id": "3", "type": "Feature", "properties": { "*Field_Area": "27.4171938026853", "Area_": "27.4171938026853", "AssetType_": "Crop Year 2023", "Descriptor": "W-16-", "ENCLOSED_AREA": "27.371", "Errors": "75f86ffe-f88f-4445-9264-5addd09817ac", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "75f86ffe-f88f-4445-9264-5addd09817ac", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.21124489034247, 44.29969997229918 ], [ -89.21125223186503, 44.299653619750025 ], [ -89.21127353779332, 44.299611804511336 ], [ -89.21130672255464, 44.29957861975002 ], [ -89.21134853779331, 44.29955731382174 ], [ -89.21139489034248, 44.29954997229918 ], [ -89.21144124289164, 44.29955731382174 ], [ -89.21148305813031, 44.29957861975002 ], [ -89.21151624289163, 44.299611804511336 ], [ -89.21153754881992, 44.299653619750025 ], [ -89.21154489034248, 44.29969997229918 ], [ -89.21153754881992, 44.299746324848336 ], [ -89.21151624289163, 44.299788140087024 ], [ -89.21148305813031, 44.29982132484834 ], [ -89.21144124289164, 44.299842630776624 ], [ -89.21139489034248, 44.29984997229918 ], [ -89.21134853779331, 44.299842630776624 ], [ -89.21130672255464, 44.29982132484834 ], [ -89.21127353779332, 44.299788140087024 ], [ -89.21125223186503, 44.299746324848336 ], [ -89.21124489034247, 44.29969997229918 ] ] ] } }, { "id": "4", "type": "Feature", "properties": { "*Field_Area": "35.1545679670981", "Area_": "35.1545679670981", "AssetType_": "Crop Year 2023", "Descriptor": "W-15-", "ENCLOSED_AREA": "35.095", "Errors": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.2112191943176, 44.29602472823923 ], [ -89.21122653584015, 44.295978375690076 ], [ -89.21124784176844, 44.29593656045139 ], [ -89.21128102652976, 44.295903375690074 ], [ -89.21132284176844, 44.29588206976179 ], [ -89.2113691943176, 44.295874728239234 ], [ -89.21141554686676, 44.29588206976179 ], [ -89.21145736210543, 44.295903375690074 ], [ -89.21149054686676, 44.29593656045139 ], [ -89.21151185279504, 44.295978375690076 ], [ -89.2115191943176, 44.29602472823923 ], [ -89.21151185279504, 44.29607108078839 ], [ -89.21149054686676, 44.296112896027076 ], [ -89.21145736210543, 44.29614608078839 ], [ -89.21141554686676, 44.296167386716675 ], [ -89.2113691943176, 44.29617472823923 ], [ -89.21132284176844, 44.296167386716675 ], [ -89.21128102652976, 44.29614608078839 ], [ -89.21124784176844, 44.296112896027076 ], [ -89.21122653584015, 44.29607108078839 ], [ -89.2112191943176, 44.29602472823923 ] ] ] } }, { "id": "5", "type": "Feature", "properties": { "*Field_Area": "151.089663418471", "Area_": "151.089663418471", "AssetType_": "Crop Year 2023", "Descriptor": "W-04-", "ENCLOSED_AREA": "150.83", "Errors": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6", "GROUP_AREA": null, "Grower": "Heartland Farms", "IGuid_": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6", "INVALID_TYPE": null, "INVALID_X": null, "INVALID_Y": null, "ISLAND_AREA": null, "LENGTH": null, "WIDTH": null, "h3index": "83275cfffffffff", "tile": "16TCQ" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -89.21943779014806, 44.279709888570515 ], [ -89.21944513167062, 44.27966353602136 ], [ -89.21946643759891, 44.27962172078267 ], [ -89.21949962236023, 44.27958853602136 ], [ -89.2195414375989, 44.27956723009307 ], [ -89.21958779014807, 44.27955988857052 ], [ -89.21963414269723, 44.27956723009307 ], [ -89.2196759579359, 44.27958853602136 ], [ -89.21970914269723, 44.27962172078267 ], [ -89.21973044862551, 44.27966353602136 ], [ -89.21973779014807, 44.279709888570515 ], [ -89.21973044862551, 44.27975624111967 ], [ -89.21970914269723, 44.27979805635836 ], [ -89.2196759579359, 44.27983124111967 ], [ -89.21963414269723, 44.27985254704796 ], [ -89.21958779014807, 44.27985988857051 ], [ -89.2195414375989, 44.27985254704796 ], [ -89.21949962236023, 44.27983124111967 ], [ -89.21946643759891, 44.27979805635836 ], [ -89.21944513167062, 44.27975624111967 ], [ -89.21943779014806, 44.279709888570515 ] ] ] } } ] } }, "namespace": "vito" }, "saveresult1": { "process_id": "save_result", "arguments": { "data": { "from_node": "CropSAR1" }, "format": "JSON", "options": {} }, "result": true } } } ```

Does this approach (a parameter of the CropSAR process) support your use case with regards to being able to turn the flag on and off?

Instead of a parameter, a job_option is also an... option but requires a bit of extra work.

bossie commented 2 weeks ago

These is the result for the 60 polygon process graph: CropSAR_60_polygons.json

Pratichhya commented 2 weeks ago

These is the result for the 60 polygon process graph: CropSAR_60_polygons.json

Are these for the previously failed polygons of j-24051737210441be92441beedd8ab5ce (60 polygons)?

bossie commented 2 weeks ago

Are these for the previously failed polygons of j-24051737210441be92441beedd8ab5ce (60 polygons)?

That is right.

Pratichhya commented 2 weeks ago

Hi @bossie, Was the result achieved when tested in staging?

bossie commented 1 week ago

No, I ran this locally, it's not on dev or staging yet. I'm taking a different approach and the plan is to get this out shortly.

Would you consider the results I attached correct?

Pratichhya commented 1 week ago

No, I ran this locally, it's not on dev or staging yet. I'm taking a different approach, and the plan is to get this out shortly.

ahh ok. Thank you for confirming

Would you consider the results I attached correct?

It looks similar to the expected result. However, for those with null, I was wondering if these didn't have any of Sentinel 1 and Sentinel 2. Also, I simply wanted to check if they still remain null when tested for the entire year.

bossie commented 1 week ago

Abandoned the PR in the nextland repo in favor of a job option:

{
  "allow_empty_cubes": true
}
bossie commented 1 week ago

@Pratichhya this is available on staging.