search

podaac / hitide-backfill-tool

Apache License 2.0
1 stars 0 forks source link

Test new github.com backfill-tool deployment in UAT #28

Open jamesfwood opened 2 months ago

jamesfwood commented 2 months ago

@davidcolemanjpl

Please test backfill tool that is now located here:

https://github.com/podaac/hitide-backfill-tool

Install command line tool from here:

https://test.pypi.org/project/podaac-hitide-backfill-tool/

Also, you'll need to update the default_message_config.json file.

Update from here: https://github.jpl.nasa.gov/podaac/hitide-backfill-tool/blob/develop/podaac/hitide_backfill_tool/default_message_config.json

Let me or Simon know if you have any issues setting it up.

Thanks!

davidcolemanjpl commented 2 months ago

@jamesfwood

Please advise regarding the HiTIDE backfill tool version under test for this effort. Also, what's the associated forge and tig versions in the backfill tool in services-uat?

Thanks!

jamesfwood commented 2 months ago

@davidcolemanjpl

backfill-tool version 0.9.0rc3 forge version 0.11.0-rc.3 tig version 0.12.0-rc.3 forge-py version 0.2.0-rc.4

The versions deployed to each environment should be specified here: https://github.com/podaac/hitide-backfill-tool/tree/develop/terraform-deploy/terraform_env

jamesfwood commented 2 months ago

@davidcolemanjpl There are also a few collections that use the new forge-py library for footprinting. These collections are:

and a few new ones coming soon.

Please also test those collections in UAT.

Thanks!

davidcolemanjpl commented 2 months ago

podaac-hitide-backfill-tool v0.9.0rc3 podaac-app-services-uat-1858 TIG v0.12.0-rc.3 FORGE v0.11.0-rc.3 FORGE-py v0.2.0-rc.4

Verified that podaac-hitide-backfill-tool works as expected for the following collections: _ASCATB-L2-Coastal VIIRS_N21-NAVO-L2P-v3.0 AQUARIUS_L2_SSS_V5 SMAP_RSS_L2_SSS_V5 AMSR2-REMSS-L2P-v8.2 MODIST-JPL-L2P-v2019.0

*When testing '_SCATSAT1_ESDR_L2_WIND_STRESSV1.1' , observed errors for podaac-services-uat-hitide-backfill-forge executions: The executions fail at the ForgePyProcess** step:

"Error": "TypeError", "Cause": "{\"errorMessage\": \"ufunc 'create_collection' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''\", \"errorType\": \"TypeError\", \"requestId\": \"d8f12ff6-465c-49a4-8b63-190877499101\", \"stackTrace\": [\" File \\"/var/task/podaac/lambda_handler/lambda_handler.py\\", line 311, in handler\n return FootprintGenerator.cumulus_handler(event, context=context)\n\", \" File \\"/var/task/cumulus_process/process.py\\", line 315, in cumulus_handler\n return run_cumulus_task(cls.handler, event, context)\n\", \" File \\"/var/task/run_cumulus_task.py\\", line 85, in run_cumulus_task\n return handle_task_exception(exception, cumulus_message, logger)\n\", \" File \\"/var/task/run_cumulus_task.py\\", line 83, in run_cumulus_task\n task_response = task_function(nested_event, context, taskargs)\n\", \" File \\"/var/task/podaac/lambda_handler/lambda_handler.py\\", line 268, in handler\n return cls.run(path=path, noclean=noclean, context=context, event)\n\", \" File \\"/var/task/podaac/lambda_handler/lambda_handler.py\\", line 276, in run\n output = process.process()\n\", \" File \\"/var/task/podaac/lambda_handler/lambda_handler.py\\", line 257, in process\n file_dict = self.footprintgenerate(file, config_file_path, granule_id)\n\", \" File \\"/var/task/podaac/lambda_handler/lambda_handler.py\\", line 199, in footprint_generate\n wkt_representation = forge.generate_footprint(lon_data, lat_data, thinning_fac=thinning_fac, alpha=alpha, is360=is360, simplify=simplify,\n\", \" File \\"/var/task/podaac/forge_py/forge.py\\", line 115, in generate_footprint\n alpha_shape = selected_function(lon_array, lat, thinning_fac=thinning_fac, alpha=alpha)\n\", \" File \\"/var/task/podaac/forge_py/forge.py\\", line 39, in scatsat_footprint\n alpha_shape = alphashape.alphashape(xy, alpha=alpha)\n\", \" File \\"/var/task/alphashape/alphashape.py\\", line 180, in alphashape\n m = MultiLineString([coords[np.array(edge)] for edge in perimeter_edges])\n\", \" File \\"/var/task/shapely/geometry/multilinestring.py\\", line 60, in new\n return shapely.multilinestrings(subs)\n\", \" File \\"/var/task/shapely/decorators.py\\", line 77, in wrapped\n return func(*args, kwargs)\n\", \" File \\"/var/task/shapely/creation.py\\", line 393, in multilinestrings\n return lib.create_collection(geometries, typ, out=out, kwargs)\n\"]}"

Can view recent related forge executions that failed: e0e25761-b08d-46a4-9e02-ba368ea09f83 f79f549a-ad1e-4f6f-8814-4f41356b94fa 0192ac6b-292a-495a-9962-8f01ac2502ef

Image

When testing '_SCATSAT1_ESDR_L2_WIND_STRESSV1.1' , observed errors for podaac-services-uat-hitide-backfill-tig executions:

podaac-services-uat-hitide-backfill-tig fails at the 'ImageProcess' step:

see recent hitide-backfill-tig executions: ece1baf0-aa16-409d-9cea-ca33fb3952ad 656c2ec8-fa45-4da7-b943-69f09f24f54d 5d435340-06c0-4bef-bbdc-3c3b353947a1

"Error": "Lambda.Unknown", "Cause": "The cause could not be determined because Lambda did not return an error type. Returned payload: {\"errorMessage\":\"2024-08-22T01:07:54.717Z 7c61f7e1-1642-4dec-a6df-96034c45a128 Task timed out after 900.11 seconds\"}" }, "delaySeconds": 0 }

Image

COWVR_STPH8_L2_EDR_V9.0:

When testing using COWVR_STPH8_L2_EDR_V8.0 (C1257664524-POCLOUD)

sliu008 commented 2 months ago

We have granules that doesn't belong in the SCATSAT1_ESDR_L2_WIND_STRESS_V1.1 collection all the granules with ancillary in them doesn't belong which is why these forge/tig are failing.

We should delete thees granules from UAT