Closed jdries closed 1 year ago
Possibly already implemented but not verfied yet: replay the process graph.
Commit has been rolled out on all environments but I'm still getting the same error:
...
Caused by: java.io.FileNotFoundException: s3:/OpenEO-data/batch_jobs/j-43f83402120e44af80e3420a4b9d8ad5/out (No such file or directory)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
at ar.com.hjg.pngj.PngHelperInternal2.ostreamFromFile(PngHelperInternal2.java:29)
... 17 more
Mind the singly slash in the s3:/ prefix. This is again going to be the case where path gets broken. For Geotiff, I solved it by letting the writer return the final, corrected path were things ended up.
Writing the png file was fixed and it has been successfully uploaded to object storage (out
):
$ s3cmd -c s3cfg_cdse ls s3://OpenEO-data/batch_jobs/j-952f478359604b2ea12ae605c442afe9/
2023-06-27 09:44 9146 s3://OpenEO-data/batch_jobs/j-952f478359604b2ea12ae605c442afe9/job_metadata.json
2023-06-27 09:44 483 s3://OpenEO-data/batch_jobs/j-952f478359604b2ea12ae605c442afe9/job_specification.json
2023-06-27 09:44 90447461 s3://OpenEO-data/batch_jobs/j-952f478359604b2ea12ae605c442afe9/out
However, job_metadata.json
lists this asset as out.png
instead:
"assets": {
"out.png": {
"href": "s3://OpenEO-data/batch_jobs/j-952f478359604b2ea12ae605c442afe9/out.png",
"type": "image/png",
"roles": [
"data"
]
}
}
The signed link in /jobs/j-952f478359604b2ea12ae605c442afe9/results therefore doesn't work:
{
"code": "Internal",
"id": "r-ba87fd1e25c84bc8a477936e69dea321",
"message": "Server error: NoSuchKey('An error occurred (NoSuchKey) when calling the GetObject operation: Unknown')"
}
Traceback (most recent call last):
File "/opt/openeo/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/openeo/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/opt/openeo/lib/python3.8/site-packages/openeo_driver/views.py", line 1319, in download_job_result_signed
return _download_job_result(job_id=job_id, filename=filename, user_id=user_id)
File "/opt/openeo/lib/python3.8/site-packages/openeo_driver/views.py", line 1102, in _download_job_result
return _stream_from_s3(result["href"], result)
File "/opt/openeo/lib/python3.8/site-packages/openeo_driver/views.py", line 1123, in _stream_from_s3
s3_file_object = s3_instance.get_object(Bucket=bucket, Key=folder)
File "/opt/openeo/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/openeo/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: Unknown
TODO:
out
and out.png
assets)Fixed on CDSE dev. Comes with some nifty metadata too :+1: :
{
"assets": {
"out.png": {
"file:nodata": [
null
],
"href": "https://openeo.dev.warsaw.openeo.dataspace.copernicus.eu/openeo/1.1/jobs/j-e9e0d36df5e548fb9e681729a466eb85/results/assets/.../out.png?expires=1688540567",
"proj:bbox": [
0.0,
6029.0,
5714.0,
0.0
],
"proj:epsg": 32633,
"proj:shape": [
5714,
6029
],
"raster:bands": [
{
"name": "1",
"statistics": {
"maximum": 255.0,
"mean": 124.42407595583,
"minimum": 0.0,
"stddev": 74.395077242258,
"valid_percent": 100.0
}
},
{
"name": "2",
"statistics": {
"maximum": 255.0,
"mean": 125.3408646216,
"minimum": 0.0,
"stddev": 77.307545349428,
"valid_percent": 100.0
}
},
{
"name": "3",
"statistics": {
"maximum": 255.0,
"mean": 123.32699309538,
"minimum": 0.0,
"stddev": 76.219919822788,
"valid_percent": 100.0
}
}
],
"roles": [
"data"
],
"title": "out.png",
"type": "image/png"
}
}
}
Still works on Terrascope as well. As a side-effect, this fix materialized the ghost out.png
asset (it replaces what used to be the real asset (out
)):