search

ESA-APEx / apex_algorithms

Hosted APEx algorithms
Apache License 2.0
0 stars 1 forks source link

failed benchmark "bap_composite" #55

Closed soxofaan closed 15 minutes ago

soxofaan commented 2 weeks ago

last failed run at this time: https://github.com/ESA-APEx/apex_algorithms/actions/runs/11642884038/job/32422979921

job finished, but benchmark failure due to result comparison:

        # Compare actual results with reference data
        reference_dir = download_reference_data(
            scenario=scenario, reference_dir=tmp_path / "reference"
        )
        # TODO: allow to override rtol/atol options of assert_job_results_allclose
>       assert_job_results_allclose(
            actual=actual_dir, expected=reference_dir, tmp_path=tmp_path
        )

tests/test_benchmarks.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/openeo/testing/results.py:384: in assert_job_results_allclose
    issues = _compare_job_results(actual, expected, rtol=rtol, atol=atol, tmp_path=tmp_path)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/openeo/testing/results.py:311: in _compare_job_results
    issues = _compare_job_result_metadata(actual=actual_path, expected=expected_path)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/openeo/testing/results.py:337: in _compare_job_result_metadata
    expected_metadata = _load_json(expected)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/openeo/testing/results.py:61: in _load_json
    return json.load(f)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/json/__init__.py:293: in load
    return loads(fp.read(),
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x7fd98528dbe0>
s = '<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message></Message><BucketName>apex-benchmarks<...stId>tx00000134cccfa6117c55b-00672622af-10a22b5da-default</RequestId><HostId>10a22b5da-default-waw3-1</HostId></Error>'
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.

        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.

        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/json/decoder.py:355: JSONDecodeError
----------------------------- Captured stdout call -----------------------------
0:00:00 Job 'cdse-j-241102fa54ca41aba25806981149b62e': send 'start'
0:00:14 Job 'cdse-j-241102fa54ca41aba25806981149b62e': created (progress 0%)
0:00:20 Job 'cdse-j-241102fa54ca41aba25806981149b62e': created (progress 0%)
0:00:26 Job 'cdse-j-241102fa54ca41aba25806981149b62e': created (progress 0%)
0:00:34 Job 'cdse-j-241102fa54ca41aba25806981149b62e': created (progress 0%)
0:00:45 Job 'cdse-j-241102fa54ca41aba25806981149b62e': running (progress N/A)
0:00:57 Job 'cdse-j-241102fa54ca41aba25806981149b62e': running (progress N/A)
0:01:13 Job 'cdse-j-241102fa54ca41aba25806981149b62e': running (progress N/A)
0:01:32 Job 'cdse-j-241102fa54ca41aba25806981149b62e': running (progress N/A)
0:01:57 Job 'cdse-j-241102fa54ca41aba25806981149b62e': running (progress N/A)
0:02:27 Job 'cdse-j-241102fa54ca41aba25806981149b62e': running (progress N/A)
0:03:06 Job 'cdse-j-241102fa54ca41aba25806981149b62e': finished (progress 100%)
------------------------------ Captured log call -------------------------------
INFO     conftest:conftest.py:103 Connecting to 'openeofed.dataspace.copernicus.eu'
INFO     openeo.config:config.py:193 Loaded openEO client config from sources: []
INFO     conftest:conftest.py:116 Checking for auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED' to drive auth against url='openeofed.dataspace.copernicus.eu'.
INFO     conftest:conftest.py:120 Extracted provider_id='CDSE' client_id='openeo-apex-benchmarks-service-account' from auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED'
INFO     openeo.rest.connection:connection.py:413 Found OIDC providers: ['CDSE']
INFO     openeo.rest.auth.oidc:oidc.py:401 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:510 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.job:job.py:378 Downloading Job result asset 'openEO_2022-05-01Z.tif' from https://openeo.dataspace.copernicus.eu/openeo/1.1/jobs/j-241102fa54ca41aba25806981149b62e/results/assets/NmE3N2ZjZDEtOWMwOC00NmU5LWI4NzUtNTRmYjk5OWFiMjAw/1c5bb9538a05050ad904fed4d9c99a23/openEO_2022-05-01Z.tif?expires=1731157291 to /home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/actual/openEO_2022-05-01Z.tif
INFO     apex_algorithm_qa_tools.scenarios:util.py:341 Downloading reference data for scenario.id='bap_composite' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference'): start 2024-11-02 13:01:34.435602
INFO     apex_algorithm_qa_tools.scenarios:util.py:341 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/bap_composite/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference/job-results.json'): start 2024-11-02 13:01:34.435965
INFO     apex_algorithm_qa_tools.scenarios:util.py:347 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/bap_composite/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference/job-results.json'): end 2024-11-02 13:01:35.172259, elapsed 0:00:00.736294
INFO     apex_algorithm_qa_tools.scenarios:util.py:341 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/bap_composite/openEO_2022-05-01Z.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference/openEO_2022-05-01Z.tif'): start 2024-11-02 13:01:35.172606
INFO     apex_algorithm_qa_tools.scenarios:util.py:347 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/bap_composite/openEO_2022-05-01Z.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference/openEO_2022-05-01Z.tif'): end 2024-11-02 13:01:35.719231, elapsed 0:00:00.546625
INFO     apex_algorithm_qa_tools.scenarios:util.py:347 Downloading reference data for scenario.id='bap_composite' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference'): end 2024-11-02 13:01:35.719447, elapsed 0:00:01.283845
INFO     openeo.testing.results:results.py:298 Comparing job results: PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/actual') vs PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_bap_composi0/reference')
soxofaan commented 2 weeks ago

problem of https://github.com/ESA-APEx/apex_algorithms/actions/runs/11642884038/job/32422979921 is weird S3 access issue which I can not reproduce anymore.

I just updated the reference data of the benchmark to latest results, hopefully that resolves the problem

soxofaan commented 2 weeks ago

next run https://github.com/ESA-APEx/apex_algorithms/actions/runs/11743961110/job/32717962982 now also failed with 

FAILED tests/test_benchmarks.py::test_run_benchmark[bap_composite] - AssertionError: Issues for file 'openEO_2022-05-01Z.tif':
Left and right DataArray objects are not close
Differing values:
L
    array([[[-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768],
            ...,
            [-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768]],

           [[-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768],
            ...,
            [-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768]],

           [[-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768],
            ...,
            [-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768]]], dtype=int16)
R
    array([[[-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768],
            ...,
            [-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768]],

           [[-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768],
            ...,
            [-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768]],

           [[-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768],
            ...,
            [-32768, -32768, ..., -32768, -32768],
            [-32768, -32768, ..., -32768, -32768]]], dtype=int16)

On visual inspection, there seem to be very small differences at the borders of the extent. The (default) tolerances at this point are rtol=1e-6 and atol=1e-6 which is not ideal in this case of integer values in the range [0-4000] which is a known issue, as noted in this todo note: https://github.com/ESA-APEx/apex_algorithms/blob/1b889b2c028a00a3cdc4db7572f959bbd46052bf/qa/benchmarks/tests/test_benchmarks.py#L66-L68 -> #60

VincentVerelst commented 2 hours ago

@HansVRP , I saw @soxofaan already committed a fix before this was assigned to me. Any action required from my side, or can we close this?

soxofaan commented 16 minutes ago

bap_composite benchmark is indeed passing at the moment last run at the moment was https://github.com/ESA-APEx/apex_algorithms/actions/runs/11997884322/job/33444012774