`test_interpolation_temperature_air_mean_200_daily` has different result on GHA/macOS

amotl commented 1 year ago

Dear @neumann-nico and @gutzbenj,

I don't know much about the interpolation subsystem, I only just spotted some anomalies and wanted to share them with you.

With kind regards, Andreas.

Observations

Two CI runs failed at two spots.

Sample # 1: Python3.9, macOS. ^1
Sample # 2: Python3.10, macOS. [^2]
Sample # 3: Python3.8, macOS. ^3

Example

=========================== short test summary info ============================
FAILED tests/core/scalar/test_summary.py::test_interpolation_temperature_air_mean_200_daily - AssertionError: DataFrame.iloc[:, 2] (column name="value") are different

DataFrame.iloc[:, 2] (column name="value") values are different (100.0 %)
[index]: [0, 1, 2]
[left]:  [272.95, 266.15, 268.75]
[right]: [273.65, 267.65, 270.45]

Thoughts

Is it a fluke, and will resolve when re-run? Or did some underlying data change? I will appreciate your thoughts on this, as I currently feel a bit in charge to take care about keeping a stable test suite, for a while ;].

Evaluation

Apparently, the issue is only on macOS, which is failing on lower Python versions. Only Python 3.11 works.

[^2]: Python3.10, macOS: https://github.com/earthobservations/wetterdienst/actions/runs/3516930241/jobs/5895858497#step:7:741

neumann-nico commented 1 year ago

@amotl Thank you for the report! I will have a look into why this is happening.

neumann-nico commented 1 year ago

The summarize feature was done by @gutzbenj, looks really nice! I could not reproduce the wrong results locally on MacOS (Python 3.9/3.10). Just a guess but might be related to the old date in the test(1934-1965)?

amotl commented 1 year ago

Hitting this again. Do you have any idea, @gutzbenj?

Python 3.10, macOS. https://github.com/earthobservations/wetterdienst/actions/runs/3535301195/jobs/5933174921#step:7:744
Python 3.9, macOS. https://github.com/earthobservations/wetterdienst/actions/runs/3535529913/jobs/5933903562#step:7:734
Python 3.8, macOS. https://github.com/earthobservations/wetterdienst/actions/runs/3535529913/jobs/5934173889#step:7:734

gutzbenj commented 1 year ago

It's a bummer that the test currently first compares values and afterwards the station id... otherwise we would know a certain station wasn't available for whatever reason.

Metadata still shows the same:

01048 19340101 20221122            228     51.1278   13.7543 Dresden-Klotzsche                        Sachsen                                                                                           
01050 19490101 20221122            112     -->51.0221   13.8470<-- Dresden-Hosterwitz                       Sachsen                                                                                           
01051 19360101 20221122            119     51.0249   13.7751 Dresden-Strehlen                         Sachsen

Also location is still the same:

stations = DwdObservationRequest(
    parameter=Parameter.TEMPERATURE_AIR_MEAN_200,
    resolution=DwdObservationResolution.DAILY,
    start_date=datetime(1934, 1, 1),
    end_date=datetime(1965, 12, 31),
)

result = stations.summarize(latitude=51.0221, longitude=13.8470) <--

gutzbenj commented 1 year ago

Found it:

Apparently

05282 19170101 19740630            246     51.1197   13.6744 Wahnsdorf bei Dresden                    Sachsen

has temperature_air_mean_200

-0.2 °C -> 272.95 K on 1934-01-01 -7.0 °C -> 266.15 K on 1940-01-01 -4.4 °C -> 268.75 K on 1950-01-01

       5282;19340101;-999;-999;-999;    5;   0.3;   8;    0.000;   3;   8.0;   6.0;  995.40;   -0.2;   99.00;    0.1;   -0.4;-999;eor
...
       5282;19400101;-999;-999;-999;    5;   0.3;   7;    1.800;  13;   7.8;   3.2;  988.70;   -7.0;   87.00;   -3.1;   -7.7;-999;eor
...
       5282;19500101;    5;-999;   3.2;    5;   0.0;   0;    1.600;   0;   6.4;   3.6; 1000.30;   -4.4;   79.00;   -1.7;   -5.0;   -5.8;eor

Concluding, Wahnsdorf bei Dresden must have jumped in as a station to deliver the data, although it farer from the latlon tuple then the other three mentioned stations from above.
Concluding:
- either our algorithm to calculate distances between stations must have changed and made Wahnsdorf the closest station (which I doubt)
- or all three stations had kind of a timeout caused by DWD so that Wahnsdorf was the only station left at that time
- or our caching temporarily didn't return the values of the above three stations so that Wahnsdorf was selected

Do you have a clue which of the options might have caused it?

amotl commented 1 year ago

Maybe add some log messages, which could give more hints about the root cause? On GHA, the hiccup happens quite often, so it shouldn't be hard to get corresponding traces.

amotl commented 1 year ago

Thanks for looking at the data.

Do you have a clue which of the options might have caused it?

I am looking at the computing environment.

The anomaly apparently is only present on macOS, with Python 3.8-3.10. While all tests succeeded on Linux, and macOS with Python 3.11, we can't get past the other Python versions on macOS.

Maybe it's enough to tag the test case as a @cflake? Otherwise, I'd say let's just skip the test on macOS for now. And then see, what it will report on Windows ;].

gutzbenj commented 1 year ago

So this to me looks like it's related to the cache, if it is platform specific...

amotl commented 1 year ago

Ok, let's purge them.

https://github.com/earthobservations/wetterdienst/actions/caches

amotl commented 1 year ago

With 903066c1c, I've disabled the test case on GHA/macOS for now. It's really a weird issue, and may want to be followed up upon.

amotl commented 1 year ago

This has been fixed with https://github.com/earthobservations/wetterdienst/commit/725b94591e6a3c2c0d4fc70b7a7da9c4c7c32fea#r92707270. Thanks, @gutzbenj!

earthobservations / wetterdienst