Convert a page of historical location forecast data into a flattened numpy array for easier use

[david-natingga-frequenz]

• Enable the conversion of protobuf LocationForecast list to a flattened numpy array.
• Enhance the historical data retrieval example with conversion to a pandas dataframe, sample output:

  
  (sdk_venv22) david@david-desktop:~/projects/frequenz/source/frequenz-api-weather$ PYTHONPATH=py python examples/iterate_hist_forecast.py "localhost:50051"
          creation_ts latitude longitude         valid_at_ts                                     feature      value
  0   2022-06-27 12:00:00     52.5      13.4 2022-06-27 13:00:00  ForecastFeature.U_WIND_COMPONENT_100_METRE  -2.629823
  1   2022-06-27 12:00:00     52.5      13.4 2022-06-27 13:00:00  ForecastFeature.V_WIND_COMPONENT_100_METRE   2.316034
  2   2022-06-27 12:00:00     52.5      13.4 2022-06-27 14:00:00  ForecastFeature.U_WIND_COMPONENT_100_METRE  -2.439865
  3   2022-06-27 12:00:00     52.5      13.4 2022-06-27 14:00:00  ForecastFeature.V_WIND_COMPONENT_100_METRE   1.965858
  4   2022-06-27 12:00:00     52.5      13.4 2022-06-27 15:00:00  ForecastFeature.U_WIND_COMPONENT_100_METRE    -1.6119
  ..                  ...      ...       ...                 ...                                         ...        ...
  337 2023-12-18 00:00:00     52.5      13.4 2023-12-18 22:00:00  ForecastFeature.V_WIND_COMPONENT_100_METRE   4.606584
  338 2023-12-18 00:00:00     52.5      13.4 2023-12-18 23:00:00  ForecastFeature.U_WIND_COMPONENT_100_METRE  11.596823
  339 2023-12-18 00:00:00     52.5      13.4 2023-12-18 23:00:00  ForecastFeature.V_WIND_COMPONENT_100_METRE   4.560944
  340 2023-12-18 00:00:00     52.5      13.4 2023-12-19 00:00:00  ForecastFeature.U_WIND_COMPONENT_100_METRE  11.380841
  341 2023-12-18 00:00:00     52.5      13.4 2023-12-19 00:00:00  ForecastFeature.V_WIND_COMPONENT_100_METRE   4.240217

[342 rows x 6 columns]

Closes https://github.com/frequenz-floss/frequenz-api-weather/issues/88.

[TalweSingh]

Also, split the code changes to the client and the example in 2 commits. And a more descriptive title for the PR itself would be great

[david-natingga-frequenz]

Also, split the code changes to the client and the example in 2 commits. And a more descriptive title for the PR itself would be great

I made the title more descriptive.

As for putting an example into a separate commit, it is problematic because the old example would not work with the new code. The reason is that the historical data iterator is changed in this PR, it returns a different object.

[shsms]

Looks like the lack of tests is starting to be a problem. I hope you've tested there's no changes in behaviour for live streaming from the changes to shared methods.

[david-natingga-frequenz]

Looks like the lack of tests is starting to be a problem. I hope you've tested there's no changes in behaviour for live streaming from the changes to shared methods.

I think the method to_ndarray_vlf is quite complicated and I do not see in it much use. In https://github.com/frequenz-floss/frequenz-api-weather/issues/93 I propose just to remove it. @shsms Do you know any code depending on that method? In this PR, I just move the method out, but I do not modify it, so its behaviour should not change.

As for my testing, I just run the live streaming locally and it works with this code just as it worked with the code before. However, you are right, there are no tests that would warrant the exact behaviour as before.

[cwasicki]

@david-natingga-frequenz This is very difficult to review. One reason is, that there are too many changes in a single commit. Can this be split into separate commits?

Do you know any code depending on that method?

Yes, our forecasts.

However, you are right, there are no tests that would warrant the exact behaviour as before.

We should have these.

[david-natingga-frequenz]

Based on the feedback of all of you, I simplified this PR and put it into 2 commits.

• I kept the function to_ndarray_vlf and the class _types.Forecasts unchaged. Any work on to_ndarray_vlf can be addressed in another PR (in order to keep this PR small), an issue already created https://github.com/frequenz-floss/frequenz-api-weather/issues/93.
• Instead I created a new class _types.HistoricalForecasts with a function flatten just for the historical data.

@cwasicki Please, have a look.

[david-natingga-frequenz]

@cwasicki Ready for a review again.