Migrate From Tahmo to OpenWeatherMap API

[MartinKalema]

WHAT DOES THIS PR DO?

• [x] Shift Weather Data Source: Transition the weather data source from TAHMO to OpenWeatherMap to enhance coverage.
• [x] Update Pipeline: Modify the Realtime-Low-Cost-Measurements pipeline to integrate the OpenWeatherMap API.
• [x] Add Backup Weather API: Introduce a backup weather API since OpenWeatherMap was deprecated in June 2024, though it remains operational for now.
• [x] Performance Improvements: Implement various optimizations to improve the performance of the weather data fetching and processing system.
• [ ] Write Unit Tests: Develop and incorporate unit tests to ensure the reliability and accuracy of the new implementations.

WHAT ISSUES ARE RELATED TO THIS PR?

• Addresses the issue of poor coverage and inefficiency in the current weather data fetching approach using TAHMO.

JIRA CARDS

• OPS-220

Summary by CodeRabbit

Summary by CodeRabbit

• New Features

  • Introduced new functionality for fetching weather data from the Open Meteo API, including methods for current and batch weather data retrieval.
  • Added a method for extracting latitude and longitude from a DataFrame to enhance data handling capabilities.

• Improvements

  • Enhanced data processing logic and error handling across multiple utility functions for improved robustness and clarity.
  • Updated function signatures to accept more flexible input formats, enhancing usability when handling weather data.
  • Enhanced type safety and documentation clarity in the get_thingspeak_read_keys function.

• Bug Fixes

  • Improved error messaging to provide clearer context during data retrieval issues.

• Documentation

  • Added configuration capabilities for the Open-Meteo Weather API.

[coderabbitai[bot]]

Walkthrough

## Walkthrough The recent updates to the AirQo ETL utility codebase significantly improve its functionality and structure. Enhancements include refined method signatures, improved data processing and error handling, and the introduction of the Open-Meteo Weather API integration. These changes aim to bolster the robustness, maintainability, and clarity of data workflows, ensuring a more efficient handling of weather data and device information. ## Changes | Files & Change Summaries | |---------------------------| | **src/workflows/airqo_etl_utils/airqo_api.py**: Updated type annotations in `get_thingspeak_read_keys` to specify a list of dictionaries and adjusted documentation for return types. | | **src/workflows/airqo_etl_utils/airqo_utils.py**: Enhanced data processing and error handling in `extract_devices_data`, refactored aggregation logic in `aggregate_low_cost_sensors_data`, and improved data merging in `merge_aggregated_weather_data`. | | **src/workflows/airqo_etl_utils/config.py**: Added `OPEN_METEO_BASE_URL` for configuration of the Open-Meteo API. | | **src/workflows/airqo_etl_utils/openmeteo_api.py**: Introduced `OpenMeteoWeatherAPI` class with methods for fetching current and batch weather data. | | **src/workflows/airqo_etl_utils/tests/test_airqo_utils.py**: Minor formatting updates for consistency without altering any existing functionality. | | **src/workflows/airqo_etl_utils/weather_data_utils.py**: Added `extract_latitude_and_longitude` method and modified `fetch_openweathermap_data_for_sites` to support varied input formats. | | **src/workflows/dags/airqo_measurements.py**: Updated `extract_hourly_weather_data` and `merge_data` functions to handle new data structures for improved processing. | | **src/workflows/dags/airnow.py**: Minor formatting adjustment (removal of a blank line). | ## Sequence Diagram(s) ```mermaid sequenceDiagram participant User participant OpenMeteoWeatherAPI participant AirQoETL User->>AirQoETL: Request weather data AirQoETL->>OpenMeteoWeatherAPI: Fetch current weather data OpenMeteoWeatherAPI-->>AirQoETL: Return weather data AirQoETL-->>User: Deliver weather data ``` > 🌦️ In code we trust, with every change, > New methods and functions, we rearrange. > Data flows clearer, errors less rife, > With Open-Meteo, we enhance our life! > A journey of bytes, through logic we roam, > In this land of code, we've crafted a home. 🌐

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

[AnthonyByansi]

Hi @MartinKalema Thanks for the implementation I see code makes numerous API calls, which may hit rate limits. how about rate limiting or caching e.g through CC: @Baalmart

from ratelimit import limits, sleep_and_retry
import time

ONE_MINUTE = 60

@sleep_and_retry
@limits(calls=30, period=ONE_MINUTE)
def limited_api_call(api_function, *args, **kwargs):
    return api_function(*args, **kwargs)

[NicholasTurner23]

Hi @MartinKalema Thanks for the implementation I see code makes numerous API calls, which may hit rate limits. how about rate limiting or caching e.g through CC: @Baalmart

from ratelimit import limits, sleep_and_retry
import time

ONE_MINUTE = 60

@sleep_and_retry
@limits(calls=30, period=ONE_MINUTE)
def limited_api_call(api_function, *args, **kwargs):
    return api_function(*args, **kwargs)

Thank you @AnthonyByansi.

Could also be nice to think through what happens if the api call fails for whatever reason even after the retries.

Best,

[AnthonyByansi]

Hi @MartinKalema Thanks for the implementation I see code makes numerous API calls, which may hit rate limits. how about rate limiting or caching e.g through CC: @Baalmart

from ratelimit import limits, sleep_and_retry
import time

ONE_MINUTE = 60

@sleep_and_retry
@limits(calls=30, period=ONE_MINUTE)
def limited_api_call(api_function, *args, **kwargs):
    return api_function(*args, **kwargs)

Thank you @AnthonyByansi.

Could also be nice to think through what happens if the api call fails for whatever reason even after the retries.

Best,

Hi @NicholasTurner23 Sure, how about we handle the failure gracefully;

1. error handling and logging: using logging module to log errors and capture exceptions. provide error messages and possibly fallback data or retry

2. retry with fallback: If all retries fail, log the error and proceed with default values.

   
   import logging

logging.basicConfig( level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s', handlers=[logging.StreamHandler()] ) logger = logging.getLogger(name)

Happy to step in and give a hand where need be, cheers! 🥂
CC: @MartinKalema / @Baalmart 

[MartinKalema]

Hi @MartinKalema Thanks for the implementation I see code makes numerous API calls, which may hit rate limits. how about rate limiting or caching e.g through CC: @Baalmart

from ratelimit import limits, sleep_and_retry
import time

ONE_MINUTE = 60

@sleep_and_retry
@limits(calls=30, period=ONE_MINUTE)
def limited_api_call(api_function, *args, **kwargs):
    return api_function(*args, **kwargs)

Thank you @AnthonyByansi. Could also be nice to think through what happens if the api call fails for whatever reason even after the retries. Best,

Hi @NicholasTurner23 Sure, how about we handle the failure gracefully;

1. error handling and logging: using logging module to log errors and capture exceptions. provide error messages and possibly fallback data or retry
2. retry with fallback: If all retries fail, log the error and proceed with default values.

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(message)s',
    handlers=[logging.StreamHandler()]
)
logger = logging.getLogger(__name__)

Happy to step in and give a hand where need be, cheers! 🥂 CC: @MartinKalema / @Baalmart

Thanks for the feedback @AnthonyByansi . The Weather API allows for 60 API calls per minute & @Mnoble-19 already implemented rate limiting within the OpenWeatherAPI class. The limit was set to 50.