The WeatherFlow Collector is a comprehensive data collection system that gathers weather data from local UDP broadcasts and remote APIs. It seamlessly feeds this data into InfluxDB V2, which can be visualized using a set of pre-configured Grafana dashboards. The application provides real-time current conditions and detailed forecasts.
New from the ground up, version 5.1 provides an extensible WeatherFlow Collector now in Python and using InfluxDB V2 for storage. There will be more updates, but this should get you going with all of the same Grafana dashboards (with just a few updates for data dictionary updates and Grafana 11).
If you have any issues or concerns, please open a new Issue and provide details if possible. I'll do my best to address them as quickly as possible. Most everything should be in here, including a new Grafana datasource for streaming real-time wind data. Importing and Exporting data needs a little more work, so stay tuned for updates shortly on those.
Overview
The WeatherFlow-Collector application is designed to collect weather data from various sources, process it, and store it in a database for analysis and real-time access. It integrates multiple data input methods, supports extensive configuration management, and provides a robust web server for data streaming. This application has comprehensive logging and event management to ensure reliable operation and timely alerts.
Modules Overview
config.py:
The config.py module manages the application's configuration settings. It provides functionalities to load configurations from files, validate them, and make them accessible throughout the application. This module ensures that all application components are configured correctly, supporting different formats and environment-specific settings. It is crucial in maintaining the application's flexibility and adaptability to various deployment scenarios.
station_metadata_manager.py:
The station_metadata_manager.py module handles metadata related to weather stations. It manages information about different weather stations, such as their locations, capabilities, and configuration details. This module allows the application to update station metadata dynamically, ensuring accurate and up-to-date information for data processing and analysis. It is essential for tracking and maintaining the integrity of the data sources.
event_manager.py:
The event_manager.py module manages events within the WeatherFlow-Collector application. It handles the creation, triggering, and processing of events based on incoming data and predefined conditions. This module allows for real-time processing and alerting, enabling the application to respond to significant changes or anomalies in the collected weather data. It ensures that important events are not missed and appropriate actions are taken promptly.
weatherflow-collector.py:
The weatherflow-collector.py script serves as the main entry point for the WeatherFlow-Collector application. It initializes all necessary components, including configuration, data handlers, and storage systems, and starts the data collection. This module orchestrates the application's overall workflow, ensuring that data is collected, processed, and stored efficiently. The central hub integrates all other modules and manages the application's lifecycle.
config_validator.py:
The config_validator.py module ensures that the application's configuration settings meet the required standards and criteria. It performs validation checks on the configuration data, providing feedback on any issues or inconsistencies. This module helps prevent misconfigurations leading to application errors or unexpected behavior.
logger.py:
The logger.py module sets up and manages logging for the WeatherFlow-Collector application. It provides various logging levels (such as DEBUG, INFO, WARNING, ERROR) and formats to capture detailed information about the application's operation. This module is crucial for monitoring the application's performance, debugging issues, and maintaining an audit trail of activities. It ensures that all significant events and errors are recorded and can be reviewed later.
provider/websocket_server.py:
The websocket_server.py module implements a WebSocket server to provide real-time data access. It manages WebSocket connections, handles data streams, and ensures secure and efficient communication between the server and clients. This module lets users receive live weather data updates, enhancing the application's interactivity and responsiveness. It supports scalable real-time data delivery to multiple clients.
handlers:
The handlers directory contains various modules for handling different data input types. These include websocket.py for WebSocket connections, system_metrics.py for system performance metrics, udp.py for UDP data packets, rest_stats.py and rest_import.py for REST API interactions, and other handlers for device and station observations and weather forecasts. Each handler specializes in processing and integrating specific data formats into the application's workflow.
collector:
The collector directory includes modules for collecting data from diverse sources. These modules include rest_import.py,rest_export.py,websocket.py,udp.py, and observation handlers. These modules are designed to interface with various data providers, gather weather data, and pass it to the processing pipeline. They ensure the application can aggregate data from multiple sources, supporting comprehensive weather monitoring.
utils:
The utils directory contains utility functions and helper modules that support data processing and metric calculations. For instance, calculate_weather_metrics.py includes functions to compute various weather-related metrics, while utils.py offers general-purpose utilities used across the application. These modules provide reusable code snippets and functions that simplify complex operations and enhance code maintainability.
processor:
The processor directory includes modules processing collected data for export and storage. The export.py module handles exporting processed data to external systems or formats, while collector_data.py manages processing raw data collected by the application. These modules ensure data is correctly formatted, enriched, and prepared for storage or further analysis, facilitating efficient data handling and integration.
storage:
The storage directory manages the application's data storage backends. It includes modules like influxdb.py for interacting with InfluxDB databases, influxdb_delete.py for data deletion operations, and file.py for file-based storage. These modules provide the necessary interfaces and methods to store, retrieve, and manage weather data in different storage systems, ensuring data persistence and accessibility.
New from the ground up, version 5.1 provides an extensible WeatherFlow Collector now in Python and using InfluxDB V2 for storage. There will be more updates, but this should get you going with all of the same Grafana dashboards (with just a few updates for data dictionary updates and Grafana 11).
If you have any issues or concerns, please open a new Issue and provide details if possible. I'll do my best to address them as quickly as possible. Most everything should be in here, including a new Grafana datasource for streaming real-time wind data. Importing and Exporting data needs a little more work, so stay tuned for updates shortly on those.
Overview
The WeatherFlow-Collector application is designed to collect weather data from various sources, process it, and store it in a database for analysis and real-time access. It integrates multiple data input methods, supports extensive configuration management, and provides a robust web server for data streaming. This application has comprehensive logging and event management to ensure reliable operation and timely alerts.
Modules Overview
config.py: The
config.pymodule manages the application's configuration settings. It provides functionalities to load configurations from files, validate them, and make them accessible throughout the application. This module ensures that all application components are configured correctly, supporting different formats and environment-specific settings. It is crucial in maintaining the application's flexibility and adaptability to various deployment scenarios.station_metadata_manager.py: The
station_metadata_manager.pymodule handles metadata related to weather stations. It manages information about different weather stations, such as their locations, capabilities, and configuration details. This module allows the application to update station metadata dynamically, ensuring accurate and up-to-date information for data processing and analysis. It is essential for tracking and maintaining the integrity of the data sources.event_manager.py: The
event_manager.pymodule manages events within the WeatherFlow-Collector application. It handles the creation, triggering, and processing of events based on incoming data and predefined conditions. This module allows for real-time processing and alerting, enabling the application to respond to significant changes or anomalies in the collected weather data. It ensures that important events are not missed and appropriate actions are taken promptly.weatherflow-collector.py: The
weatherflow-collector.pyscript serves as the main entry point for the WeatherFlow-Collector application. It initializes all necessary components, including configuration, data handlers, and storage systems, and starts the data collection. This module orchestrates the application's overall workflow, ensuring that data is collected, processed, and stored efficiently. The central hub integrates all other modules and manages the application's lifecycle.config_validator.py: The
config_validator.pymodule ensures that the application's configuration settings meet the required standards and criteria. It performs validation checks on the configuration data, providing feedback on any issues or inconsistencies. This module helps prevent misconfigurations leading to application errors or unexpected behavior.logger.py: The
logger.pymodule sets up and manages logging for the WeatherFlow-Collector application. It provides various logging levels (such as DEBUG, INFO, WARNING, ERROR) and formats to capture detailed information about the application's operation. This module is crucial for monitoring the application's performance, debugging issues, and maintaining an audit trail of activities. It ensures that all significant events and errors are recorded and can be reviewed later.provider/websocket_server.py: The
websocket_server.pymodule implements a WebSocket server to provide real-time data access. It manages WebSocket connections, handles data streams, and ensures secure and efficient communication between the server and clients. This module lets users receive live weather data updates, enhancing the application's interactivity and responsiveness. It supports scalable real-time data delivery to multiple clients.handlers: The
handlersdirectory contains various modules for handling different data input types. These includewebsocket.pyfor WebSocket connections,system_metrics.pyfor system performance metrics,udp.pyfor UDP data packets,rest_stats.pyandrest_import.pyfor REST API interactions, and other handlers for device and station observations and weather forecasts. Each handler specializes in processing and integrating specific data formats into the application's workflow.collector: The
collectordirectory includes modules for collecting data from diverse sources. These modules includerest_import.py,rest_export.py,websocket.py,udp.py,and observation handlers. These modules are designed to interface with various data providers, gather weather data, and pass it to the processing pipeline. They ensure the application can aggregate data from multiple sources, supporting comprehensive weather monitoring.utils: The
utilsdirectory contains utility functions and helper modules that support data processing and metric calculations. For instance,calculate_weather_metrics.pyincludes functions to compute various weather-related metrics, whileutils.pyoffers general-purpose utilities used across the application. These modules provide reusable code snippets and functions that simplify complex operations and enhance code maintainability.processor: The
processordirectory includes modules processing collected data for export and storage. Theexport.pymodule handles exporting processed data to external systems or formats, whilecollector_data.pymanages processing raw data collected by the application. These modules ensure data is correctly formatted, enriched, and prepared for storage or further analysis, facilitating efficient data handling and integration.storage: The
storagedirectory manages the application's data storage backends. It includes modules likeinfluxdb.pyfor interacting with InfluxDB databases,influxdb_delete.pyfor data deletion operations, andfile.pyfor file-based storage. These modules provide the necessary interfaces and methods to store, retrieve, and manage weather data in different storage systems, ensuring data persistence and accessibility.