Data Pipeline Guidance

Microsoft patterns & practices

:memo:

An updated version of the Cold Storage Processor and the Simulator are available as part of our IoT Journey project. The updated versions are:

• Long-term Storage event processor
• Scenario Simulator

This reference implementation is a work-in-progress project. It is meant to demonstrate proven practices regarding the high-scale, high-volume ingestion of data in a typical event processing system.

The project makes heavy use of Microsoft Azure Event Hubs, a cloud-scale telemetry ingestion service. Familiarity with the general concepts underlying Event Hubs is very useful for understanding the source in this reference implementation.

Overview

The two primary concerns of this project are:

• Facilitating cold storage of data for later analytics. That is, translating the chatty stream of events into chunky blobs.

• Dispatching incoming events to specific handlers. That is, examining an incoming event and passing it along to an appropriate handler function. The emphasis of our dispatcher solution is on speed and overal throughput.

Next Steps

• Scenarios & Requirements
• Architecture Overview
• Getting Started
• Contribute :smile:

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.