glassflow / glassflow-python-sdk

GlassFlow Python SDK to publish and consume data to your pipelines at Glassflow.dev

https://docs.glassflow.dev

MIT License

9 stars 0 forks source link

[Feature]:integrate with Microsoft Synapse Analytics #31

Open judemoore opened 3 weeks ago

judemoore commented 3 weeks ago

[x ] I checked the integrations page and my source/destination is not listed there.

What is the data source/destination?

CosmosDB to Delta Lake format using Synapse integrated linked services. This would also support Microsoft Fabric too.

Your plans for the source/destination

[ ] I want to run it myself
[ ] I need the data for my hobby project
[x ] I need the data for my work
[ x] I want to run it in production
I want to load data to: [e.g. BigQuery, Redshift] Power Bi

What data would you like loaded/sent?

Describe how it should be loaded/consumed Either by SQL Server connection to serverless SQL pools or queries in Synapse Workspace.

Besides you who else will benefit?

Our end users plus all Microsoft based cloud platforms

Boburmirzo commented 3 weeks ago

@judemoore Thanks for the request!

This is an interesting integration proposal. It is possible to ingest real-time changes from CosmosDB using GlassFlow SDK in Python. I have already tested it out.

Could you elaborate more on your specific use case? And if Microsoft Azure does not have a similar integration service?

Then I can pass over to our team to discuss with you more integration part. Thanks! CC @ashish-bagri @PabloPardoGarcia