Automating your pipelines with Self Describing Data

[gantir]

Title

Automating your pipelines with Self Describing Data

Description

Every data stream has to comply to a schema, go through basic type checks and other validation criteria. If all of your data has to go through this, why do we have to write source specific pipelines. I present an approach where the data describes itself and build generic pipelines which span across schemas.

Duration

• [x] 30 min

Audience

This is talk has no prerequisite, an experience with data pipelines will be useful.

Outline

• The Problem
• Live Demo
• Self Describing Data
• Self Describing Schema
• Schema Repository
• In practice

Additional notes

About me: I have built and scaled technology products at multiple startups. Currently building a marketing analytics solution.

---

• [ ] Don't record this talk.

[vinayak-mehta]

@gantir Can you please post the link to this talk's slides?

[vinayak-mehta]

@gantir Are you available to present this talk in the meetup next month?

[gantir]

Hi Vinayak,

Which date are you planning the next meeting?

Thanks and regards, rAm g

i Think, i Wait, i Fast -- Siddhartha http://ramjeeganti.com

On Sun, Jun 16, 2019 at 8:24 PM Vinayak Mehta notifications@github.com wrote:

@gantir https://github.com/gantir Are you available to present this talk in the meetup next month?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydatabangalore/talks/issues/1?email_source=notifications&email_token=AACTVOPDFMAO3YZ3E2EI4TLP2ZH3XA5CNFSM4HP5XK7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXZOYHA#issuecomment-502459420, or mute the thread https://github.com/notifications/unsubscribe-auth/AACTVOOVIA3TKJSWGEQSLCTP2ZH3XANCNFSM4HP5XK7A .

[NirantK]

@gantir the next session is on the 13th of July. Would you be available?

cc @vinayak-mehta

[gantir]

Yes. I can make it.

On Sat, Jul 6, 2019, 10:19 Nirant notifications@github.com wrote:

@gantir https://github.com/gantir the next session is on the 13th of July. Would you be available?

cc @vinayak-mehta https://github.com/vinayak-mehta

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydatabangalore/talks/issues/1?email_source=notifications&email_token=AACTVOOJCHYCDWN5MRGY6ELP6AP3ZA5CNFSM4HP5XK7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZKSM3Y#issuecomment-508896879, or mute the thread https://github.com/notifications/unsubscribe-auth/AACTVOOHSU44KWB6AX436HTP6AP3ZANCNFSM4HP5XK7A .

[vinayak-mehta]

Awesome! You can find the event details here: https://www.meetup.com/pydata-bangalore/events/262728799/

[vinayak-mehta]

I'll update the event description with your talk. See you there!

[vinayak-mehta]

@gantir Can you post the slides here?

[gantir]

@vinayak-mehta , Here goes the link:

https://docs.google.com/presentation/d/1R6x2K5EyQ736iTOjIS2tASGMzjhi1Q79_Io5RPlIKnU/edit?usp=sharing

[vinayak-mehta]

Thanks!