search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
98 stars 70 forks source link

Document how to build custom mappers #801

Open DouweM opened 2 years ago

DouweM commented 2 years ago

https://sdk.meltano.com and https://docs.meltano.com/concepts/plugins#mappers lack documentation on how InlineMapper can be used to create a custom mappers that modify Singer messages in-flight. https://github.com/MeltanoLabs/meltano-map-transform/blob/main/meltano_map_transform/mapper.py serves as an example, but it's not clear what a bare-minimum mapper looks like.

I shared these notes with a user who'd like to do ETL:

https://github.com/MeltanoLabs/meltano-map-transform/blob/main/meltano_map_transform/mapper.py can serve as an example. As a start, you could fork that repo, comment out all the def map_X_message methods, and add a new map_schema_message and map_record_message that modify the incoming messages as you like, for example snake_casing UpperCase fields. You can then add this custom plugin to meltano.yml using the sample mappers: code under https://docs.meltano.com/concepts/plugins#mappers, with the appropriate pip_url and namespace: custom to mark the plugin as custom instead of coming from the Hub. In your case you may also not need config, so you could specify just a single mapper and mapping. You can then run the mapping along with the tap and target using meltano run tap-foo your-mapping target-bar.

DouweM commented 2 years ago

@tayloramurphy Can this be prioritized as it's missing documentation on an exciting feature?

tayloramurphy commented 2 years ago

FYI @afolson as this is a documentation issue.

sbalnojan commented 2 years ago

I think this should be deprioritized for now. First we needs docs (that includes references, and another tutorial) on the two existing mappers.

stale[bot] commented 1 year ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

edgarrmondragon commented 1 year ago

Still relevant

stale[bot] commented 3 months ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.