When stream_maps has an unused "mapper" the tap / mapper should fail

[visch]

Let's say I mistype the value for __source__

{
    "stream_maps": {
        "customers_remove_pii": {
            "__source__": "customerrr",
        },
    },
}

The correct source value in this case would be customer. Stream maps doesn't let you know that anything is wrong. I think the mapper should fail in this case and say the customerrr source doesn't exist.

I understand that sometimes you may not be selecting customer and may want this to not fail, but I think we could check the catalog to be sure customer exists when doing a stream_map as it could be very important that this continues to work. In that case we could print a warning? Or info saying something like "There is a stream map for the customer stream but the stream is not currently selected"

This made a seperate problem I was having harder to track down so thought I'd throw it in!

[aaronsteers]

Thanks, @visch. This could work for taps - since they know the catalog in advance - but targets and independent mappers would not have any visibility to the catalog and wouldn't know the stream names in advance.

I do agree it could be helpful though, even if just for taps that have the mapper capability. 👍

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.