Open joshua-janicas opened 6 months ago
visch
commented
6 months ago I wonder if a mapper would do this today with filters. Try setting the filter to False for every record https://sdk.meltano.com/en/v0.43.1/stream_maps.html#filtering-out-records-from-a-stream-using-filter-operation
Might work 🤷
visch
commented
6 months ago If I really needed this, I'd mess with the CLI test options and manually do it so something like
meltano invoke tap-name --test schema > out cat out | meltano invoke target-name 
joshua-janicas
commented
1 month ago So I've finally gotten around to trying this out. I was trying to use a tap tap-mssql-static that inherits from tap-mssql.
plugins:
extractors:
- name: tap-mssql
variant: buzzcutnorman
pip_url: git+https://github.com/BuzzCutNorman/tap-mssql.git@v0.1.0
config:
dialect: mssql
driver_type: pyodbc
sqlalchemy_eng_params:
fast_executemany: 'True'
sqlalchemy_url_query:
driver: ODBC Driver 18 for SQL Serverplugins:
extractors:
- name: tap-mssql-static
inherit_from: tap-mssql
select:
- Static-Country.*
- Static-Region.*A few things:
1) Running meltano invoke tap-name --test schema > out ignores the specific tap I'm trying to use and selects all the tables from the database that exist.
2) Then running cat out | meltano invoke target-snowflake breaks on tables in Snowflake that does not exist to begin with (particularly because of (1) listing all the tables instead of the one I want).
Stack trace is below, note that the first Admin-AdminCart already exists as a table (from another tap i have configured), but because Admin-AdminCartEtag does not exist it blows up. In reality I was hoping neither would show up, the tables that it should be trying to do is Static-Country and Static-Region.
Even if it did that though, if those two tables didn't exist it'd be crashing out anyways. Thoughts?
joshua-janicas
commented
1 month ago I am trying visch's first idea with the mappers and that is more what I am looking for I think.
plugins:
loaders:
- name: target-snowflake-overwrite
inherit_from: target-snowflake
config:
load_method: overwriteplugins:
extractors:
- name: tap-mssql-static
inherit_from: tap-mssql
select:
- Static-Country.*
- Static-Region.*
- name: tap-mssql-static-empty
inherit_from: tap-mssql-static
config:
stream_maps:
Static-Country:
__filter__: 'False'
Static-Region:
__filter__: 'False'Running meltano run tap-mssql-static-empty target-snowflake-overwrite behaves as I expect it to - so I'll go down that path.
Separate issue is I was trying to use the "*" https://sdk.meltano.com/en/v0.43.1/stream_maps.html#applying-a-mapping-across-two-or-more-streams but it dies because it's trying to apply the filter to all the tables it finds instead of the one's I'm trying to filter on via the inherited select.
stream_maps:
"*":
__filter__: 'False'
Feature scope
Taps (catalog, state, tests, etc.)
Description
Feature Idea
Use Case