mohaseeb / beam-nuggets

Collection of transforms for the Apache beam python SDK.
http://mohaseeb.com/beam-nuggets/
MIT License
87 stars 38 forks source link

Fix reading and writing data in different schema from public (Postgres) #47

Open sdmj45 opened 2 years ago

sdmj45 commented 2 years ago

This pull request is to fix the issues #27 #46 unable to read and write data in different schema from public(Postgres)

When calling relational_db.ReadFromDB() we can specify the schema which is 'public' by default. Example:

source_config = relational_db.SourceConfiguration(
    drivername='postgresql',
    host='localhost',
    port=5432,
    username='postgres',
    password='postgres',
    database='postgres',
)
relational_db.ReadFromDB(
            source_config=source_config,
            table_name='table_name',
            schema='schema'
        )

And when writing to Postgres, we can also specify the schema in relational_db.TableConfiguration

 output_table_config = relational_db.TableConfiguration(
        name='test',
        schema='test_schema',
        create_if_missing=True,
        primary_key_columns=['id']
    )

relational_db.Write(
            source_config=source_config,
            table_config=output_table_config)