mohaseeb / beam-nuggets

Collection of transforms for the Apache beam python SDK.
http://mohaseeb.com/beam-nuggets/
MIT License
87 stars 38 forks source link

Fail to read data from tables in different schema (Postgres) #27

Open mattew opened 4 years ago

mattew commented 4 years ago

Hi,

I'm trying to read data from a table in a different schema than public but I get the error message that the table can't be found. I've tried to use the schema name both as the table name and in the query but to no success. I think it could be quite good to include the possibility to use different schemas, not only the default one. So I did research.. :)

I read that SQLAlchemy accepts the schema name and after some digging I found that i could supply it in the load_table function in relational_db_api.py. After I added it (see below) and also added it in all the calling places it seemed to work:

def load_table(session, name, schema):
    table_class = None
    engine = session.bind
    if engine.dialect.has_table(engine, name, schema=schema):
        metadata = MetaData(bind=engine)
        table_class = create_table_class(
            Table(name, metadata, autoload=True, schema=schema)
        )
    return table_class

I set the default value to public (in the ReadFromDB function) but I'm not sure if that will only work for Postgres. And I didn't look at the write functions at all...

//Mattias

Xepe commented 4 years ago

Hello,

did you find a solution for this ? @mattew

Regards

mattew commented 3 years ago

Hi,

In the end we ran another solution and not using Beam/Dataflow...

But as I wrote, when updated the load_table function and added the schema as "public" in the ReadFromDB function it worked. I didn't look at the write functions.

//Mattias

canbadroit commented 3 years ago

Is this planned for release any time ?