search

schireson / pytest-mock-resources

Pytest Fixtures that let you actually test against external resource (Postgres, Mongo, Redshift...) dependent code.
https://pytest-mock-resources.readthedocs.io/en/latest/quickstart.html
MIT License
179 stars 19 forks source link

redshift_fixture does not play nice with pandas.DataFrame.to_sql #25

Closed the-real-json closed 5 years ago

the-real-json commented 5 years ago

When attempting to use insert data into the redshift fixture via pandas.DataFrame.to_sql, no error message of any kind is emitted but no rows get inserted. Since there is a workaround (see below), it would be great if at least we can track down any error messages being dropped and report back to the user.

Ideally, of course, we fix the underlying issue to make to_sql work correctly, but the urgency is not as great since a workaround exists.

The workaround is

pandas_sql_engine = SQLDatabase(engine=_redshift, schema=schema)
sql_table = SQLTable(
    name=table,
    pandas_sql_engine=pandas_sql_engine,
    frame=data,
    schema=schema,
    index=False,
)
sql_table.insert()
oakhan3 commented 5 years ago

Closing this for now as I can't seem to reproduce the issue locally or in CI!