knizhnik / imcs

In-Memory Columnar Store extension for PostgreSQL
Apache License 2.0
203 stars 33 forks source link

Issue with simple projection - Error: compressed data is corrupted #66

Closed pavitrakumar78 closed 3 years ago

pavitrakumar78 commented 3 years ago

Hi,

I was doing some tests with same table names on different databases with imcs and somehow I saw the on database's table's loaded memory when using another database - are the imcs tables not unique by database or is it just table?

Mainly, because of my testing above, I've been getting projection errors everywhere.. even simple queries like: select symbol_date from us_eod_get('AAPL.US', '2019-12-19', '2021-01-20'); does not work and gives me the error ERROR: compressed data is corrupted and it's the same in the log also.

I've tried fully re-installed imcs (Manually deleting all views/functions etc.,) + recreating the concerned tables (by dropping them completely). The error still persists. Is there any way around this?

I only need cs_project to get un-truncated data and there doesn't seem to be any other way to get it. I've tried setting the imcs.output_string_limit to zero, but sqlachemy's session still produces truncated output and I'm not sure why.

Can you please help with this, @knizhnik ?

knizhnik commented 3 years ago

Sorry, right now IMCS really do not support work with different databases. Or more precisely it IMCS key doesn't include database name. I think that it can be the reason of your problem.

If it is critical - I can add database Oid to IMCS key.

pavitrakumar78 commented 3 years ago

Thank you for the quick reply.

Can you please do add it?

So, for example, if we have 2 databases 'stock_dbandstock_db_1each with the table (imcs)us_eod`

would switching connecting to the right database cause imcs to use the us_eod of the connected db after the fix?

If it does not break any existing features, I'm fine with reinstalling imcs + imcs-tables fully.

pavitrakumar78 commented 3 years ago

Sorry for the late reply! Thank you for the update!