lost data when load data

amutu commented 10 years ago

this may be the reason of #34 when load data use table_load(),some columns of cs has less data than the ori table. this is the reproduce step:

crash=> select crashlog_truncate();

crashlog_truncate

(1 row)

crash=> select crashlog_load(filter := 'logtime >= $start$2014-04-14$start$ and logtime < $end$2014-04-14 01:00:00$end$'); crashlog_load --------------- 902405 (1 row)

---------------!!!here ,class1 and class2 has different cnt compare with other columns.---- crash=> select * from (select x.clientversion a ,cs_count(x.uin) b,cs_count(x.uin) c,cs_count(x.logtime) d, cs_count(x.class1) e, cs_count(x.class2) f, cs_count(x.class3) g ,cs_count(x.revision) h,cs_count(x.phoneid) i from wx_version,crashlog_get(clientversion) as x where x is not null) t where a = 604176600 and (b <> c or b<> d or b<> e or b<> f or b <> g or b<> h or b<> i); a | b | c | d | e | f | g | h | i -----------+-----+-----+-----+---+----+-----+-----+----- 604176600 | 171 | 171 | 171 | 5 | 29 | 171 | 171 | 171 (1 row)

-----102 id has this problem crash=> select count(1) from (select x.clientversion a ,cs_count(x.uin) b,cs_count(x.uin) c,cs_count(x.logtime) d, cs_count(x.class1) e, cs_count(x.class2) f, cs_count(x.class3) g ,cs_count(x.revision) h,cs_count(x.phoneid) i from wx_version,crashlog_get(clientversion) as x where x is not null) t where b <> c or b<> d or b<> e or b<> f or b <> g or b<> h or b<> i;

count

102 (1 row)

-----ori table cnt crash=> select x.clientversion,count(x.uin) b,count(x.uin) c,count(x.logtime) d, count(x.class1) e, count(x.class2) f, count(x.class3) g ,count(x.revision) h,count(x.phoneid) i from crashlog_p_20140414 x where logtime <= '2014-04-14 01:00:00' and clientversion = 604176600 group by 1; clientversion | b | c | d | e | f | g | h | i
---------------+-----+-----+-----+-----+-----+-----+-----+----- 604176600 | 171 | 171 | 171 | 171 | 171 | 171 | 171 | 171 (1 row)

amutu commented 10 years ago

I can't reproduce this bug on 64fb76fbfb68285c35802b33cdf3da99a293fb95 tree,so I think it is the "Support load of unlimited varying size strings to columnar store using dictionary" commit introduce the bug.

in the bug branch ,the imcs.dictionary_size is set to 0

knizhnik commented 10 years ago

I have no idea how IMCS dictionary can some influence on this problem. It works only if you are importing to columnar store attributes with unlimited type (VARCHAR). But you are not using such attributes, are you?

I tried to reproduce the problem but didn't succeed. Please notice that snapshot of your data I am using is different from what you have now. So I have to adjust your queries:

postgres=# select crash_log_load(filter := 'logtime >= $start$2014-04-09$start$ and logtime < $end$2014-04-09 01:00:00$end$');

crash_log_load

(1 row) postgres=# select * from (select x.clientversion a ,cs_count(x.uin) b,cs_count(x.uin) c,cs_count(x.logtime) d, cs_count(x.class1) e, cs_count(x.class2) f, cs_count(x.class3) g ,cs_count(x.revision) h,cs_count(x.phoneid) i from wx_version,crash_log_get(clientversion) as x where x is not null) t where a = 604176600 and (b <> c or b<> d or b<> e or b<> f or b <> g or b<> h or b<> i); a | b | c | d | e | f | g | h | i ---+---+---+---+---+---+---+---+--- (0 rows)

select count(1) from (select x.clientversion a ,cs_count(x.uin) b,cs_count(x.uin) c,cs_count(x.logtime) d, cs_count(x.class1) e, cs_count(x.class2) f, cs_count(x.class3) g ,cs_count(x.revision) h,cs_count(x.phoneid) i from wx_version,crash_log_get(clientversion) as x where x is not null) t where b <> c or b<> d or b<> e or b<> f or b <> g or b<> h or b<> i;

count

(1 row)

May be the problem is caused by "pinning" of old version of IMCS data in shared memory? Did your try to restart Postgres and clear all shared memory segments?

knizhnik / imcs