search

onemoredata / bagger

Massive log storage in PostgreSQL
BSD 2-Clause "Simplified" License
12 stars 1 forks source link

LW_tests return varied number of tests #117

Open einhverfr opened 10 months ago

einhverfr commented 10 months ago

The 51_lwtest.t returns anywhere between 28 and 36 tests, usually 34 or 36. We need to figure out why and fix it.

einhverfr commented 10 months ago

I have tracked down this problem. The issue is that tests sometimes take a little bit of time to hit the anyevent queue. As a result some of the time, the config messages are never processed all the way before we exit. In particular, the Etcd messages don't get processed all the way through by the time the process exits.

This seems likely to be fairly deep in the Net::Etcd layer. This lends support to the question of forking the driver and fixing it for our purposes or of changing our approach to dealing with Etcd. These however take more effort on the research side.

einhverfr commented 10 months ago

Currently the tests always seem to run 36 or occasionally 38 tests.

einhverfr commented 10 months ago

Here's a diff between two test outputs that shows the problem. in the 38 test case, the config stuff wrote to etcd before the script exited. In the other, they had not.

27,36c27,38
< ok 27 - Write to kvstore succeeded: /PostgresInstance/host2/5432
< ok 28 - Write to kvstore succeeded: /PostgresInstance/host3/5432
< ok 29 - Got expected message table storage.config: INSERT: id[integer]:13 key[text]:'testing1' value[json]:'"1"'
< ok 30 - Got back expected hashref for table storage.config: INSERT: id[integer]:13 key[text]:'testing1' value[json]:'"1"'
< ok 31 - Got correct key /Config/testing1
< ok 32 - Got expected message table storage.config: INSERT: id[integer]:14 key[text]:'testing2' value[json]:'"Foo"'
< ok 33 - Got back expected hashref for table storage.config: INSERT: id[integer]:14 key[text]:'testing2' value[json]:'"Foo"'
< ok 34 - Got correct key /Config/testing2
< ok 35 - Got expected message COMMIT
< ok 36 - Got back expected hashref for COMMIT
---
> ok 27 - Got expected message table storage.config: INSERT: id[integer]:13 key[text]:'testing1' value[json]:'"1"'
> ok 28 - Got back expected hashref for table storage.config: INSERT: id[integer]:13 key[text]:'testing1' value[json]:'"1"'
> ok 29 - Got correct key /Config/testing1
> ok 30 - Got expected message table storage.config: INSERT: id[integer]:14 key[text]:'testing2' value[json]:'"Foo"'
> ok 31 - Got back expected hashref for table storage.config: INSERT: id[integer]:14 key[text]:'testing2' value[json]:'"Foo"'
> ok 32 - Got correct key /Config/testing2
> ok 33 - Write to kvstore succeeded: /PostgresInstance/host2/5432
> ok 34 - Write to kvstore succeeded: /PostgresInstance/host3/5432
> ok 35 - Write to kvstore succeeded: /Config/testing2
> ok 36 - Write to kvstore succeeded: /Config/testing1
> ok 37 - Got expected message COMMIT
> ok 38 - Got back expected hashref for COMMIT