stream element duplicates

[tgsell]

I have spotted sporadic stream element duplicates in our GSN database. The only value which differs within a duplicate, are the TIMED entries. They differ in a few milliseconds. All other values are exactly the same and are distinctive duplicates. The VSs in which I have spotted these duplicates are using the local wrapper connected to one other VS and the scriptlet processor with a simple time conversion groovy script.

Do you have ever met the same problem or have any clue what could cause this duplicates?

[ebiiii]

The local wrapper is using the DataDistributer to subscribe to another VS and so far I haven't had those kind of issues with the DataDistributer (but it would be my first suspect)... Are the duplicates also happening in the source VS? Is the TIMED field also provided by this other VS or is it the "received time" of the stream element in the local wrapper? The scriplet processor has a single output for each single input, right? So I don't think it would generate duplicates... What is the data rate? in the order of milliseconds, seconds or minutes (or larger)? Maybe with high data rate some race condition may appear...

[nkryvych]

I also noticed that there are some duplicated values returned by GSN rest (in swissex), if you look at the following link you'll see each timestamp repeated 3 times. see http://montblanc.slf.ch:22001/multidata?nb=ALL&from=01/01/2015+00:00:00&to=02/01/2015+00:00:00&vs%5B0%5D=wannengrat_wan5&field%5B0%5D=wind_direction&vs%5B1%5D=wannengrat_wan5&field%5B1%5D=wind_speed_vector_av&vs%5B2%5D=wannengrat_wan5&field%5B2%5D=wind_speed_max&vs%5B3%5D=wannengrat_wan5&field%5B3%5D=wind_speed_scalar_av&time_format=ISO&download_format=csv&download_mode=inline

this happens for different VS.

[ebiiii]

Do you have a way to see what is submitted to the wrapper? (like an csv file or a dump of the network trafic, to make sure duplication happens inside GSN and also to try reproducing the bug) Which wrapper is used there?

[joelfiddes]

The following is a section of the source csv file from the example link above (WAN5). It shows no duplication, indicating that duplication is occurring inside GSN:

01.01.2015 00:20:00,78823,98.7,-6.433,1.676,1.623,303.1,14.46,2.249,-6.221,75.26,-0.294,4.326,200$ 01.01.2015 00:30:00,78824,99.2,-6.39,0.888,0.828,337.7,20.01,2.562,-6.027,75.45,-0.387,3.554,200.$ 01.01.2015 00:40:00,78825,99,-6.811,0.69,0.418,91.1,48.22,1.154,-5.931,75.15,-3.624,0.418,200.1,2$ 01.01.2015 00:50:00,78826,99.1,-7.174,0.696,0.602,125.6,23.05,1.78,-6.178,75.02,-2.679,1.126,199.$ 01.01.2015 01:00:00,78827,99.4,-7.62,0.178,0.175,358.9,6.664,0.548,-6.591,75.32,-2.958,-0.949,198$ 01.01.2015 01:10:00,78828,99.3,-7.846,0.612,0.408,343.3,41.8,1.252,-6.92,75.47,-0.867,2.284,198.9$ 01.01.2015 01:20:00,78829,98.8,-7.671,0.669,0.411,260.7,50.27,1.584,-6.848,75.4,-1.301,3.635,199.$ 01.01.2015 01:30:00,78830,99.1,-7.536,0.905,0.574,320.9,49.03,1.858,-6.647,75.71,0.403,3.828,200,$ 01.01.2015 01:40:00,78831,98.1,-7.083,1.467,1.02,257.3,44.71,2.503,-6.595,75.12,0.465,4.777,200.6$ 01.01.2015 01:50:00,78832,97.5,-6.916,1.224,1.129,274.4,22.54,2.621,-6.273,74.98,-1.115,4.278,200$ 01.01.2015 02:00:00,78833,97.2,-6.761,1.523,1.445,274.1,18.33,2.562,-6.233,75.21,-0.65,4.487,201.$

VS info: Processing class: gsn.processor.ScriptletProcessor wrapper="csv"

[jpcik]

i think this is a different problem to the one identified by tgsell. The Wan5 seems to be a result of several previous problems (bugs), e.g. insertion of null values, and it seems that the wrapper has inserted the same data several times. I would think that it is a matter of cleaning the table from a given date, set the csv checkpoint to that point in time and rerun the csv wrapper.

[nkryvych]

I think JP probably is right. Sensors wannengrat_wan6 and wannengrat_wan4 have duplicates, but that duplicates started to appear for later data, there are no duplicates for 2006 for example. Also there are no duplicates for some other sensors (for example imis_fka_2). Joel, did you check data in the DB?

On Thu, Apr 23, 2015 at 8:29 PM, Jean-Paul Calbimonte < notifications@github.com> wrote:

i think this is a different problem to the one identified by tgsell. The Wan5 seems to be a result of several previous problems (bugs), e.g. insertion of null values, and it seems that the wrapper has inserted the same data several times. I would think that it is a matter of cleaning the table from a given date, set the csv checkpoint to that point in time and rerun the csv wrapper.

— Reply to this email directly or view it on GitHub https://github.com/LSIR/gsn/issues/48#issuecomment-95678155.

[joelfiddes]

Which data in database NAtaliya? Your link above which first identified the problem pulls data from the database..

Looks like a case of rereading in WAN data. Is this likely to be as widespread as the NULL value bug JP if you think connected?

[ebiiii]

I tried to run the current version of GSN (branch master) with a single VS (WAN5.xml as you sent it to me) and inputting a csv file with the same data as you have here. But no duplicates appeared... As Jean-Paul said, that could have happened by resetting the csv-checkpoint.

Regarding the original bug from tgsell, I haven't seen so far a similar bug on the other deployment I've checked.

[tgsell]

The data rate in the VS I have spotted the duplicates is very high, in the order of a few milliseconds, for tens of thousands of stream elements.

The source VS does not contain any duplicates, thus, they have to be produced somewhere in between. The TIMED field is not provided by the other VS, it is the "received time". And the scriplet processor has a single output for each single input.

Ok, right now I have done some more investigations in the DB. It looks as if the duplicates only occur for stream elements from the source VS with the same TIMED value. Like this:

Source VS: PK TIMED DATA 1 12345678 blabla1 2 12345678 blabla2 3 22334455 blabla3

Receiving VS (local wrapper): PK DATA 1 blabla1 2 blabla1 3 blabla2 4 blabla3

Thus, right now it looks as if it has something to do with equal TIMED fields in two stream elements in the source VS (which I thought should not be a problem) and their propagation to the next VS using the local wrapper.

[ebiiii]

Now it reminds me something... I had to solve this issue in the OpenSense branch: https://github.com/LSIR/gsn/commit/787000f4797b7334ad44e2719848be6bdff01324#diff-e0f4452911cf37f8ee3782b4fa10685dL297 . Maybe this can also solve your problem. I'll keep this issue open until I also merge the fix into the master branch (I don't know why I didn't)

[ebiiii]

opened a pull request #53