spring-xd tail stream stops passing data

[ktelep]

It appears that the spring-xd tail module fails with a Java exception when an incomplete line of data is written to the tail_collect.log file.

Symptoms include an unexpected data format error (missing commas that are expected)

Undeploying and redeploying the stream resolves the issue.

[ktelep]

The plan is to replace the tail with a direct POST. Tail seems to be unreliable when we start working with larger volumes of data.

Tactical Solution

1. Add urllib to tilt_collect.py script adding HTTP POST capability
2. tilt_collect.py still grabs data from tilt_server, but then PUTs it to Spring locally

Strategic Solution

1. Export spring-xd HTTP port via firewall to the internet (maybe behind load balancer to avoid DDOS?)
2. Push data from tilt directly to spring-xd via HTTP POST

[ktelep]

Tactical solution complete in the latest commit. collect-post.py in contrib folder will support sending the data via POST to a local HTTP spring-xd server and is a drop in replacement for the existing collect.py script. Update the collect-post.py script with the appropriate URLs and Ports once spring stream is setup.

Example stream:

stream create --name "httptest" --definition "http --port=6667 | file" --deploy

Or for a more complete definition:

stream create --name "tilt_http" --definition "http --port=6667 | jdbc --columns='timestamp, iso_timestamp, devid, TiltLR, TiltFB, Direction, latitude, longitude, altitude, OS' --validationQuery='select 1;' --testOnBorrow=False" --deploy

With the SQL Table Definition:

create table tilt_http ( timestamp varchar(25), iso_timestamp varchar(40), devid varchar(20), TiltLR decimal, TiltFB decimal, Direction decimal, latitude decimal, longitude decimal, altitude decimal, OS varchar(20));