Issue 342 | TCP/UD Socket based Table Streamer

[dilawar]

About

This PR implements a socket streamer and includes a couple of tests. Related to #342 . The socket streamer:

• Opens a TCP socket on the localhost and on port 31416. Or open a unix domain socket.
• Stream table data on this socket in JSON format whenever there is "some" data to send e.g. { "/table1" : [ [0,1], [0.1,3], [0.2.4.3]]} where data is in pair of time and value.
• Clear up table data after streaming it.
• The socket server is launched in its separate thread which is killed as soon as destructor of 'SocketStreamer' class is called, i.e. if any data is left in the output buffer of socket, it goes away with it. Hence, user should sleep for a while before moose is terminated. May be we can force data to be flushed in blocking mode..
• Socket does not block moose simulations. Very quick simulations requires sufficient delay before moose.start and after moose.start. Its the job developer to have sufficient decay. At least 0.5 second is recommended.
• Socket is automatically created and all tables are added to it when environment variable MOOSE_STREAMER_ADDRESS is set e.g. MOOSE_STREAMER_ADDRESS=file:///tmp/MOOSE. This would be useful for FindSim.

Read the added test tests/python/test_socket_streamer_uds.py or tests/python/test_socket_streamer_tcp.py on how to use it.

This feature is meant for developers. It will be useful if simulation takes long time and one needs to update plots/data or some other visualization once in a while without waiting for moose to finish simulation. Another possibility is to use moose.Streamer which writes data to a csv file every 10 seconds and read this file to update the plots/UI periodically. Socket has the advantange of being on TCP and can be connected easily on network.

Dependencies

None. c++11 is required. Work in Linux and OSX.

Build

This feature is enabled by default in cmake. To disable it, pass '-DWITH_TCP_STREAMER=OFF` to cmake.

• Tests Yes. A python test in tests/python directory. It tests for basic functionality.

Documentation

• NA. Only doxygen documentation is available.
• Examples and snippets None.

Notes

The streamer works fine but has not been benchmarked for performance.

Below is a run of test script when moose was compiled in DEBUG mode.

[dilawars@Ace _build (issue_342)]$ python ../tests/python/test_socket_streamer.py 
[INFO] Using moose form /home/dilawars/Work/GITHUB/DILAWAR/moose-core/_build/python/moose/__init__.pyc
[INFO] Socket client is running now
Py: Trying to connect to 127.0.0.1, 31416
<moose.Table2: id=460, dataIndex=0, path=/compt[0]/a[0]/tab[0]> <moose.Table2: id=461, dataIndex=0, path=/compt[0]/tabB[0]>Py: [Errno 111] Connection refused
Py: Trying to connect to 127.0.0.1, 31416
 <moose.Table2: id=462, dataIndex=0, path=/compt[0]/tabB[0]/tabC[0]>
Py: [Errno 111] Connection refused
Py: Trying to connect to 127.0.0.1, 31416
Py: Connected with socket.
MOOSE is done
recieved data:
 {"/compt/a/tab":[[0,1],[1,0.923317],[2,0.847771],[3,0.781319],[4,0.722945],[5,0.671728]],"/compt/tabB":[[0,2],[1,2.07705],[2,2.15295],[3,2.21971],[4,2.27835],[5,2.32981]],"/compt/tabB/tabC":[[0,0.5],[1,0.576988],[2,0.652833],[3,0.719545],[4,0.778146],[5,0.82956]]}
{"/compt/a/tab":[[6,0.626839],[7,0.587531],[8,0.553138],[9,0.523068],[10,0.496793]],"/compt/tabB":[[6,2.3749],[7,2.41439],[8,2.44894],[9,2.47914],[10,2.50554]],"/compt/tabB/tabC":[[6,0.874621],[7,0.914079],[8,0.948601],[9,0.978785],[10,1.00516]]}
{"/compt/a/tab":[[11,0.473846],[12,0.453815],[13,0.436338],[14,0.421093],[15,0.4078]],"/compt/tabB":[[11,2.52859],[12,2.54871],[13,2.56626],[14,2.58158],[15,2.59493]],"/compt/tabB/tabC":[[11,1.02819],[12,1.0483],[13,1.06584],[14,1.08114],[15,1.09448]]}
{"/compt/a/tab":[[16,0.396212],[17,0.386113],[18,0.377313],[19,0.369647],[20,0.362969]],"/compt/tabB":[[16,2.60657],[17,2.61671],[18,2.62555],[19,2.63325],[20,2.63996]],"/compt/tabB/tabC":[[16,1.10611],[17,1.11625],[18,1.12508],[19,1.13278],[20,1.13948]]}
{"/compt/a/tab":[[21,0.357154],[22,0.352089],[23,0.347679],[24,0.34384],[25,0.340497]],"/compt/tabB":[[21,2.6458],[22,2.65089],[23,2.65532],[24,2.65917],[25,2.66253]],"/compt/tabB/tabC":[[21,1.14532],[22,1.1504],[23,1.15483],[24,1.15868],[25,1.16203]]}
{"/compt/a/tab":[[26,0.337587],[27,0.335055],[28,0.33285],[29,0.330931],[30,0.329261]],"/compt/tabB":[[26,2.66545],[27,2.668],[28,2.67021],[29,2.67214],[30,2.67382]],"/compt/tabB/tabC":[[26,1.16495],[27,1.1675],[28,1.16971],[29,1.17164],[30,1.17331]]}
{"/compt/a/tab":[[31,0.327807],[32,0.326542],[33,0.325441],[34,0.324483],[35,0.323649]],"/compt/tabB":[[31,2.67528],[32,2.67655],[33,2.67765],[34,2.67862],[35,2.67945]],"/compt/tabB/tabC":[[31,1.17477],[32,1.17604],[33,1.17715],[34,1.17811],[35,1.17894]]}
{"/compt/a/tab":[[36,0.322924],[37,0.322292],[38,0.321743],[39,0.321265],[40,0.320849]],"/compt/tabB":[[36,2.68018],[37,2.68082],[38,2.68137],[39,2.68185],[40,2.68227]],"/compt/tabB/tabC":[[36,1.17967],[37,1.18031],[38,1.18086],[39,1.18134],[40,1.18176]]}
{"/compt/a/tab":[[41,0.320487],[42,0.320172],[43,0.319897],[44,0.319659],[45,0.319451]],"/compt/tabB":[[41,2.68263],[42,2.68295],[43,2.68322],[44,2.68346],[45,2.68367]],"/compt/tabB/tabC":[[41,1.18212],[42,1.18243],[43,1.18271],[44,1.18295],[45,1.18316]]}
{"/compt/a/tab":[[46,0.319271],[47,0.319114],[48,0.318977],[49,0.318858],[50,0.318754]],"/compt/tabB":[[46,2.68385],[47,2.68401],[48,2.68415],[49,2.68427],[50,2.68437]],"/compt/tabB/tabC":[[46,1.18334],[47,1.1835],[48,1.18363],[49,1.18375],[50,1.18386]]}

[INFO] All tests passed

Addendum

I have added a server which can handle upto 100 clients. The server can be launched by

python -m moose.server

Job to this server can be submit by scripts/submit.py script. It sends the required data to server, run the simulation and collects the artifacts generated by simulation. Artifacts are sent back in an archive file (tar.bz2 format). e.g.

./submit.py -s ghevar.ncbs.res.in:31417 chemDoseResponse.py 

There are some rough edges to handle and more are likely to show up with integration with FindSim framework. But to get going, this merge is required.

A GUI client is developed here https://github.com/dilawar/moose-client

[dilawar]

@surbhit21 Any comment?

[dilawar]

@upibhalla review is needed.

[dilawar]

This PR is now in good shape and can be merged for production testing by @surbhit21 @BhallaLab/findsim

I have added a server which can handle upto 100 clients. The server can be launched by

python -m moose.server

Job to this server can be submit by scripts/submit.py script. It sends the required data to server, run the simulation and collects the artifacts generated by simulation. Artifacts are sent back in an archive file (tar.bz2 format). e.g.

./submit.py -s ghevar.ncbs.res.in:31417 chemDoseResponse.py 

There are some rough edges to handle and more are likely to show up with integration with FindSim framework. But to get going, this merge is required.

[dilawar]

@upibhalla @surbhit21 I am going to merge this into master today itself. This PR does not modify any existing code. Merging it required for further work on another PR which has a GUI for submitting jobs to moose server.

Have a look if you find time.