search

soft-matter / trackpy

Python particle tracking toolkit
http://soft-matter.github.io/trackpy
Other
443 stars 131 forks source link

Stream Example help #439

Open apiszcz opened 7 years ago

apiszcz commented 7 years ago

I am trying to reproduce the streaming example: http://soft-matter.github.io/trackpy/v0.3.0/tutorial/on-disk.html

When I review the test.h5 result here, no particle column is present. This must be my error, since the example clearly shows a particle column. (BTW: those columns are some condensed visually on the page in the example).

Any pointers appreciated.

I'm also curious if I can write to the h5 with one process and read with another in the streaming case. I have not reviewed the code but speculate there is no queue control 'built-in'.

Example :

records=1000
tracks=100
df = pd.DataFrame(np.random.randn(records, 2),columns=['x','y'])
# create frame #
a1=np.arange(0,records)
frame=a1%tracks
df['frame']=frame
df=df.sort_values(by='frame')

store = pd.HDFStore('test.h5')
store.put('df',df)
store.close()

with tp.PandasHDFStore('test.h5') as s:
    for linked in tp.link_df_iter(s, 30, neighbor_strategy='KDTree'):
        s.put(linked)
danielballan commented 7 years ago

Any pointers appreciated.

I think your trouble is coming from mixing pd.HDFStore and tp.PandasHDFStore. The latter assumes a certain structure that the former does not provide.

I'm also curious if I can write to the h5 with one process and read with another in the streaming case. I have not reviewed the code but speculate there is no queue control 'built-in'.

The latest version of HDF5 does support SWMR (Single Writer Multiple Readers) so it should be possible to build something like that, but as you guessed no queue control is built in. If you develop a working example it would be great to add it to the documentation.