search

taichi-dev / taichi

Productive, portable, and performant GPU programming in Python.
https://taichi-lang.org
Apache License 2.0
25.5k stars 2.28k forks source link

Serialize and deserialize sparse data structrure from/to buffer and file #5428

Open xuhao1 opened 2 years ago

xuhao1 commented 2 years ago

Concisely describe the proposed feature I would like to serialize/deserialize sparse data structure with to a buffer or to file, this will benefit online transmission and data logging/replay

Describe the solution you'd like (if any)

#Serialize
B = B.pointer(ti.ijk, (K, K, K))
v = ti.field(ti.i32)
B.place(v)
#To buffer.
buf = ti.data_to_buffer(v)
#To file
ti.data_to_file(v)

#Deserialize
B, v = ti.from_buffer(buf) #B is the SNode, v is the value
B, v = ti.from_file(file)

Additional comments Personally, I think this feature is very important for using taichi for high-performance computation, which requires distributed computing and data logging. The reason the store/load sparse data is hope it can have compact structure to save the memory.

jim19930609 commented 2 years ago

This seems to be a feature request to our AOT system. For now AOT only serializes the "NdarrayType" or "SNodeTreeType", and we should enable serialization for the underlying data buffers as well.

Updated LLVM AOT tasks to address this issue: https://github.com/taichi-dev/taichi/issues/4800

xuhao1 commented 2 years ago

@jim19930609 Thanks! I will take a look.