Online evaluation

[phil0stine]

I am wondering, since the data preparation happens in batch before being sent to the network, whether it would be reasonable to adapt it to run on-demand as a new cloud arrives, or whether the processing of each new input would make it prohibitively costly. Thanks

[HuguesTHOMAS]

Hi @phil0stine,

If you need to use KPConv for a real time application, then yes, it would be better to run on command.

It would be slower than a batch processing, in term of fps, but not that much. In term of delay though, it would be much better.

Best, Hugues

[phil0stine]

Thanks, my runtime concern comes from both the input preparation steps as well as the voting scheme to get final predictions. Do you think these could even run at 1fps?

[HuguesTHOMAS]

It depends a lot on the size of input point clouds. If you want to be faster, just sumbsaple your cloud a bit more, you will not lose that much performance doing so.

Concerning the voting scheme, it is not mandatory. You can decide for yourself if you want to use it or not. If you don't use a voting scheme and just keep the raw predictions, I believe that you can reach something like 10 fps for point clouds with 10000 points. (This is just an assumption, I did not test it).

[phil0stine]

Thanks very much! The size of the point cloud would be the order of magnitude of a standard lidar.

Side note, do you think parts of your input pipeline make sense to add to custom tf ops? I am thinking of this post in particular in which they converted everything to happen from within the graph, including KDTree apparently.

[HuguesTHOMAS]

My input pipeline already uses a custom op to compute the neighbors with a KDTree. In the post you link, they use a custom op with FLANN. In my implementation I use a custom op with Nanoflann, which is basically the same.

[phil0stine]

Hi, one last question. How would you suggest getting around the need for precomputing flat_inputs before Network creation, in an online setting?

[HuguesTHOMAS]

Hi,

In any case, you need to compute the subsamplings of layer points, and convolution neighbors at some point. In my opinion, this should be done on CPU. For an online setup, you could decide to avoid using an input queue, which creates a lot of delay between the time you process a frame, and the time it was actually measured by the sensor. Maybe use placeholder ? I am not familiar with the most efficient methods for real time testing in tensorflow.

If you want to use placeholders, it implies redesigning the whole input pipeline, which can be a lot of work, especially if you want to efficiently use CPU. Indeed the parallelization is currently done by the number of threads in the input pipeline, but without it, you might need to parallelize each input function, including subsampling and neighbors computations.

[phil0stine]

Hi yes I planned to use placeholders but that where I ran into problems with the input pipeline. It seems like the network constructor depends on having already processed the data; if I could defer some of the processing to be on demand but still set up the network I think it can work, does this sound reasonable? How fast is another question

On Thu, Apr 16, 2020, 3:37 PM Hugues THOMAS notifications@github.com wrote:

Hi,

In any case, you need to compute the subsamplings of layer points, and convolution neighbors at some point. In my opinion, this should be done on CPU. For an online setup, you could decide to avoid using an input queue, which creates a lot of delay between the time you process a frame, and the time it was actually measured by the sensor. Maybe use placeholder ? I am not familiar with the most efficient methods for real time testing in tensorflow.

If you want to use placeholders, it implies redesigning the whole input pipeline, which can be a lot of work, especially if you want to efficiently use CPU. Indeed the parallelization is currently done by the number of threads in the input pipeline, but without it, you might need to parallelize each input function, including subsampling and neighbors computations.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/HuguesTHOMAS/KPConv/issues/60#issuecomment-614854899, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYU5XV26CM7LFVLTZJTHXTRM5M7PANCNFSM4KVQWSUQ .

[HuguesTHOMAS]

Hi, yes it seems reasonable indeed, You can create your own model class (just copy KPFCNN_model.py and modify what you want in there). In the init method, you can handle the input data arriving as a list.

[phil0stine]

Ideally, I would use the input queue for training and placeholders for online inference. I am sure you are right that placeholders for training would be horribly slow; but I think I need them for inference.

Your suggestion to update the Network init method to handle the input as a list seems like it would preclude input queues; Just want to make sure I fully understand the advice.