search

Dobiasd / frugally-deep

A lightweight header-only library for using Keras (TensorFlow) models in C++.
MIT License
1.06k stars 235 forks source link

Support for views, spans, iterators? #415

Open stellarpower opened 6 months ago

stellarpower commented 6 months ago

Hi,

Thanks for this! C++ is making NNs a little bit tolerable. If I could train with this in lieu of python at all, life would be much better!!

It seems that tensor/tensors only have constructors for working directly with vectors of data. Would it be possible to support some other contiguous views, such as iterators, std::span, or maybe something from the ranges library, etc.?

I'm currently iterating over a long time series of data, and feeding in data one sequence at a time, to an LSTM-based model. It'd be nice if I could provide some non-owning reference to the start of the current page, and then the same for where frugally-deep can place the outputs, as currently I am copying into a shorter vector, running inference, and copying the output. I assume a copy of about 512 values is negligible compared to everything else that will be going on, but it'd still be nice to have. I assume a span would be the lowest-overhead way to achieve this.

Are there any technical reasons why this wouldn't be possible?

Thanks

Dobiasd commented 6 months ago

Hi, and thanks for the nice feedback and the interesting suggestion. Using std::span somewhere in frugally-deep would raise the compiler requirement from C++14 to C++20. So I'm a bit hesitant to add this feature.

stellarpower commented 6 months ago

True; but you could enable it conditionally if the compiler has this built-in. There are also third-party implementations of span for pre C++-20 compilers.

Spans also meet requirements for contiguous iterators, so it might be best to template it and be agnostic to the exact implementation - just assuming we have a non-owning view on the input which we can run over and read, and possibly also a view on the output we can increment and write to.

Dobiasd commented 5 months ago

Copying the input data from a span into a vector should usually not be significant regarding speed or memory usage, since in the prediction step of most models, much more is happening. With such a small gain as the potential win, I'm not sure if it's worth the complexity.

Are you interested in proposing an implementation in a PR?

stellarpower commented 5 months ago

I'm pretty short on time right now, but if I end up looking into it as part of the current project, then definitely happy to get something committed.