Closed untom closed 9 years ago
How about this strategy:
argmax
, we do allocate memory regardless of handler. Even if the handler is PyCuda, we might have our own kernels, or some other library for doing pooling which does use the memory. The library is not focused on extreme efficiency I'd say, so for now we don't need to worry about this. We can see if this becomes an issue for training our models, and think of something like approach 1 above.One problem with that: argmax
contains integers. However, all of our internals are assumed to be floating point numbers, and there currently is no way to request a different dtype.
Yes, these and dropout masks for examples will have to be float for now. We can discuss plans to work around this in the future, but it will involve additional kernels, for example.
While implementing convolutions/pooling, I've stumbled upon the following problem: Depending on how you implement the operations, you might need more/less storage. Specifically, the GPU and CPU implementations might need different amounts of it / might not need it at all. Currently I have two examples for this:
In both cases, one of the two handlers needs additional storage, while the other doesn't. What's even weirder: the
argmax
can be seen as a buffer, and could be handled by the buffer manager. However, that'd lead to wasting memory on the GPU, where we'd allocate the buffer but never use it (which might also be confusing to users who inspect these buffers expecting them to mean something). The descriptors OTOH are cudnn-specific structures and probably not meant to be stored in buffers.I can think of twho solutions
handler.allocate_pool/conv_specific_memory(...)
that returns some sort of opaque datastructure (maybe a list of descriptors, allocations), which is then stored within each layer and always passed to the conv/pooling methods...I'm not superhappy with either solution, since both are slightly ugly. I like solution 1 slightly more, but it has the additional problem of making the API a bit more complicated. What do you guys think?