search

hanatos / vkdt

raw photography workflow that sucks less
https://jo.dreggn.org/vkdt
BSD 2-Clause "Simplified" License
389 stars 36 forks source link

Computational graph #22

Closed ib00 closed 3 years ago

ib00 commented 3 years ago

Is the computational graph (graph and nodes) general enough that could be repurposed for say deep learning?

Or would it be better to start from scratch?

hanatos commented 3 years ago

hi,

the graph is very general and completely disconnected from the ui. the command line interface can serve as a code example how graph processing without gui (also without the compile time dependencies) can be done.

i'm not interested in deep learning. vkdt does ship with an implementation of adam, but it's based on finite differences for the gradients. you'll likely want to implement some automatic differentiation scheme for large scale optimisation problems. i'm using it for thing like fitting a vignetting model to an image, with comparatively low dimensionality.

deep learning on images is so mainstream nowadays i cannot imagine starting from scratch would be good use of time.

trougnouf commented 3 years ago

Adam would likely not be needed since the model should be optimized with PyTorch or Tensorflow, what's needed are 2D convolutions and ReLU activations.

hanatos commented 3 years ago

oh you only want to implement the inferencing? that should not be hard indeed. it seems it's even possible to use nv's tensor cores in glsl (GL_NV_cooperative_matrix).

hanatos commented 3 years ago

not sure an issue is the correct place to discuss such bluesky endeavours.. but anyways i would be interested in fast/high-quality denoising with offline training.

some blur routines are here for instance: https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/api.h#L178

convolution you probably want to implement in a similar fashion with generic weights. i suppose the simplest module to take as a starting point would be exposure: https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/exposure/main.comp (such a waste of memory bandwidth)

to construct a whole interconnected block of nodes inside a module, this can be done in a cpu-side main.c, similar to: https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/denoise/main.c#L53 (the wavelet-based denoiser)

if you want to make your networks recurrent, you can define "feedback" connectors which will give you the previous frame in an animation.

hanatos commented 3 years ago

(anyways let me know if you need support setting up stuff, maybe direct email is easier)