Input as timeseries - Githubissues

ulkursln commented 4 years ago

Thank you for the ResNet support.
Is it possible run ResNet code with timeseries instead of an image? data format of time series that is used from the trained ResNet is: (x,y,z) ( (length of time series/#timestamps), #timestamps, #channels)

suraj-srinivas commented 4 years ago

Sure, it is definitely possible to run the ResNet code with timeseries. The only requirement for a custom model definition is the presence of getBias() and getFeatures() functions.

If your model definition is a simple modification of existing default pytorch resnet.py example then you may perform the same modification in this repository's model/resnet.py file and it should work as-is.

suraj-srinivas commented 4 years ago

I have updated the readme to include more details about the interface structure. Do let me know if that helps.

ulkursln commented 4 years ago

Thank you very much for your quick feedback. I have already read your paper and reviewed the code. I like the idea that you offer. It would be great to reach further users/researchers. I think supporting following items would increase the numbers of target users/researchers:

I have trained custom Resnet by using Keras with Tensorflow backend. Since there is no easy and robust way to convert the model and weight classes developed in Keras into Pytorch, it is more reasonable to support Keras models in visualization project. I reviewed the code, it seems it is highly dependent to Pytorch models. What do you think about this issue, is it easy to support Keras models as well?
I use batch normalization, dropout and L2 regularizations in the design of ResNet model. In addition, the number of layers and filters are different in our custom ResNet model comparing to the classic ResNet architecture. So, I think, it would be better to add feature for supporting custom networks easily.
If you reviewed the literature about the visualization of CNN models that take timeseries as an input, you will see there is not much study on it. With the increase in the usage of CNN models for timeseries , it is also necessary to handle timeseries as input by considering the differences between the conv1d and conv2d functions(the effect of channels might differ between two functions)

I am aware this is an open source project so, i put them here to the all who study on the project.

suraj-srinivas commented 4 years ago

I don't have plans for Keras support at the moment. The goal is to improve and maintain the Pytorch variant.
BatchNorm, Dropout, L2 shouldn't be problematic. The current implementation handles all that. I agree with you that it would be desirable to include support for any arbitrary custom network. However, an arbitrary model implies there is an arbitrary DAG possible. The general way to obtain an ordered listing of bias/feature nodes is to perform a graph traversal (BFS). This would be possible if all model definitions were amenable to this form of graph search naturally. This means that given any feature node, there must be a unified way to obtain the set of child nodes. While this is possible to have, this involves placing heavy restrictions on the way the user can write the model definition, which seemed the same as asking them to implement getBias() and getFeatures(). What do you think, is it cognitively easier for the user to adhere to a standard for model specification than implementing getBias()/getFeatures()? Particularly when the average user would want to use the saliency routine as a black box? Or is there another way I haven't thought of?
That's interesting, I didn't know of the paucity of visualization for time-series models. I realise now that the FullGrad.saliency code I wrote works only for image data, and needs slight modification to works with other forms of data. I will fix this such that support for 1D / 3D convolutions is also included, which might be useful for speech, nlp, etc.

Thanks for these suggestions! These are really helpful for me to improve the code.

ulkursln commented 4 years ago

Sure, it is not reasonable to force users for standard model specifications. In this case, I think developing code as parametric as possible and adding wiki pages to give information about handling custom networks on a sample case would improve usability.

suraj-srinivas commented 4 years ago

Hi, I've re-implemented the algorithm such that it now works with any ReLU / BN model. Hopefully it is easier to use now, and may be of interest!

idiap / fullgrad-saliency

Input as timeseries #2