search

satyanshu404 / Prunning-of-NN

0 stars 0 forks source link

Can you flesh this out a bit more? #1

Open rishacha opened 1 year ago

rishacha commented 1 year ago

HI @satyanshu404 ,

I really liked what you've tried to accomplish by pruning your model at various sparsities -

The Lottery Ticket Hypothesis - A randomly-initialized, dense neural network contains a subnetwork that is initialized such that, when trained in isolation it can match the test accuracy of the original network after training for at most the same number of iterations

Is it possible to go beyond 50% sparsity - which is showing a compression ratio of 2.0?

Also can you add a few things like -

  1. Explanation (with a few sample images) and EDA of your dataset
  2. Explanation of your model - Why you've chosen what length at each layer
  3. A tensorboard training graph instead of logs (looks better in general)
  4. Add some explainability params to your model - Clients ask us about this all the time.

On a parallel line of thought (w.r.t. your email), we're stuck with issues on OCR and the accuracy for OCR. As a side project, can you look at better detection and accurate extraction. I can forward a case to my manager for this if you're able to do something on these lines -

  1. Improving detection of text on some difficult examples
  2. Using generative pre-training for OCR
  3. Combining OCR with a transformer model like BERT to improve accuracy.

Thanks for reaching out. Hope to hear back from you soon. Good Luck !

satyanshu404 commented 1 year ago

I'm glad that you loved my work regarding Lottery Ticket Hypothesis

Regarding your query about going beyond 50% sparsity level, it is possible but making the network more sparse can lead to underfitting or high bias, this is one factor and there are other factors such as dataset size and network architecture. This is what I learned in college, which is why I avoided going beyond 50%. And Fun fact is that I achieved higher accuracy from 60% sparsity level with 2.5% compression ratio.

I will definitely add the things you mentioned.

Regarding your OCR project, I can certainly help, but I need to familiarize myself with concepts such as transformers or BERT first. Frankly, I was planning to going beyond MLP or CNN models, but I had to put them on hold due to my end-semester exams.

HI @satyanshu404 ,

I really liked what you've tried to accomplish by pruning your model at various sparsities -

The Lottery Ticket Hypothesis - A randomly-initialized, dense neural network contains a subnetwork that is initialized such that, when trained in isolation it can match the test accuracy of the original network after training for at most the same number of iterations

Is it possible to go beyond 50% sparsity - which is showing a compression ratio of 2.0?

Also can you add a few things like -

  1. Explanation (with a few sample images) and EDA of your dataset
  2. Explanation of your model - Why you've chosen what length at each layer
  3. A tensorboard training graph instead of logs (looks better in general)
  4. Add some explainability params to your model - Clients ask us about this all the time.

On a parallel line of thought (w.r.t. your email), we're stuck with issues on OCR and the accuracy for OCR. As a side project, can you look at better detection and accurate extraction. I can forward a case to my manager for this if you're able to do something on these lines -

  1. Improving detection of text on some difficult examples
  2. Using generative pre-training for OCR
  3. Combining OCR with a transformer model like BERT to improve accuracy.

Thanks for reaching out. Hope to hear back from you soon. Good Luck !