search

Sujeeth13 / Analog_NAS

2 stars 0 forks source link

How to calculate number of Params? #3

Open asaficontact opened 3 months ago

asaficontact commented 3 months ago

Can we use an equation to calculate the number of params or would we need to implement it in pytorch and calculate the number of params that way? Try to find the easiest way possible.

We will need this helper function for issue #2

Sujeeth13 commented 3 months ago

I found two ways to do param count calculations. 1) build the entire network and then find the number of learnable parameters. 2) i made a formula which is pretty accurate

the first way obviously is very compute heavy, so we should go with the second way although sometimes im not sure why but i am off by a couple 100 parameters (not so much tbh and wont affect the rl agent training) but still we can look into this. Also i wanted to check how many architectures am i going wrong in but everytime i run a loop to cacluate the param count using method 1, system crashes, maybe too much RAM usage..

Sujeeth13 commented 3 months ago

Update: it is definetely RAM issue since some models have more that billion parameters. But if i skip those models then the formula is very accurate actually. it is sometimes off but only by a couple 100 parameters. I think it might be due to how i am counting the skip connections paramters tbh. We can fix this but in the meantime we should start training the agent as this wont affect the training. I pushed the code to param_feature branch if you want to review.

asaficontact commented 3 months ago

@Sujeeth13 Sounds good, I will review it today. I don't think off by around 100 parameters would make much of a difference at all. We can use the formulae for training, while we try to improve on it. Did you find the formulae in a paper or blog? Can you link it here?