search

matjesg / deepflash2

A deep-learning pipeline for segmentation of ambiguous microscopic images.
https://matjesg.github.io/deepflash2/
Apache License 2.0
58 stars 11 forks source link

Ensembling Issue #18

Closed jaideep11061982 closed 3 years ago

jaideep11061982 commented 3 years ago

@matjesg
There comes another issue which is while Ensembling of two models. Some how there is Either a Memory Leak or GPU RAM leak so our Sub are failing. we are clueless on this. This succeeded once but now not succeeding at all. any help would be appreciated.

We use Ensembler Model that does mean of predictions of model we want to ensemble. Single model works fine. two models ensembling fails.

2) How can we use Ensemble Learner class.Willl it help

matjesg commented 3 years ago

Hi @jaideep11061982 , I'm sorry that you're facing these issues - could you provide a more detailed description/error messages?

You can find the code for ensembling (averaging) models here: https://github.com/matjesg/deepflash2/blob/master/deepflash2/learner.py#L490

matjesg commented 3 years ago

@jaideep11061982 Does V15 (https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub) fix the issue?

jaideep11061982 commented 3 years ago

Hi Matjesg Thanks .will check it out,will this affect score ?

Our all sub works fine with more n more models using your base inference except when just specifically 2 of model ,when they get ensembled we get sub errors ,but its non reproduce able over train and test data .we are clueless what is causing this issue. We submitted the kernel with nine models all went fine ,but just these 2 when ensembled causing they causing an issue .I kept monitoring ram usage during train data inference it never went beyond 8.5 gb still buffer of 5 gbs.

On Fri, 7 May 2021, 12:19 Matthias Griebel, @.***> wrote:

@jaideep11061982 https://github.com/jaideep11061982 Does V15 ( https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub) fix the issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matjesg/deepflash2/issues/18#issuecomment-834113139, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJDFNZGCVQR6HY3XLY5U7FTTMOEOXANCNFSM44CRZONA .

matjesg commented 3 years ago

The score increased slightly to 0.922 LB, but I'm not sure if we already have the best settings.

It's really hard to ensure that the submission is working without having any proper logs and information, I don't know what causes the errors and I still don't know if or if not V14 works...

jaideep11061982 commented 3 years ago

Thanks very much appreciate it Can you help me know what are

  1. prameters shift =0.8 means also its high or low value significance
  2. its impact on lb if you know . . 3.The purpose of get gaussian weights matrix.

On Fri, 7 May 2021, 15:43 Matthias Griebel, @.***> wrote:

The score increased slightly to 0.922 LB, but I'm not sure if we already have the best settings.

It's really hard to ensure that the submission is working without having any proper logs and information, we don't really know what causes the errors (I still don't know if or if not V14 works)...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matjesg/deepflash2/issues/18#issuecomment-834238474, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJDFNZHPWCE4QZUP4Q5XEWDTMO4KZANCNFSM44CRZONA .