Elastic Weight Consolidation + regularization refactor

[varisd]

• Implemented EWC from: https://arxiv.org/pdf/1612.00796.pdf
• Moved the regularization to a separate module/classes.
• Added GradientRunner for easier gradient output.
• Added script for gradient averaging.

NOTE: currently is not working with DelayedUpdateTrainer due to a problem with trainer variable restoration.

[varisd]

Note, that currently this branch removes the default l1, l2 logging into summaries

[jlibovicky]

Oh, I didn't even notice. I'd like to keep the L2 plot in TensorBoard.

[varisd]

The L2 plot is back.

[jindrahelcl]

Právě kvůli ty sémantice bych to nedělal.. regularizers jsou regularizers a new regularizers and then some..

Dne pá 3. 8. 2018 21:48 uživatel Dušan Variš notifications@github.com napsal:

@varisd commented on this pull request.

In neuralmonkey/trainers/generic_trainer.py https://github.com/ufal/neuralmonkey/pull/742#discussion_r207649319:

unweighted losses for fetching

• self.losses = [o.loss for o in objectives] + [l1_value, l2_value]
• tf.summary.scalar("train_l1", l1_value,
• collections=["summary_train"])
• tf.summary.scalar("train_l2", l2_value,
• collections=["summary_train"])
• self.losses = [o.loss for o in objectives] + reg_values

• we always want to include l2 values in the summary

• if L2Regularizer not in [type(r) for r in self.regularizers]:
• reg_values.append(L2Regularizer().value(regularizable))

Ten L2Regularizer by se mel pridat i do self.regularizers.

reg_values se potom, co se pridaji do summaries, dale uz nikde nepouzivaji, takze neni problem tam pridat tenhle "default" regularizer

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ufal/neuralmonkey/pull/742#discussion_r207649319, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwcs4sS68v6cTEfJRcyTu-TgJa6xpDFks5uNKkLgaJpZM4VkjNe .

[jindrahelcl]

S/new/ne/

Dne pá 3. 8. 2018 21:54 uživatel Jindra Helcl jindra.helcl@gmail.com napsal:

Právě kvůli ty sémantice bych to nedělal.. regularizers jsou regularizers a new regularizers and then some..

Dne pá 3. 8. 2018 21:48 uživatel Dušan Variš notifications@github.com napsal:

@varisd commented on this pull request.

In neuralmonkey/trainers/generic_trainer.py https://github.com/ufal/neuralmonkey/pull/742#discussion_r207649319:

unweighted losses for fetching

• self.losses = [o.loss for o in objectives] + [l1_value, l2_value]
• tf.summary.scalar("train_l1", l1_value,
• collections=["summary_train"])
• tf.summary.scalar("train_l2", l2_value,
• collections=["summary_train"])
• self.losses = [o.loss for o in objectives] + reg_values

• we always want to include l2 values in the summary

• if L2Regularizer not in [type(r) for r in self.regularizers]:
• reg_values.append(L2Regularizer().value(regularizable))

Ten L2Regularizer by se mel pridat i do self.regularizers.

reg_values se potom, co se pridaji do summaries, dale uz nikde nepouzivaji, takze neni problem tam pridat tenhle "default" regularizer

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ufal/neuralmonkey/pull/742#discussion_r207649319, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwcs4sS68v6cTEfJRcyTu-TgJa6xpDFks5uNKkLgaJpZM4VkjNe .

[varisd]

Jo, testy jsem jeste neresil.

Ten gradient runner pouzivam, abych si vyprintil gradienty z celeho datasetu (vetsinou validacniho, protoze neni tak velky). Z toho si pak bokem udelam prumer gradientu pro kazdou vahu modelu; ty pak muzu pouzit v EWC.

[jindrahelcl]

Tak kdyby se ti podařilo tuhle pipelinu nasimulovat v tests/tests_run.sh tak by to bylo úplně nejlepší.

[varisd]

ping, uz jsem tuhle vetev rebasoval minimalne trikrat... pokud ji nemate v planu mergovat, dejte mi vedet

[jlibovicky]

Mno já jsem myslel, že se to zamerguje, až bude hotovej refaktoříček s tf datasetem (stejně jako hromada commitů, který si syslim ve štelářku a nedělám z toho PR).