search

bobye / batchnorm_prune

Tensorflow codes for "Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers"
30 stars 9 forks source link

enable the modularity of core API #1

Open bobye opened 5 years ago

bobye commented 5 years ago

  1. Need to remove dependency on explicitly computing the pruning weights. (Current is hard-coding)

  2. Need to properly handle input.op.name in compressed mode. (Current is hard-coding)

shz1314 commented 5 years ago

hi， i want to follow your work， shall you show me the detail about how changing the code？ Very thanks!!!

bobye commented 5 years ago

@shz1314 what specific changes you want to make? I recommend you try to get the code running first.

shz1314 commented 5 years ago

Hi, my error is because of python3.5. i want to use your method to accelerate my RCNN, but i need Deconvolution layer, i see your paper have similar work， so shall you show the code about 5.3 SEGMENTATION EXPERIMENT in your paper. So thanks. Sun Jose

bobye commented 5 years ago

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字 才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 notifications@github.com wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA .

shz1314 commented 5 years ago

您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字 才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 notifications@github.com wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

shz1314 commented 5 years ago

您的邮箱是多少啊 我想把我的网络结构发给您 基于tensorflow的 你可以看一下这种结构是否可以使用这种方法！有 您的帮助我应该更容易在检测阶段复现出您的方法!

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字 才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 notifications@github.com wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bobye commented 5 years ago

我文章中用的deconv layer并没有用batchnorm，所以也没有做任何channel pruning，pruning只发生在带batchnorm的conv layer中（但不包含每个residual module的最后一个conv）。我分享的是我重新实现的代码，和当时公司做实验的代码不是同一份，公司代码中实现的deconv并不能直接调用。你如果需要实现deconv好像也并不难，但是需要理解框架代码的基本工作原理。

实现任何新的layer都要做四件事情：

  1. 完成layer本身的功能（定义变量，设置operator等）
  2. 构建channel to channel的flow graph，知道前一层layer哪个channel的值会影响后一层哪些channel
  3. （如果layer不包含batchnorm可忽略）记录batchnorm的channel cost，就是加入pruning 这个layer某一个channel会带了多少资源的节省（cost）
  4. 定义如果某个channel被prune后，它所影响的layer的bias会发生什么规则（就是shadow这个变量记录的operator map，具体规则在文章中有提到）

我最近有空可以把deconv这部分代码补上，但是我实现过的是不包含batchnorm的（所以也不支持pruning） 实际上prune deconv layer本身也是比较麻烦的事情，因为deconv layer的经常会连在residual module里（比如fully convolutional network）。我在文章中也说明并没有prune 任何residual module里输出层的channel数。

On Tue, Sep 3, 2019 at 6:54 PM shz1314 notifications@github.com wrote:

您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字 才能做到其它网络上去。

但是segmentation的代码是基于公司代码改的不方便分享。

On Sun, Sep 1, 2019 at 9:07 PM shz1314 notifications@github.com wrote:

Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049>,

or mute the thread < https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y5S36M3CZXYDJ5O2ZTQH4IO7A5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52CS6A#issuecomment-527706488, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56YZUDMZB7UCW6RUNJIDQH4IO7ANCNFSM4F4MWSFA .

shz1314 commented 5 years ago

非常感谢您的回复，迷糊灌顶，我先来搞一搞。 还有两点建议： 1其实这块原理我还是刚入门，但是你提到权重大小，还是输出大小决定怎么进行剪枝，我之前使用过权重归一化（weight normalization）的方法，感觉可以用其 来衡量权重大小从而剪枝？ 2 如果使用densenet的block结构，是不是每一个block里都可以做到剪枝呢。

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月4日(星期三) 中午1:34 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

我文章中用的deconv layer并没有用batchnorm，所以也没有做任何channel pruning，pruning只发生在带batchnorm的conv layer中（但不包含每个residual module的最后一个conv）。我分享的是我重新实现的代码，和当时公司做实验的代码不是同一份，公司代码中实现的deconv并不能直接调用。你如果需要实现deconv好像也并不难，但是需要理解框架代码的基本工作原理。

实现任何新的layer都要做四件事情：

  1. 完成layer本身的功能（定义变量，设置operator等）
  2. 构建channel to channel的flow graph，知道前一层layer哪个channel的值会影响后一层哪些channel
  3. （如果layer不包含batchnorm可忽略）记录batchnorm的channel cost，就是加入pruning 这个layer某一个channel会带了多少资源的节省（cost）

  4. 定义如果某个channel被prune后，它所影响的layer的bias会发生什么规则（就是shadow这个变量记录的operator map，具体规则在文章中有提到）

    我最近有空可以把deconv这部分代码补上，但是我实现过的是不包含batchnorm的（所以也不支持pruning） 实际上prune deconv layer本身也是比较麻烦的事情，因为deconv layer的经常会连在residual module里（比如fully convolutional network）。我在文章中也说明并没有prune 任何residual module里输出层的channel数。

    On Tue, Sep 3, 2019 at 6:54 PM shz1314 notifications@github.com wrote:

    您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

    ------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

    谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字 才能做到其它网络上去。

    但是segmentation的代码是基于公司代码改的不方便分享。

    On Sun, Sep 1, 2019 at 9:07 PM shz1314 notifications@github.com wrote:

    Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

    — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049>,

    or mute the thread < https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA>

    .

    — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

    — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y5S36M3CZXYDJ5O2ZTQH4IO7A5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52CS6A#issuecomment-527706488, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56YZUDMZB7UCW6RUNJIDQH4IO7ANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

shz1314 commented 5 years ago

您好，我观察了下剪枝的结果，在一个残差网络块结构中 输入输出的通道数为（64，64）（64，64）修剪形如（64，32）（32，64），但是我 之前的工作构建残差网络的时候就考虑过这点，从而我的初始通道结构就为（64，32）（64，32），如果接着使用这种方法， 是微调我的结果，还是？ 而且我之前试过手调通道数，将原来的深层通道数为256，全部设置为128，此时性能不减！

------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月4日(星期三) 中午1:34 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

我文章中用的deconv layer并没有用batchnorm，所以也没有做任何channel pruning，pruning只发生在带batchnorm的conv layer中（但不包含每个residual module的最后一个conv）。我分享的是我重新实现的代码，和当时公司做实验的代码不是同一份，公司代码中实现的deconv并不能直接调用。你如果需要实现deconv好像也并不难，但是需要理解框架代码的基本工作原理。

实现任何新的layer都要做四件事情：

  1. 完成layer本身的功能（定义变量，设置operator等）
  2. 构建channel to channel的flow graph，知道前一层layer哪个channel的值会影响后一层哪些channel
  3. （如果layer不包含batchnorm可忽略）记录batchnorm的channel cost，就是加入pruning 这个layer某一个channel会带了多少资源的节省（cost）

  4. 定义如果某个channel被prune后，它所影响的layer的bias会发生什么规则（就是shadow这个变量记录的operator map，具体规则在文章中有提到）

    我最近有空可以把deconv这部分代码补上，但是我实现过的是不包含batchnorm的（所以也不支持pruning） 实际上prune deconv layer本身也是比较麻烦的事情，因为deconv layer的经常会连在residual module里（比如fully convolutional network）。我在文章中也说明并没有prune 任何residual module里输出层的channel数。

    On Tue, Sep 3, 2019 at 6:54 PM shz1314 notifications@github.com wrote:

    您好，非常感谢您的回复，因为您做图像分割的时候肯定需要解码过程，应该也是使用反卷积层吧，能否只公开您重新构建的反卷积层呢，目前我的想法是将我的网络嵌入到您的代码中！

    ------------------ 原始邮件 ------------------ 发件人: "Yijun Li"notifications@github.com; 发送时间: 2019年9月3日(星期二) 晚上10:59 收件人: "bobye/batchnorm_prune"batchnorm_prune@noreply.github.com; 抄送: "Sun Jose"981449149@qq.com;"Mention"mention@noreply.github.com; 主题: Re: [bobye/batchnorm_prune] enable the modularity of core API (#1)

    谢谢你的详细说明，我抽空可以帮你看看code，有一个地方确实需要hard code input tensor的名字 才能做到其它网络上去。

    但是segmentation的代码是基于公司代码改的不方便分享。

    On Sun, Sep 1, 2019 at 9:07 PM shz1314 notifications@github.com wrote:

    Hi, i run first, but when training the first step, i get error, none /batchnome/gamma, i think the error is caused by you say(Need to properly handle input.op.name in compressed mode.)

    — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y37OTV2TZLBXFHBSIDQHRRLZA5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UOZQI#issuecomment-526970049>,

    or mute the thread < https://github.com/notifications/unsubscribe-auth/AAH56Y6QO7APF4H3IDI7DVLQHRRLZANCNFSM4F4MWSFA>

    .

    — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

    — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bobye/batchnorm_prune/issues/1?email_source=notifications&email_token=AAH56Y5S36M3CZXYDJ5O2ZTQH4IO7A5CNFSM4F4MWSFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD52CS6A#issuecomment-527706488, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH56YZUDMZB7UCW6RUNJIDQH4IO7ANCNFSM4F4MWSFA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.