search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.78k stars 6.79k forks source link

Can't use mx.np.concatenate with hybridize #18652

Closed austinmw closed 4 years ago

austinmw commented 4 years ago

Description

I’m trying to adapt the GoogLeNet/InceptionV1 implementation in the online book d2l.ai to be compatible with hybridization. However, I’m currently facing issues with mx.np.concatenate.

Error Message

AssertionError Traceback (most recent call last)

in () 113 d2l.train_ch13(net, train_iter=train_dl, test_iter=valid_dl, 114 loss=loss, trainer=optimizer, --> 115 num_epochs=10, ctx_list=ctx_list) ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/d2l/d2l.py in train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, ctx_list, split_f) 1333 timer.start() 1334 l, acc = train_batch_ch13( -> 1335 net, features, labels, loss, trainer, ctx_list, split_f) 1336 metric.add(l, acc, labels.shape[0], labels.size) 1337 timer.stop() ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/d2l/d2l.py in train_batch_ch13(net, features, labels, loss, trainer, ctx_list, split_f) 1307 X_shards, y_shards = split_f(features, labels, ctx_list) 1308 with autograd.record(): -> 1309 pred_shards = [net(X_shard) for X_shard in X_shards] 1310 ls = [loss(pred_shard, y_shard) for pred_shard, y_shard 1311 in zip(pred_shards, y_shards)] ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/d2l/d2l.py in (.0) 1307 X_shards, y_shards = split_f(features, labels, ctx_list) 1308 with autograd.record(): -> 1309 pred_shards = [net(X_shard) for X_shard in X_shards] 1310 ls = [loss(pred_shard, y_shard) for pred_shard, y_shard 1311 in zip(pred_shards, y_shards)] ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in __call__(self, *args) 756 hook(self, args) 757 --> 758 out = self.forward(*args) 759 760 for hook in self._forward_hooks.values(): ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in forward(self, x, *args) 1211 'Find all contexts = {}'.format(ctx_set)) 1212 with ctx: -> 1213 return self._call_cached_op(x, *args) 1214 with ctx: 1215 try: ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args) 1044 def _call_cached_op(self, *args): 1045 if self._cached_op is None: -> 1046 self._build_cache(*args) 1047 assert self._cached_op, "cached op is not None" 1048 if self._callback: ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _build_cache(self, *args) 996 997 def _build_cache(self, *args): --> 998 data, out = self._get_graph(*args) 999 data_names = {data.name: i for i, data in enumerate(data)} 1000 params = self.collect_params() ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _get_graph(self, *args) 988 params = {i: j.var() for i, j in self._reg_params.items()} 989 with self.name_scope(): --> 990 out = self.hybrid_forward(symbol, *grouped_inputs, **params) # pylint: disable=no-value-for-parameter 991 out, self._out_format = _flatten(out, "output") 992 in hybrid_forward(self, F, x) 97 98 def hybrid_forward(self, F, x): ---> 99 x = self.net(x) 100 return x 101 ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in __call__(self, *args) 756 hook(self, args) 757 --> 758 out = self.forward(*args) 759 760 for hook in self._forward_hooks.values(): ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in forward(self, x, *args) 1225 params = {i: j.var() for i, j in self._reg_params.items()} 1226 with self.name_scope(): -> 1227 return self.hybrid_forward(symbol, x, *args, **params) 1228 1229 def hybrid_forward(self, F, x, *args, **kwargs): ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/nn/basic_layers.py in hybrid_forward(self, F, x) 117 def hybrid_forward(self, F, x): 118 for block in self._children.values(): --> 119 x = block(x) 120 return x 121 ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in __call__(self, *args) 756 hook(self, args) 757 --> 758 out = self.forward(*args) 759 760 for hook in self._forward_hooks.values(): ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in forward(self, x, *args) 1225 params = {i: j.var() for i, j in self._reg_params.items()} 1226 with self.name_scope(): -> 1227 return self.hybrid_forward(symbol, x, *args, **params) 1228 1229 def hybrid_forward(self, F, x, *args, **kwargs): in hybrid_forward(self, F, x) 37 p4 = self.p4_2(self.p4_1(x)) 38 # Concatenate the outputs on the channel dimension ---> 39 return np.concatenate((p1, p2, p3, p4), axis=1) 40 #return F.concat(p1, p2, p3, p4, dim=1) 41 #return F.concat(p1, p2, p3, p4, dim=1) ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/numpy/multiarray.py in concatenate(seq, axis, out) 4884 array([1., 2., 3., 4., 5., 6.]) 4885 """ -> 4886 return _mx_nd_np.concatenate(seq, axis=axis, out=out) 4887 4888 ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/numpy/_op.py in concatenate(seq, axis, out) 3009 [3., 4., 6.]]) 3010 """ -> 3011 return _npi.concatenate(*seq, axis=axis, out=out) 3012 3013 ~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/register.py in concatenate(*data, **kwargs) AssertionError: Positional arguments must have NDArray type, but got <_Symbol conv3_relu_fwd>

To Reproduce

Steps to reproduce

import d2l # d2l.ai book code

import mxnet as mx
from mxnet import gluon, metric, np, npx
from mxnet.gluon import nn
npx.set_np()

from mxnet import nd

ctx_list = [npx.gpu(i) for i in range(npx.num_gpus())]
mx.random.seed(42, ctx='all')

class Inception(nn.HybridBlock):
    # c1- c4 are the number of output channels for each layer in the path
    def __init__(self, c1, c2, c3, c4, **kwargs):
        super().__init__(**kwargs)
        # Path 1 is a single 1 x 1 convolutional layer
        self.p1_1 = nn.Conv2D(c1, kernel_size=1, activation='relu')
        # Path 2 is a 1 x 1 convolutional layer followed by a 3 x 3
        # convolutional layer
        self.p2_1 = nn.Conv2D(c2[0], kernel_size=1, activation='relu')
        self.p2_2 = nn.Conv2D(c2[1], kernel_size=3, padding=1, activation='relu')
        # Path 3 is a 1 x 1 convolutional layer followed by a 5 x 5
        # convolutional layer
        self.p3_1 = nn.Conv2D(c3[0], kernel_size=1, activation='relu')
        self.p3_2 = nn.Conv2D(c3[1], kernel_size=5, padding=2,
                              activation='relu')
        # Path 4 
        self.p4_1 = nn.MaxPool2D(pool_size=3, strides=1, padding=1)
        self.p4_2 = nn.Conv2D(c4, kernel_size=1, activation='relu')

    def hybrid_forward(self, F, x):
        p1 = self.p1_1(x)
        p2 = self.p2_2(self.p2_1(x))
        p3 = self.p3_2(self.p3_1(x))
        p4 = self.p4_2(self.p4_1(x))
        # Concatenate the outputs on the channel dimension
        return np.concatenate((p1, p2, p3, p4), axis=1)
        #return F.concat(p1, p2, p3, p4, dim=1) # also doesn't work

class GoogLeNet(nn.HybridBlock):
    """
    GoogLeNet uses a stack of a total of 9 inception blocks and global average pooling
    """
    def __init__(self, classes=1000, **kwargs):
        super().__init__(**kwargs)

        self.net = nn.HybridSequential()

        # First component uses a 64-channel 7 x 7 convolutional layer
        self.net.add(
            nn.Conv2D(64, kernel_size=7, strides=2, padding=3, activation='relu'),
            nn.MaxPool2D(pool_size=3, strides=2, padding=1)
        )

        # Second component uses two convolutional layers:
        # first a 64-channel 1 x 1 convolutional layer,
        # then a 3 x 3 convolutional layer that triples the number of channels.
        # This corresponds to the second path in the Inception block.
        self.net.add(
            nn.Conv2D(64, kernel_size=1, activation='relu'),
            nn.Conv2D(192, kernel_size=3, padding=1, activation='relu'),
            nn.MaxPool2D(pool_size=3, strides=2, padding=1)
        )

        # Third component connects to complete Inception blocks in series
        # The number of output channels of the first block is 64+128+32+32=256
        # and the ratio to the output channels of the four paths is 2:4:1:1.
        # The number of output channels of the second block is 128+192+96+64=480
        # and the ratio to the output channels per path is 4:6:3:2
        self.net.add(
            Inception(64, (96, 128), (16, 32), 32),
            Inception(128, (128, 192), (32, 96), 64)
        )

        # Fourth component connects five Inception blocks in series
        self.net.add(
            Inception(196, (96, 208), (16, 48), 64),
            Inception(160, (112, 224), (24, 64), 64),
            Inception(128, (128, 256), (24, 64), 64),
            Inception(112, (144, 288), (32, 64), 64),
            Inception(256, (160, 320), (32, 128), 128),
            nn.MaxPool2D(pool_size=3, strides=2, padding=1)
        )

        # Fifth component has two Inception blocks followed by output layer
        self.net.add(
            Inception(256, (160, 320), (32, 128), 128),
            Inception(384, (192, 384), (48, 128), 128),
            nn.Dense(classes)
        )

    def hybrid_forward(self, F, x):
        x = self.net(x)
        return x

net = GoogLeNet(classes=10)
net.initialize()
net.hybridize()

train_dl, valid_dl = d2l.load_data_fashion_mnist(batch_size=128, resize=96)

loss = gluon.loss.SoftmaxCrossEntropyLoss()
optimizer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1})

d2l.train_ch13(net, train_iter=train_dl, test_iter=valid_dl, 
               loss=loss, trainer=optimizer, 
               num_epochs=10, ctx_list=ctx_list)

What have you tried to solve it?

Also tried return F.concat(p1, p2, p3, p4, dim=1) which results in a different error:

TypeError: Operator concat registered in backend is known as concat in Python. This is a legacy operator which can only accept legacy ndarrays, while received an MXNet numpy ndarray. Please call as_nd_ndarray() upon the numpy ndarray to convert it to a legacy ndarray, and then feed the converted array to this operator.

Environment

We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:

----------Python Info----------
Version      : 3.6.5
Compiler     : GCC 7.2.0
Build        : ('default', 'Apr 29 2018 16:14:56')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 10.0.1
Directory    : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version      : 1.6.0
Directory    : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet
Num GPUs     : 1
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform     : Linux-4.15.0-1060-aws-x86_64-with-debian-buster-sid
system       : Linux
node         : ip-172-31-74-28
release      : 4.15.0-1060-aws
version      : #62-Ubuntu SMP Tue Feb 11 21:23:22 UTC 2020
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
Stepping:            7
CPU MHz:             3191.599
BogoMIPS:            4999.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0020 sec, LOAD: 0.6654 sec.
Timing for GluonNLP GitHub: https://github.com/dmlc/gluon-nlp, DNS: 0.0005 sec, LOAD: 0.4951 sec.
Timing for GluonNLP: http://gluon-nlp.mxnet.io, DNS: 0.0408 sec, LOAD: 0.1209 sec.
Timing for D2L: http://d2l.ai, DNS: 0.0307 sec, LOAD: 0.0119 sec.
Timing for D2L (zh-cn): http://zh.d2l.ai, DNS: 0.0150 sec, LOAD: 0.1406 sec.
Timing for FashionMNIST: https://repo.mxnet.io/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0542 sec, LOAD: 0.3796 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0017 sec, LOAD: 0.0988 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.0032265186309814453 sec.
leezu commented 4 years ago

Please use F.np.concatenate

austinmw commented 4 years ago

@leezu Thanks, that worked! Is F.np.<op> the new way to do things going forward? Can't find any documentation for this syntax.

leezu commented 4 years ago

@austinmw it's the correct syntax for the experimental numpy support in MXNet 1.x. Sorry for the unclear documentation. With the upcoming MXNet 2 release, users can directly use np.<op> inside HybridBlock and don't need F anymore. The documentation will be updated accordingly in time for the first "official" MXNet 2 Alpha or Beta release.

austinmw commented 4 years ago

Awesome, thanks!