search

aws-neuron / aws-neuron-sdk

Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services
https://aws.amazon.com/machine-learning/neuron/
Other
440 stars 145 forks source link

StyleGAN2 Pytorch Tracing Issue #371

Closed PigletOS closed 2 years ago

PigletOS commented 2 years ago

Environment torch 1.9.1 torch-neuron 1.9.1.2.0.392.0

Problem I tried to use torch_neuron to trace the StyleGAN2 model. But it failed with the error message "Error handling operator: Reshape_8 ". I also tried to trace the specific layers and found the error is related to the "ModulatedConv2d" layer. When it traced the "ModulatedConv2d" layer, it went into pdb mode.

Python Code and Logs torch_neuron_test.zip

Logs when tracing the StyleGAN2 model

model.py:488: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  if len(styles) < 2:
model.py:273: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out = F.conv2d(input, weight, padding=self.padding, groups=batch)
model.py:258: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out = F.conv_transpose2d(input, weight, padding=0, stride=2, groups=batch)
/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
INFO:Neuron:There are 36 ops of 3 different types in the TorchScript that are not compiled by neuron-cc: aten::normal_, aten::new_empty, aten::repeat, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 2578, fused = 2473, percent fused = 95.93%
INFO:Neuron:Compiling function _NeuronGraph$414 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpdj44676g/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpdj44676g/graph_def.neff --io-config {"inputs": {"0:0": [[1, 18, 512], "float32"]}, "outputs": ["ExpandDims:0"]} --verbose 35'
.Analyzing dependencies of sg00/Block1
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
12/02/2021 11:50:18 AM INFO [Stargazer]: Generating Arch 'Inferentia-1.0'
12/02/2021 11:50:18 AM INFO [Stargazer]: INFO: Pre SG DRAM bytes loaded or saved 106496
12/02/2021 11:50:18 AM INFO [Stargazer]: INFO: Pre SG average DMA size 649 bytes
12/02/2021 11:50:18 AM INFO [Stargazer]: Num Loads in Func = 2
12/02/2021 11:50:18 AM INFO [Stargazer]: Num Saves in Func = 1
12/02/2021 11:50:18 AM INFO [Stargazer]: Num Input Loads in Func= 2
12/02/2021 11:50:18 AM INFO [Stargazer]: Num Output Saves in Func= 1
12/02/2021 11:50:18 AM INFO [Stargazer]: Num Spill Loads in Func= 0
12/02/2021 11:50:18 AM INFO [Stargazer]: Num Spill Saves in Func= 0
12/02/2021 11:50:18 AM INFO [Stargazer]: Wavegraph code generation for Inferentia:
12/02/2021 11:50:18 AM INFO [Stargazer]:     Engine              File
12/02/2021 11:50:18 AM INFO [Stargazer]:     ------              ----
12/02/2021 11:50:18 AM INFO [Stargazer]:     PE-Array            pe.bin
12/02/2021 11:50:18 AM INFO [Stargazer]:     Pool-Eng            pool.bin
12/02/2021 11:50:18 AM INFO [Stargazer]:     Act-Eng             act.bin
12/02/2021 11:50:18 AM INFO [Stargazer]: 
12/02/2021 11:50:18 AM INFO [Stargazer]: Fixing data race is 0
12/02/2021 11:50:18 AM INFO [Stargazer]: Data race checker engines
12/02/2021 11:50:18 AM INFO [Stargazer]: [Sailfish] Data race analysis initially
12/02/2021 11:50:18 AM INFO [Stargazer]: [Sailfish] Data race analysis found no races, run time: 0:00:00
12/02/2021 11:50:18 AM INFO [Stargazer]: [Sailfish] Remove redundant edges
12/02/2021 11:50:18 AM INFO [Stargazer]: Data race checker engines
12/02/2021 11:50:18 AM INFO [Stargazer]: Transitive reduction start 
12/02/2021 11:50:18 AM INFO [Stargazer]: Transitive reduction removed 0 redundant edges, time: 0:00:00
12/02/2021 11:50:18 AM INFO [Stargazer]: Sync Critical Load Chains Start
12/02/2021 11:50:18 AM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Start...
12/02/2021 11:50:18 AM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Done.
12/02/2021 11:50:18 AM DEBUG [Stargazer]: Load Graph NumRoots; 2
12/02/2021 11:50:18 AM INFO [Stargazer]: Sync Critical Load Chains added 0 new Load-2-Load syncs
12/02/2021 11:50:18 AM INFO [Stargazer]: Sync Critical Load Chains Done.0:00:00
12/02/2021 11:50:18 AM INFO [Stargazer]: Out wavegraph bin file is wavegraph-bin.json
12/02/2021 11:50:18 AM INFO [Stargazer]: Writing NN JSON to file 'wavegraph-bin.json'
12/02/2021 11:50:18 AM INFO [Stargazer]: Virtual memory peak = 8183936 K bytes
12/02/2021 11:50:18 AM INFO [Stargazer]: PASSED - Total time: 0:00:00
/data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  sys.exit(main())

Compiler status PASS
WARNING:tensorflow:From /data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/ops/aten.py:1751: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
INFO:Neuron:Compiling function _NeuronGraph$415 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpf0axpacu/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpf0axpacu/graph_def.neff --io-config {"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"]}, "outputs": ["Reshape_5:0", "Const_12:0", "Const_13:0", "Const_14:0"]} --verbose 35'
.[11:50:22] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 512, 9216]  [1, 6, 6, 9216]
...........................................................................................................................................................................................................................................................................................................................................................................................INFO:Neuron:Compile command returned: -9
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$415; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpf0axpacu/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpf0axpacu/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"]}, "outputs": ["Reshape_5:0", "Const_12:0", "Const_13:0", "Const_14:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpf0axpacu/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpf0axpacu/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"]}, "outputs": ["Reshape_5:0", "Const_12:0", "Const_13:0", "Const_14:0"]}' --verbose 35
INFO:Neuron:Compiling function _NeuronGraph$416 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1m7_ng6t/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1m7_ng6t/graph_def.neff --io-config {"inputs": {"0:0": [[18, 512, 4, 4], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
..12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: ***************************************************************
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: ***************************************************************
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: 
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: 
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Error class:    RuntimeError
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Error location: Unknown
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1m7_ng6t/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1m7_ng6t/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 4, 4], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: 
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Internal details:
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: 
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Version information:
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 01:57:
17 PM ERROR 59590 [neuron-cc]:   NEFF version Dynamic
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   MXNet not available
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]:   TF not available
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: 
12/02/2021 01:57:17 PM ERROR 59590 [neuron-cc]: Artifacts stored in: /tmp/tmp1m7_ng6t

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$416; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1m7_ng6t/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1m7_ng6t/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 4, 4], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1m7_ng6t/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1m7_ng6t/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 4, 4], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
INFO:Neuron:Compiling function _NeuronGraph$417 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpfi4vg3kr/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpfi4vg3kr/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 8, 8], "float32"], "1:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
.[13:57:21] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 512, 9216]  [1, 10, 10, 9216]
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................INFO:Neuron:Compile command returned: -9
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$417; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpfi4vg3kr/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpfi4vg3kr/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 8, 8], "float32"], "1:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpfi4vg3kr/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpfi4vg3kr/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 8, 8], "float32"], "1:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
INFO:Neuron:Compiling function _NeuronGraph$418 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8z4khcn0/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8z4khcn0/graph_def.neff --io-config {"inputs": {"0:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
..12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: ***************************************************************
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: ***************************************************************
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: 
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: 
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Error class:    RuntimeError
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Error location: Unknown
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8z4khcn0/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8z4khcn0/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: 
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Internal details:
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: 
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Version information:
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   NEFF version Dynamic
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   MXNet not available
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]:   TF not available
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: 
12/02/2021 04:03:32 PM ERROR 54490 [neuron-cc]: Artifacts stored in: /tmp/tmp8z4khcn0

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$418; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8z4khcn0/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8z4khcn0/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8z4khcn0/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8z4khcn0/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 8, 8], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
INFO:Neuron:Compiling function _NeuronGraph$419 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp0g1j1kf9/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp0g1j1kf9/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 16, 16], "float32"], "1:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
.[16:03:35] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 512, 9216]  [1, 18, 18, 9216]
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................INFO:Neuron:Compile command returned: -9
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$419; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp0g1j1kf9/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp0g1j1kf9/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 16, 16], "float32"], "1:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp0g1j1kf9/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp0g1j1kf9/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 16, 16], "float32"], "1:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
.INFO:Neuron:Compiling function _NeuronGraph$420 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1qtosiwx/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1qtosiwx/graph_def.neff --io-config {"inputs": {"0:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
...12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: ***************************************************************
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: ***************************************************************
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: 
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: 
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Error class:    RuntimeError
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Error location: Unknown
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1qtosiwx/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1qtosiwx/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: 
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Internal details:
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: 
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Version information:
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   NEFF version Dynamic
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   MXNet not available
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]:   TF not available
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: 
12/02/2021 06:07:03 PM ERROR 66869 [neuron-cc]: Artifacts stored in: /tmp/tmp1qtosiwx

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$420; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1qtosiwx/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1qtosiwx/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1qtosiwx/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1qtosiwx/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 16, 16], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
INFO:Neuron:Compiling function _NeuronGraph$421 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpnq6gvtej/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpnq6gvtej/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 32, 32], "float32"], "1:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
.[18:07:08] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 512, 9216]  [1, 34, 34, 9216]
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................INFO:Neuron:Compile command returned: -9
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$421; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpnq6gvtej/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpnq6gvtej/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 32, 32], "float32"], "1:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpnq6gvtej/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpnq6gvtej/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 32, 32], "float32"], "1:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
.INFO:Neuron:Compiling function _NeuronGraph$422 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmporrccnw1/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmporrccnw1/graph_def.neff --io-config {"inputs": {"0:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
...12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: ***************************************************************
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: ***************************************************************
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: 
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: 
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Error class:    RuntimeError
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Error location: Unknown
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmporrccnw1/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmporrccnw1/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: 
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Internal details:
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: 
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Version information:
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   NEFF version Dynamic
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   MXNet not available
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]:   TF not available
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: 
12/02/2021 07:45:29 PM ERROR 49139 [neuron-cc]: Artifacts stored in: /tmp/tmporrccnw1

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$422; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmporrccnw1/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmporrccnw1/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmporrccnw1/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmporrccnw1/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 32, 32], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
.INFO:Neuron:Compiling function _NeuronGraph$423 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpjnfgb4az/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpjnfgb4az/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 64, 64], "float32"], "1:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
.[19:45:36] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 512, 9216]  [1, 66, 66, 9216]
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................INFO:Neuron:Compile command returned: -9
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$423; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpjnfgb4az/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpjnfgb4az/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 64, 64], "float32"], "1:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpjnfgb4az/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpjnfgb4az/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 64, 64], "float32"], "1:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
...INFO:Neuron:Compiling function _NeuronGraph$424 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpja0xxsfd/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpja0xxsfd/graph_def.neff --io-config {"inputs": {"0:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
...12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: ***************************************************************
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: ***************************************************************
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: 
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: 
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Error class:    RuntimeError
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Error location: Unknown
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpja0xxsfd/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpja0xxsfd/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: 
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Internal details:
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: 
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Version information:
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   NEFF version Dynamic
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   MXNet not available
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]:   TF not available
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: 
12/02/2021 09:27:37 PM ERROR 47066 [neuron-cc]: Artifacts stored in: /tmp/tmpja0xxsfd

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$424; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpja0xxsfd/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpja0xxsfd/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpja0xxsfd/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpja0xxsfd/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 512, 64, 64], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
..INFO:Neuron:Compiling function _NeuronGraph$425 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp3gds3kqm/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp3gds3kqm/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 128, 128], "float32"], "1:0": [[18, 256, 128, 128], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
.[21:27:46] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 256, 4608]  [1, 130, 130, 4608]
......................................................................................................................................................12/02/2021 09:36:05 PM CRITICAL [WalrusDriver]: too many reloads required
Warning: too many spills required, turning optimizations off
................................................................................................................Warning: no anti dependence analysis due to too many spills
.....................Warning: too many spills, skip scheduling
...............12/02/2021 09:44:21 PM INFO [Stargazer]: Generating Arch 'Inferentia-1.0'
12/02/2021 09:44:21 PM INFO [Stargazer]: INFO: Pre SG DRAM bytes loaded or saved 45569179280
12/02/2021 09:44:21 PM INFO [Stargazer]: INFO: Pre SG average DMA size 1708 bytes
.........12/02/2021 09:44:21 PM INFO [Stargazer]: Num Loads in Func = 270712
12/02/2021 09:44:21 PM INFO [Stargazer]: Num Saves in Func = 26403
12/02/2021 09:44:21 PM INFO [Stargazer]: Num Input Loads in Func= 3738
12/02/2021 09:44:21 PM INFO [Stargazer]: Num Output Saves in Func= 1440
12/02/2021 09:44:21 PM INFO [Stargazer]: Num Spill Loads in Func= 266974
12/02/2021 09:44:21 PM INFO [Stargazer]: Num Spill Saves in Func= 24963
12/02/2021 09:44:44 PM INFO [Stargazer]: Wavegraph code generation for Inferentia:
12/02/2021 09:44:44 PM INFO [Stargazer]:     Engine              File
12/02/2021 09:44:44 PM INFO [Stargazer]:     ------              ----
12/02/2021 09:44:44 PM INFO [Stargazer]:     PE-Array            pe.bin
12/02/2021 09:44:44 PM INFO [Stargazer]:     Pool-Eng            pool.bin
12/02/2021 09:44:44 PM INFO [Stargazer]:     Act-Eng             act.bin
12/02/2021 09:44:44 PM INFO [Stargazer]: 
12/02/2021 09:44:53 PM INFO [Stargazer]: Trivial Allocator: 5 events used
12/02/2021 09:44:53 PM INFO [Stargazer]: Trivial Allocator: 19 queues used
..............................12/02/2021 09:45:30 PM INFO [Stargazer]: Out wavegraph bin file is wavegraph-bin.json
12/02/2021 09:45:30 PM INFO [Stargazer]: Writing NN JSON to file 'wavegraph-bin.json'
12/02/2021 09:46:28 PM INFO [Stargazer]: Virtual memory peak = 32528196 K bytes
12/02/2021 09:46:28 PM INFO [Stargazer]: PASSED - Total time: 0:02:07
........................................................................................................................................................................................................................................................................................./data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  sys.exit(main())
/data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  sys.exit(main())

Compiler status PASS
..........INFO:Neuron:Compiling function _NeuronGraph$426 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpm51asnw8/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpm51asnw8/graph_def.neff --io-config {"inputs": {"0:0": [[18, 256, 128, 128], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
..12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: ***************************************************************
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: ***************************************************************
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: 
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: 
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Error class:    RuntimeError
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Error location: Unknown
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpm51asnw8/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpm51asnw8/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 256, 128, 128], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: 
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Internal details:
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: 
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Version information:
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   NEFF version Dynamic
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   MXNet not available
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]:   TF not available
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: 
12/02/2021 10:02:50 PM ERROR 81599 [neuron-cc]: Artifacts stored in: /tmp/tmpm51asnw8

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$426; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpm51asnw8/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpm51asnw8/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 256, 128, 128], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpm51asnw8/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpm51asnw8/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 256, 128, 128], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
..INFO:Neuron:Compiling function _NeuronGraph$427 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpr33bdlet/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpr33bdlet/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 256, 256], "float32"], "1:0": [[18, 128, 256, 256], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
..[22:03:02] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 128, 2304]  [1, 258, 258, 2304]
........................................................................................................................................................................Analyzing dependencies of sg00/Block1
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
.***************************************************
.......................................................................................12/02/2021 10:17:18 PM INFO [Stargazer]: Generating Arch 'Inferentia-1.0'
12/02/2021 10:17:18 PM INFO [Stargazer]: INFO: Pre SG DRAM bytes loaded or saved 3128329360
12/02/2021 10:17:18 PM INFO [Stargazer]: INFO: Pre SG average DMA size 1365 bytes
....12/02/2021 10:17:18 PM INFO [Stargazer]: Num Loads in Func = 24033
12/02/2021 10:17:18 PM INFO [Stargazer]: Num Saves in Func = 12638
12/02/2021 10:17:18 PM INFO [Stargazer]: Num Input Loads in Func= 1542
12/02/2021 10:17:18 PM INFO [Stargazer]: Num Output Saves in Func= 3456
12/02/2021 10:17:18 PM INFO [Stargazer]: Num Spill Loads in Func= 22491
12/02/2021 10:17:18 PM INFO [Stargazer]: Num Spill Saves in Func= 9182
12/02/2021 10:17:29 PM INFO [Stargazer]: Wavegraph code generation for Inferentia:
12/02/2021 10:17:29 PM INFO [Stargazer]:     Engine              File
12/02/2021 10:17:29 PM INFO [Stargazer]:     ------              ----
12/02/2021 10:17:29 PM INFO [Stargazer]:     PE-Array            pe.bin
12/02/2021 10:17:29 PM INFO [Stargazer]:     Pool-Eng            pool.bin
12/02/2021 10:17:29 PM INFO [Stargazer]:     Act-Eng             act.bin
12/02/2021 10:17:29 PM INFO [Stargazer]: 
12/02/2021 10:17:30 PM INFO [Stargazer]: Fixing data race is 0
12/02/2021 10:17:31 PM INFO [Stargazer]: Data race checker engines
12/02/2021 10:17:32 PM INFO [Stargazer]: [Sailfish] Data race analysis initially
..12/02/2021 10:17:38 PM INFO [Stargazer]: [Sailfish] Data race analysis found no races, run time: 0:00:05
12/02/2021 10:17:38 PM INFO [Stargazer]: [Sailfish] Remove redundant edges
12/02/2021 10:17:39 PM INFO [Stargazer]: Data race checker engines
12/02/2021 10:17:40 PM INFO [Stargazer]: Transitive reduction start 
12/02/2021 10:17:40 PM INFO [Stargazer]: Transitive reduction removed 2453 redundant edges, time: 0:00:00
.12/02/2021 10:17:41 PM INFO [Stargazer]: Sync Critical Load Chains Start
12/02/2021 10:17:41 PM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Start...
12/02/2021 10:17:41 PM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Done.
12/02/2021 10:17:41 PM DEBUG [Stargazer]: Load Graph NumRoots; 82
.12/02/2021 10:17:42 PM INFO [Stargazer]: Sync Critical Load Chains added 2260 new Load-2-Load syncs
12/02/2021 10:17:42 PM INFO [Stargazer]: Sync Critical Load Chains Done.0:00:00
..........12/02/2021 10:17:55 PM INFO [Stargazer]: Out wavegraph bin file is wavegraph-bin.json
12/02/2021 10:17:55 PM INFO [Stargazer]: Writing NN JSON to file 'wavegraph-bin.json'
12/02/2021 10:18:20 PM INFO [Stargazer]: Virtual memory peak = 21155908 K bytes
12/02/2021 10:18:20 PM INFO [Stargazer]: PASSED - Total time: 0:01:01
....................................................................................................../data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  sys.exit(main())
/data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  sys.exit(main())

Compiler status PASS
.............INFO:Neuron:Compiling function _NeuronGraph$428 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpmts4nst2/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpmts4nst2/graph_def.neff --io-config {"inputs": {"0:0": [[18, 128, 256, 256], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
..12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: ***************************************************************
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: ***************************************************************
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: 
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: 
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Error class:    RuntimeError
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Error location: Unknown
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpmts4nst2/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpmts4nst2/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 128, 256, 256], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: 
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Internal details:
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: 
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Version information:
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   NEFF version Dynamic
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   MXNet not available
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]:   TF not available
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: 
12/02/2021 10:24:57 PM ERROR 7996 [neuron-cc]: Artifacts stored in: /tmp/tmpmts4nst2

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$428; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpmts4nst2/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpmts4nst2/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 128, 256, 256], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpmts4nst2/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpmts4nst2/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 128, 256, 256], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
....INFO:Neuron:Compiling function _NeuronGraph$429 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpjxc4ks6p/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpjxc4ks6p/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 512, 512], "float32"], "1:0": [[18, 64, 512, 512], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
..[22:25:17] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 64, 1152]  [1, 514, 514, 1152]
................................................................................................................................................................................................................Analyzing dependencies of sg00/Block1
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
..**************************.*************************
............................................12/02/2021 10:39:29 PM INFO [Stargazer]: Generating Arch 'Inferentia-1.0'
12/02/2021 10:39:29 PM INFO [Stargazer]: INFO: Pre SG DRAM bytes loaded or saved 3422373248
12/02/2021 10:39:29 PM INFO [Stargazer]: INFO: Pre SG average DMA size 2157 bytes
....12/02/2021 10:39:29 PM INFO [Stargazer]: Num Loads in Func = 12781
12/02/2021 10:39:29 PM INFO [Stargazer]: Num Saves in Func = 17333
12/02/2021 10:39:29 PM INFO [Stargazer]: Num Input Loads in Func= 4646
12/02/2021 10:39:29 PM INFO [Stargazer]: Num Output Saves in Func= 9216
12/02/2021 10:39:29 PM INFO [Stargazer]: Num Spill Loads in Func= 8135
12/02/2021 10:39:29 PM INFO [Stargazer]: Num Spill Saves in Func= 8117
12/02/2021 10:39:37 PM INFO [Stargazer]: Wavegraph code generation for Inferentia:
12/02/2021 10:39:37 PM INFO [Stargazer]:     Engine              File
12/02/2021 10:39:37 PM INFO [Stargazer]:     ------              ----
12/02/2021 10:39:37 PM INFO [Stargazer]:     PE-Array            pe.bin
12/02/2021 10:39:37 PM INFO [Stargazer]:     Pool-Eng            pool.bin
12/02/2021 10:39:37 PM INFO [Stargazer]:     Act-Eng             act.bin
12/02/2021 10:39:37 PM INFO [Stargazer]: 
12/02/2021 10:39:39 PM INFO [Stargazer]: Fixing data race is 0
12/02/2021 10:39:39 PM INFO [Stargazer]: Data race checker engines
12/02/2021 10:39:40 PM INFO [Stargazer]: [Sailfish] Data race analysis initially
..12/02/2021 10:39:45 PM INFO [Stargazer]: [Sailfish] Data race analysis found no races, run time: 0:00:04
12/02/2021 10:39:45 PM INFO [Stargazer]: [Sailfish] Remove redundant edges
12/02/2021 10:39:45 PM INFO [Stargazer]: Data race checker engines
12/02/2021 10:39:46 PM INFO [Stargazer]: Transitive reduction start 
12/02/2021 10:39:46 PM INFO [Stargazer]: Transitive reduction removed 12040 redundant edges, time: 0:00:00
12/02/2021 10:39:47 PM INFO [Stargazer]: Sync Critical Load Chains Start
12/02/2021 10:39:47 PM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Start...
12/02/2021 10:39:47 PM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Done.
12/02/2021 10:39:47 PM DEBUG [Stargazer]: Load Graph NumRoots; 24
12/02/2021 10:39:48 PM INFO [Stargazer]: Sync Critical Load Chains added 1828 new Load-2-Load syncs
12/02/2021 10:39:48 PM INFO [Stargazer]: Sync Critical Load Chains Done.0:00:00
...........12/02/2021 10:40:03 PM INFO [Stargazer]: Out wavegraph bin file is wavegraph-bin.json
12/02/2021 10:40:03 PM INFO [Stargazer]: Writing NN JSON to file 'wavegraph-bin.json'
12/02/2021 10:40:23 PM INFO [Stargazer]: Virtual memory peak = 22658916 K bytes
12/02/2021 10:40:23 PM INFO [Stargazer]: PASSED - Total time: 0:00:54
....................................................................................../data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  sys.exit(main())
/data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  sys.exit(main())

Compiler status PASS
......................INFO:Neuron:Compiling function _NeuronGraph$430 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpcaucy839/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpcaucy839/graph_def.neff --io-config {"inputs": {"0:0": [[18, 64, 512, 512], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]} --verbose 35'
...12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: ***************************************************************
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:  An Internal Compiler Error has occurred
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: ***************************************************************
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: 
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Error message:  Error handling operator: Reshape_8 (type: reshape)
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: 
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Error class:    RuntimeError
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Error location: Unknown
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpcaucy839/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpcaucy839/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 64, 512, 512], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: 
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Internal details:
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: 
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Version information:
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   HWM version 1.7.0.0-0
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   NEFF version Dynamic
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   TVM version 1.7.2.0+0
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   NumPy version 1.20.0
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   MXNet not available
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]:   TF not available
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: 
12/02/2021 10:46:48 PM ERROR 29943 [neuron-cc]: Artifacts stored in: /tmp/tmpcaucy839

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$430; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpcaucy839/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpcaucy839/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 64, 512, 512], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmpcaucy839/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpcaucy839/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 64, 512, 512], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["mul:0", "Reshape_17:0", "Const_108:0", "Const_109:0", "Const_110:0"]}' --verbose 35
........INFO:Neuron:Compiling function _NeuronGraph$431 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1czvd4is/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1czvd4is/graph_def.neff --io-config {"inputs": {"0:0": [[18, 1, 1024, 1024], "float32"], "1:0": [[18, 32, 1024, 1024], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]} --verbose 35'
..[22:47:24] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[3, 3, 32, 576]  [1, 1026, 1026, 576]
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Analyzing dependencies of sg00/Block1
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
.***************************************************
....................................................................................12/03/2021 03:13:11 AM INFO [Stargazer]: Generating Arch 'Inferentia-1.0'
12/03/2021 03:13:11 AM INFO [Stargazer]: INFO: Pre SG DRAM bytes loaded or saved 7244318144
12/03/2021 03:13:11 AM INFO [Stargazer]: INFO: Pre SG average DMA size 3072 bytes
............12/03/2021 03:13:11 AM INFO [Stargazer]: Num Loads in Func = 36859
12/03/2021 03:13:11 AM INFO [Stargazer]: Num Saves in Func = 36838
12/03/2021 03:13:11 AM INFO [Stargazer]: Num Input Loads in Func= 18446
12/03/2021 03:13:11 AM INFO [Stargazer]: Num Output Saves in Func= 18432
12/03/2021 03:13:11 AM INFO [Stargazer]: Num Spill Loads in Func= 18413
12/03/2021 03:13:11 AM INFO [Stargazer]: Num Spill Saves in Func= 18406
12/03/2021 03:13:39 AM INFO [Stargazer]: Wavegraph code generation for Inferentia:
12/03/2021 03:13:39 AM INFO [Stargazer]:     Engine              File
12/03/2021 03:13:39 AM INFO [Stargazer]:     ------              ----
12/03/2021 03:13:39 AM INFO [Stargazer]:     PE-Array            pe.bin
12/03/2021 03:13:39 AM INFO [Stargazer]:     Pool-Eng            pool.bin
12/03/2021 03:13:39 AM INFO [Stargazer]:     Act-Eng             act.bin
12/03/2021 03:13:39 AM INFO [Stargazer]: 
12/03/2021 03:13:46 AM INFO [Stargazer]: Fixing data race is 0
12/03/2021 03:13:47 AM INFO [Stargazer]: Data race checker engines
12/03/2021 03:13:49 AM INFO [Stargazer]: [Sailfish] Data race analysis initially
..12/03/2021 03:13:58 AM INFO [Stargazer]: [Sailfish] Data race analysis found no races, run time: 0:00:09
...12/03/2021 03:13:59 AM INFO [Stargazer]: [Sailfish] Remove redundant edges
12/03/2021 03:14:00 AM INFO [Stargazer]: Data race checker engines
12/03/2021 03:14:03 AM INFO [Stargazer]: Transitive reduction start 
12/03/2021 03:14:04 AM INFO [Stargazer]: Transitive reduction removed 29332 redundant edges, time: 0:00:01
.12/03/2021 03:14:07 AM INFO [Stargazer]: Sync Critical Load Chains Start
12/03/2021 03:14:07 AM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Start...
12/03/2021 03:14:07 AM DEBUG [Stargazer]: SyncCritLoads buildLoadGraph Done.
12/03/2021 03:14:07 AM DEBUG [Stargazer]: Load Graph NumRoots; 5
.12/03/2021 03:14:13 AM INFO [Stargazer]: Sync Critical Load Chains added 4255 new Load-2-Load syncs
12/03/2021 03:14:13 AM INFO [Stargazer]: Sync Critical Load Chains Done.0:00:05
....12/03/2021 03:14:25 AM ERROR [Stargazer]: File waveop_fields.hpp:19, Assertion 'x >= static_cast<T>(0)' failed: Waveop copy6:0_TO_copy6:0_local_234_i0_Load_I-316-0(SBAtomLoad): check checkPositive failed for field ref_file_sz

.INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$431; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1czvd4is/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1czvd4is/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 1024, 1024], "float32"], "1:0": [[18, 32, 1024, 1024], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp1czvd4is/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp1czvd4is/graph_def.neff --io-config '{"inputs": {"0:0": [[18, 1, 1024, 1024], "float32"], "1:0": [[18, 32, 1024, 1024], "float32"], "tensor.3:0": [[18, 18, 512], "float32"]}, "outputs": ["Reshape_6:0", "Const_14:0", "Const_15:0", "Const_16:0"]}' --verbose 35
.....INFO:Neuron:Compiling function _NeuronGraph$432 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8paifo8b/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8paifo8b/graph_def.neff --io-config {"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"], "2:0": [[18, 512, 8, 8], "float32"], "3:0": [[18, 512, 16, 16], "float32"], "4:0": [[18, 512, 32, 32], "float32"], "5:0": [[18, 512, 64, 64], "float32"], "6:0": [[18, 256, 128, 128], "float32"], "7:0": [[18, 128, 256, 256], "float32"], "8:0": [[18, 64, 512, 512], "float32"], "9:0": [[18, 1, 1024, 1024], "float32"], "10:0": [[18, 32, 1024, 1024], "float32"]}, "outputs": ["add_310:0"]} --verbose 35'
.........[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 32, 54]  [1, 1024, 1024, 576]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 64, 54]  [1, 512, 512, 1152]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 128, 54]  [1, 256, 256, 2304]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 256, 54]  [1, 128, 128, 4608]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 512, 54]  [1, 64, 64, 9216]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 512, 54]  [1, 32, 32, 9216]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 512, 54]  [1, 16, 16, 9216]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 512, 54]  [1, 8, 8, 9216]
[03:15:06] /opt/brazil-pkg-cache/packages/DmlcTvm/DmlcTvm-1.7.2.0/AL2_x86_64/generic-flavor/src/src/relay/pass/do_replication.cc:66: Replication is not applied to group convolution. Kernel and data input channels do not match[1, 1, 512, 54]  [1, 4, 4, 9216]
......./data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  sys.exit(main())
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: ***************************************************************
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:  An Internal Compiler Error has occurred
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: ***************************************************************
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: 
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Error message:  Axis                   for (i1_228_1: range(0, 8, 1)) { # indent=18
                    |V2[3, 3, 2, 3, 131, 1, 128] |V2<1 x 64> TongaSB partitions[5] |V2 [3, 3, 2, 3, 131, 1, 128] %'Reshape_68:0'[n_274_0_3437_4338_0_0,n_274_0_3437_4338_0_1,n_274_0_3437_4338_1,n_274_1_3437_4338,64i1_228_0_0+16i1_228_0_1+i2_228+2i1_228_1+2,0,2i0.64+i4_228] = tensor_copy(|V2<1 x 64> TongaSB partitions[8] |V2 [2, 2, 3, 3, 2, 3, 2, 4, 1, 512] %'Reshape_67:0'[i2_228,i4_228,n_274_0_3437_4338_0_0,n_274_0_3437_4338_0_1,n_274_0_3437_4338_1,n_274_1_3437_4338,i1_228_0_0,i1_228_0_1,0,i0.64+64i1_228_1]) # id=4937, instances=13824 # dl = Reshape_68 if i2_228 == 0 and i4_228 == 0
                  } doesn't appear in params or loopnest
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: 
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Error class:    ValueError
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Error location: Unknown
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Command line:   /data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8paifo8b/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8paifo8b/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"], "2:0": [[18, 512, 8, 8], "float32"], "3:0": [[18, 512, 16, 16], "float32"], "4:0": [[18, 512, 32, 32], "float32"], "5:0": [[18, 512, 64, 64], "float32"], "6:0": [[18, 256, 128, 128], "float32"], "7:0": [[18, 128, 256, 256], "float32"], "8:0": [[18, 64, 512, 512], "float32"], "9:0": [[18, 1, 1024, 1024], "float32"], "10:0": [[18, 32, 1024, 1024], "float32"]}, "outputs": ["add_310:0"]}' --verbose 35
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: 
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Internal details:
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/CommandDriver.py", line 223, in neuroncc.driver.CommandDriver.CommandDriver.run
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/commands/CompileCommand.py", line 485, in neuroncc.driver.commands.CompileCommand.CompileCommand.run
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/Pipeline.py", line 30, in neuroncc.driver.Pipeline.Pipeline.runSingleInput
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/Job.py", line 289, in neuroncc.driver.Job.SingleInputJob.run
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 419, in neuroncc.driver.jobs.Frontend.Frontend.runSingleInput
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 369, in neuroncc.driver.jobs.Frontend.Frontend.runTVMFrontend
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 370, in neuroncc.driver.jobs.Frontend.Frontend.runTVMFrontend
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 380, in neuroncc.driver.jobs.Frontend.Frontend.runTVMFrontend
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/driver/jobs/Frontend.py", line 350, in neuroncc.driver.jobs.Frontend.Frontend.runTVMFrontend.tensorize
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/Frontend.py", line 198, in neuroncc.starfish.penguin.Frontend.tensorizeRelay
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/Frontend.py", line 199, in neuroncc.starfish.penguin.Frontend.tensorizeRelay
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/Frontend.py", line 221, in neuroncc.starfish.penguin.Frontend.tensorizeRelay
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/Frontend.py", line 111, in neuroncc.starfish.penguin.Frontend.compile_cu
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 378, in neuroncc.starfish.penguin.DotTransform.PassManager.transformFunction
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 124, in neuroncc.starfish.penguin.DotTransform.DotTransform.runOnFunction
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 178, in neuroncc.starfish.penguin.DotTransform.DotTransform.run_with_exception_handling
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 160, in neuroncc.starfish.penguin.DotTransform.DotTransform.run_with_exception_handling
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 186, in neuroncc.starfish.penguin.DotTransform.DotTransform.timed_run_
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 188, in neuroncc.starfish.penguin.DotTransform.DotTransform.timed_run_
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 189, in neuroncc.starfish.penguin.DotTransform.DotTransform.timed_run_
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 206, in neuroncc.starfish.penguin.DotTransform.DotTransform.run_
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 207, in neuroncc.starfish.penguin.DotTransform.DotTransform.run_
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 287, in neuroncc.starfish.penguin.DotTransform.DotTransform.transformFunction
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 281, in neuroncc.starfish.penguin.DotTransform.DotTransform.runTransforms
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 141, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TensorInitialization.transformStmts
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/DotTransform.py", line 115, in neuroncc.starfish.penguin.DotTransform.DotTransform.transform
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 324, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TensorInitialization.transformTensor
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 254, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TensorInitialization.memsetTile
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 47, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TongaMemsetSimplifier.codegenMemset
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 40, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TongaMemsetSimplifier.access_without_predicates
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 41, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TongaMemsetSimplifier.access_without_predicates
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/IntegerSetAnalysis.py", line 473, in neuroncc.starfish.penguin.IntegerSetAnalysis.IntegerSetAnalysis.create_access
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/targets/tonga/passes/TensorInitialization.py", line 28, in neuroncc.starfish.penguin.targets.tonga.passes.TensorInitialization.TongaMemsetSimplifier.create_access_without_domain
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/IntegerSetAnalysis.py", line 410, in neuroncc.starfish.penguin.IntegerSetAnalysis.IntegerSetAnalysis.create_access_map
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/IntegerSetAnalysis.py", line 254, in neuroncc.starfish.penguin.IntegerSetAnalysis.IntegerSetWrapper.create_access_map_impl
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/IntegerSetAnalysis.py", line 228, in neuroncc.starfish.penguin.IntegerSetAnalysis.IntegerSetWrapper.build_aff
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   File "neuroncc/starfish/penguin/IntegerSetAnalysis.py", line 197, in neuroncc.starfish.penguin.IntegerSetAnalysis.IntegerSetWrapper.build_aff.build
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: 
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Version information:
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   Neuron Compiler version 1.7.3.0+c02f9fde4
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   HWM version 1.7.0.0-0
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   NEFF version Dynamic
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   TVM version 1.7.2.0+0
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   NumPy version 1.20.0
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   MXNet not available
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]:   TF not available
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: 
12/03/2021 03:15:28 AM ERROR 54523 [neuron-cc]: Artifacts stored in: /tmp/tmp8paifo8b

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$432; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8paifo8b/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8paifo8b/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"], "2:0": [[18, 512, 8, 8], "float32"], "3:0": [[18, 512, 16, 16], "float32"], "4:0": [[18, 512, 32, 32], "float32"], "5:0": [[18, 512, 64, 64], "float32"], "6:0": [[18, 256, 128, 128], "float32"], "7:0": [[18, 128, 256, 256], "float32"], "8:0": [[18, 64, 512, 512], "float32"], "9:0": [[18, 1, 1024, 1024], "float32"], "10:0": [[18, 32, 1024, 1024], "float32"]}, "outputs": ["add_310:0"]}' --verbose 35
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/convert.py", line 391, in op_converter
    item, inputs, compiler_workdir=sg_workdir, **kwargs)
  File "/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/decorators.py", line 194, in trace
    'neuron-cc failed with the following command line call:\n{}'.format(command))
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp8paifo8b/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp8paifo8b/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[18, 18, 512], "float32"], "1:0": [[18, 512, 4, 4], "float32"], "2:0": [[18, 512, 8, 8], "float32"], "3:0": [[18, 512, 16, 16], "float32"], "4:0": [[18, 512, 32, 32], "float32"], "5:0": [[18, 512, 64, 64], "float32"], "6:0": [[18, 256, 128, 128], "float32"], "7:0": [[18, 128, 256, 256], "float32"], "8:0": [[18, 64, 512, 512], "float32"], "9:0": [[18, 1, 1024, 1024], "float32"], "10:0": [[18, 32, 1024, 1024], "float32"]}, "outputs": ["add_310:0"]}' --verbose 35
INFO:Neuron:Number of arithmetic operators (post-compilation) before = 2578, compiled = 146, percent compiled = 5.66%
INFO:Neuron:The neuron partitioner created 19 sub-graphs
INFO:Neuron:Neuron successfully compiled 4 sub-graphs, Total fused subgraphs = 19, Percent of model sub-graphs successfully compiled = 21.1%
INFO:Neuron:Compiled these operators (and operator counts) to Neuron:
INFO:Neuron: => aten::Int: 36
INFO:Neuron: => aten::_convolution: 3
INFO:Neuron: => aten::add: 9
INFO:Neuron: => aten::leaky_relu: 3
INFO:Neuron: => aten::linear: 3
INFO:Neuron: => aten::mul: 27
INFO:Neuron: => aten::pow: 3
INFO:Neuron: => aten::rsqrt: 3
INFO:Neuron: => aten::select: 4
INFO:Neuron: => aten::size: 30
INFO:Neuron: => aten::slice: 3
INFO:Neuron: => aten::sum: 3
INFO:Neuron: => aten::unsqueeze: 1
INFO:Neuron: => aten::view: 18
INFO:Neuron:Not compiled operators (and operator counts) to Neuron:
INFO:Neuron: => aten::Int: 654 [supported]
INFO:Neuron: => aten::_convolution: 39 [supported]
INFO:Neuron: => aten::add: 315 [supported]
INFO:Neuron: => aten::constant_pad_nd: 32 [supported]
INFO:Neuron: => aten::flip: 16 [supported]
INFO:Neuron: => aten::floor_divide: 32 [supported]
INFO:Neuron: => aten::leaky_relu: 14 [supported]
INFO:Neuron: => aten::linear: 23 [supported]
INFO:Neuron: => aten::mul: 316 [supported]
INFO:Neuron: => aten::new_empty: 17 [not supported]
INFO:Neuron: => aten::normal_: 17 [not supported]
INFO:Neuron: => aten::permute: 32 [supported]
INFO:Neuron: => aten::pow: 14 [supported]
INFO:Neuron: => aten::repeat: 2 [not supported]
INFO:Neuron: => aten::reshape: 56 [supported]
INFO:Neuron: => aten::rsqrt: 14 [supported]
INFO:Neuron: => aten::select: 23 [supported]
INFO:Neuron: => aten::size: 355 [supported]
INFO:Neuron: => aten::slice: 151 [supported]
INFO:Neuron: => aten::sub: 96 [supported]
INFO:Neuron: => aten::sum: 14 [supported]
INFO:Neuron: => aten::transpose: 8 [supported]
INFO:Neuron: => aten::view: 192 [supported]
WARNING:Neuron:torch.neuron.trace was unable to compile > 50% of the operators in the compiled model!
WARNING:Neuron:Please review the torch.neuron.analyze_model output and if you believe you are seeing a failure
WARNING:Neuron:Lodge an issue on https://github.com/aws/aws-neuron-sdk/issues if you believe the model is not compiling as expected
..................................................torch.Size([1, 18, 512])
.....................................................................................................................................................................................................................

Logs when tracing the ModulatedConv2d layer

/data/projects/torch_neuron_test/model.py:258: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out = F.conv_transpose2d(input, weight, padding=0, stride=2, groups=batch)
/data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
INFO:Neuron:All operators are compiled by neuron-cc (this does not guarantee that neuron-cc will successfully compile)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 125, fused = 125, percent fused = 100.0%
WARNING:tensorflow:From /data/anaconda3/envs/pytorch19/lib/python3.7/site-packages/torch_neuron/ops/aten.py:1751: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
INFO:Neuron:Compiling function _NeuronGraph$15 with neuron-cc
INFO:Neuron:Compiling with command line: '/data/anaconda3/envs/pytorch19/bin/neuron-cc compile /tmp/tmp626b2rvw/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp626b2rvw/graph_def.neff --io-config {"inputs": {"0:0": [[1, 16, 64, 64], "float32"], "1:0": [[1, 512], "float32"]}, "outputs": ["Reshape_15:0"]} --verbose 35'
./data/anaconda3/envs/pytorch19/bin/neuron-cc:8: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  sys.exit(main())
--Call--
> /data/anaconda3/envs/pytorch19/lib/python3.7/contextlib.py(116)__exit__()
-> def __exit__(self, type, value, traceback):
(Pdb) ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
aws-taylor commented 2 years ago

Hello @PigletOS,

Thank you for the report. We are investigating the issue.

-Taylor

awsrjh commented 2 years ago

HI PigletOS

we are still working on this. We will keep you posted with updates.

aws-zejdaj commented 2 years ago

Hi @PigletOS , We have implemented a workaround for the Framework error on Reshape_8. However, the model has very low ratio of compute to weight size as well as internal state size therefore the performance on Inferentia will be limited.