model-profiler treats points within the model as inputs

[G4V]

Hello.

I'm working with a simple example of taking the stock LSTM from within PyTorch, converting it to ONNX and then to Glow. This I can do successfully and all stages the resulting models work as expected (with opset 9).

I'm now trying to quantize the Glow model. The LSTM contains three inputs (input, h_0 and c0) but the profiler expects seven. The four extra inputs are in fact outputs from the two internal layers of the LSTM.

input: input
input: c_0
input: LSTM_6.Y_h
input: h_0
input: LSTM_2.Y_c
input: LSTM_6.Y_c
input: LSTM_2.Y_h

The command line for the model-profiler -

model-profiler -model=model.onnx -dump-profile='.' -input-dataset=input,rawbin,file,inputdataset.txt -input-dataset=h_0,rawbin,file,h_0dataset.txt -input-dataset=c_0,rawbin,file,c_0dataset.txt -onnx-define-symbol=batch,1

The parameters passed to the ONNX exporter.

torch.onnx.export(model, (x,(h, c)),
                      args.dest,
                      opset_version=9,
                      input_names=['input', 'h_0', 'c_0'],
                      output_names=['output', 'h_n', 'c_n'],
                      dynamic_axes = {'input': {0: 'batch'}, 'output': {0: 'batch'}},
                      example_outputs = (o,(hn, cn)),
                      verbose=True
              )

The additional inputs show up as internal state when viewing the ONNX model in Netron.

To see what happens, I tried giving these extra 'inputs' representative data, but the output of running inference through bundle was incorrect.

Any help on what may be going on, much appreciated.

Thanks in advance.

[jfix71]

@G4V What error are you seeing exactly? It would be useful to also see the loaded graph in Glow that causes whatever issue you're seeing.

We don't have a flag for this it seems, but you can add a single line below to have this happen: https://github.com/pytorch/glow/blob/c095b25748e640cc5aabcf27d6970b127b57e3ca/tools/loader/ModelProfiler.cpp#L242 cctx.dumpFinalGraph = true;

And then look for some file called final_graph_*.dot

[G4V]

Hi @jfix71, The error I'm seeing is -

ERROR: Not all the model inputs where provided with the 'input-dataset' parameter!

The LSTM I'm exporting through ONNX is just the stock LSTM provided by PyTorch. It takes three inputs and gives three outputs. https://pytorch.org/docs/master/generated/torch.nn.LSTM.html

For some reason, when passed to the model profiler it expects seven inputs. The additional inputs are -

LSTM_6.Y_h, LSTM_2.Y_c, LSTM_6.Y_c, LSTM_2.Y_h

I added in your line and re-ran. I've also dumped the dot file before the transformation. It's very odd as there are several nodes in the graph before transformation that do show these 'inputs' as having no parents. Both dot files in the zip...

dotfiles.zip

I'm now not sure whether these inputs are an artifact of the ONNX exporter or that Glow should be ignoring these. These inputs are not identified as such when viewing the ONNX file in Netron, so it may be that they're tagged differently?

By giving the model-profiler representative data for those inputs I can get the model-profiler to produce the yaml file which I can then feed into the model-compiler, but the resultant model, when used as a replacement for the non quantized equivalent, mainly spits out zeros. This may or may not be due to these additional inputs?

I've also tried removing the check to ensure all inputs have been provided to see if these could safely be ignored, but unsurprisingly the model-profiler failed further down the line.

All very strange.

Many thanks for taking a look!

[jfix71]

Unfortunately this is a pretty esoteric bug dealing with LSTMs and the model-profiler which I'm not super familiar with, and I don't currently have the cycles to dig in here. @mciprian13 wrote much of this code and would probably be better equipped to provide some tips or insight.

[G4V]

Thanks @jfix71. Although these inputs shouldn't be required, supplying representative values seems to work. I debugged the issue where I was seeing mainly zeros at the output and it was to do with a node in the graph that couldn't be represented effectively in I8, which I've manged to work around.

One other oddity though, is that if I set the precision to I16 I end up with just the floating point variant.

[mciprian13]

@G4V Sorry for delayed response. The Glow ONNX importer when importing and LSTM module it creates some additional graph placeholders corresponding to the LSTM states (hidden state "H" and cell state "C") which you can use to define the initial state of the LSTM module before running the inference (most commonly buffers filled with 0's are used). These placeholders are also updated automatically when doing inference allowing you to track the LSTM state while doing multiple subsequent inferences. The LSTM states are exported unconditionally in the graph interface when compiled, whether the original model (viewed in Netron) exposes them or not. Therefore when using model-profiler you should also provide data for these extra input graph variables (4 in your case which I assume comes from the fact that you have 2 LSTM cells) which most probably in your case should be filled with 0's.

One other oddity though, is that if I set the precision to I16 I end up with just the floating point variant.

This comes from the fact that when Glow quantizes a graph it first searches if it has int16 implementations for the operators. If not (which I assume happens in your case) then the float operators are used. So in your case it means that Glow does not have int16 implementation for the operators of your model.

[G4V]

Thanks @mciprian13.

The inputs for the hidden and cell states for both cells are already covered by c_0 and h_0, though. The others prepended with LSTM are only ever internal to the LSTM.

input: input
input: c_0
input: LSTM_6.Y_h
input: h_0
input: LSTM_2.Y_c
input: LSTM_6.Y_c
input: LSTM_2.Y_h

From the graph, they look like outputs rather than inputs?

[mciprian13]

@G4V I pushed here #4830 a PR which modifies the default way in which an RNN operator is loaded (either RNN, GRU or LSTM operator) by not exporting by default the state variables as placeholders. That way the imported model will have the same number of inputs and outputs as the original model. This way the profiling using the model-profiler should be somewhat easier to use.