search

Xilinx / Vitis-AI

Vitis AI is Xilinx’s development stack for AI inference on Xilinx hardware platforms, including both edge devices and Alveo cards.
https://www.xilinx.com/ai
Apache License 2.0
1.49k stars 633 forks source link

Alveo U200 compilation error #287

Open daivikbhatia opened 3 years ago

daivikbhatia commented 3 years ago

Hi, While compiling the quantized model for alveo U200 I'm getting the following error message: (vitis-ai-tensorflow) Vitis-AI /workspace > ARCH=/opt/vitis_ai/compiler/arch/DPUCADF8H/U200/arch.json

(vitis-ai-tensorflow) Vitis-AI /workspace > vai_c_tensorflow \

   --frozen_pb=./quantize/quantize_eval_model.pb \
   --arch=${ARCH} \
   --output_dir=launchmodel \
   --net_name=SignLanguageMNISTnet
  • VITIS_AI Compilation - Xilinx Inc.

    [INFO] Namespace(inputs_shape=None, layout='NHWC', model_files=['./quantize/quantize_eval_model.pb'], model_type='tensorflow', out_filename='launchmodel/SignLanguageMNISTnet_org.xmodel', proto=None) [INFO] tensorflow model: quantize/quantize_eval_model.pb [INFO] parse raw model :100%|█| 30/30 [00:00<00:00, 10626.56it/s]
    [INFO] infer shape (NHWC) :100%|█| 43/43 [00:00<00:00, 4001.40it/s]
    [INFO] infer shape (NHWC) :100%|█| 39/39 [00:00<00:00, 18076.90it/s]
    [INFO] generate xmodel :100%|█| 39/39 [00:00<00:00, 268.09it/s]
    [INFO] generate xmodel: /workspace/launchmodel/SignLanguageMNISTnet_org.xmodel [UNILOG][INFO] The compiler log will be dumped at "/tmp/vitis-ai-user/log/xcompiler-20210205-044100-116" [UNILOG][INFO] Compiling subgraph : subgraph_activation_1_1/Relu #############################################

    Hyper Graph Construction

    ############################################# #############################################

    Hyper Graph Construction

    ############################################# Floyd & Warshall BFS #############################################

    Parameters Assimilation: # 10

    ############################################# #############################################

    Assimilating Fix Neurons: # 8

    ############################################# #############################################

    Assimilating Relu: # 2

    ############################################# #############################################

    Assimilating LeakyRelu: # 0

    ############################################# #############################################

    I like VALID more than SAME

    ############################################# conv2d_1_1/convolution SAME [0, 0, 0, 0] conv2d_2_1/convolution SAME [0, 0, 0, 0] #############################################

    I like VALID more than SAME: # 2

    ############################################# #############################################

    Assimilating Padding:# 0

    ############################################# #############################################

    CPU nodes Must Go

    ############################################# Inputs ['input_1_1/aquant'] Outputs ['max_pooling2d_2_1/MaxPool'] FPGA True: data input_1_1/aquant
    FPGA False: const batch_normalization_2_1/FusedBatchNormV3_1/offset/wquant_const_const
    FPGA False: const batch_normalization_1_1/FusedBatchNormV3_1/offset/wquant_const_const
    FPGA True: conv2d conv2d_1_1/convolution
    FPGA True: mul batch_normalization_1_1/FusedBatchNormV3_1/mul
    FPGA True: add batch_normalization_1_1/FusedBatchNormV3_1/add
    FPGA True: maxpool2d max_pooling2d_1_1/MaxPool
    FPGA True: conv2d conv2d_2_1/convolution
    FPGA True: mul batch_normalization_2_1/FusedBatchNormV3_1/mul
    FPGA True: add batch_normalization_2_1/FusedBatchNormV3_1/add
    FPGA True: maxpool2d max_pooling2d_2_1/MaxPool
    delete these dict_keys(['input_1_1/aquant', 'batch_normalization_2_1/FusedBatchNormV3_1/offset/wquant_const_const', 'batch_normalization_1_1/FusedBatchNormV3_1/offset/wquant_const_const', 'batch_normalization_1_1/FusedBatchNormV3_1/add', 'max_pooling2d_1_1/MaxPool', 'conv2d_2_1/convolution', 'batch_normalization_2_1/FusedBatchNormV3_1/mul', 'batch_normalization_2_1/FusedBatchNormV3_1/add']) Delete Node batch_normalization_2_1/FusedBatchNormV3_1/offset/wquant_const_const delete arch batch_normalization_2_1/FusedBatchNormV3_1/mul-batch_normalization_2_1/FusedBatchNormV3_1/add True delete arch batch_normalization_2_1/FusedBatchNormV3_1/offset/wquant_const_const-batch_normalization_2_1/FusedBatchNormV3_1/add True delete arch batch_normalization_2_1/FusedBatchNormV3_1/add-max_pooling2d_2_1/MaxPool True delete node max_pooling2d_2_1/MaxPool delete arch batch_normalization_1_1/FusedBatchNormV3_1/mul-batch_normalization_1_1/FusedBatchNormV3_1/add True delete arch batch_normalization_1_1/FusedBatchNormV3_1/offset/wquant_const_const-batch_normalization_1_1/FusedBatchNormV3_1/add True delete arch batch_normalization_1_1/FusedBatchNormV3_1/add-max_pooling2d_1_1/MaxPool True delete arch max_pooling2d_1_1/MaxPool-conv2d_2_1/convolution True delete arch conv2d_2_1/convolution-batch_normalization_2_1/FusedBatchNormV3_1/mul True delete node batch_normalization_2_1/FusedBatchNormV3_1/mul Schedule boost 0 data input_1_1/aquant False 1 1 conv2d conv2d_1_1/convolution True 1 2 mul batch_normalization_1_1/FusedBatchNormV3_1/mul True 1 Outputs ['batch_normalization_1_1/FusedBatchNormV3_1/mul'] Inputs ['input_1_1/aquant'] Floyd & Warshall BFS #############################################

    Avg Pool -> Conv

    ############################################# #############################################

    Inner Products -> Conv

    ############################################# #############################################

    Scale -> Conv

    ############################################# #############################################

    Concat of concat

    ############################################# Floyd & Warshall BFS #############################################

    topological schedule BFS

    ############################################# #############################################

    WEIGHT & BIAS into Tensors

    ############################################# #############################################

    topological DFS

    ############################################# DFS_t input_1_1/aquant #############################################

    TFS

    ############################################# #############################################

    INC

    ############################################# INC #############################################

    Singleton

    ############################################# 0 data input_1_1/aquant Ops 0 Shape [1, 28, 28, 1] IN [] OUT ['conv2d_1_1/convolution'] 1 conv2d conv2d_1_1/convolution Ops 197568 Shape [1, 28, 28, 28] IN ['input_1_1/aquant'] OUT ['batch_normalization_1_1/FusedBatchNormV3_1/mul'] 2 no good batch_normalization_1_1/FusedBatchNormV3_1/mul Ops 0 Shape [1, 28, 28, 28] IN ['conv2d_1_1/convolution'] OUT [] #############################################

    Given a Graph and Schedule boost : We crete Live Tensor

    ############################################# #############################################

    Reset Live Structure

    ############################################# #############################################

    Attempting Code Generation boost

    ############################################# #############################################

    Element Wise: reuse one of the operands

    ############################################# #############################################

    Concatenation: I love concatenation memory reuse

    ############################################# #############################################

    Memory Management given a Schedule

    ############################################# /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/SC/HwAbstraction/gen_param_ddr_demo.py(123)bias_data_int() -> shift_bias = pos_i + pos_w - pos_b #-1 (Pdb)

This gives me only 1 file,SignLanguageMNISTnet_org.xmodel in output directory. Also when I tried to compile the same quantized file for alveo U50, the compilation ran just fine and gave correct 4 output files. Can any one please look into this issue and tell what's this error about alveo U200 and how to solve it?

paolodalberto commented 3 years ago

that is interesting: it looks like a debug stop. Also the compiler really does not care if it is U200 or U50 ... Let me see if I can retrieve the model and see if I can reproduce ....

paolodalberto commented 3 years ago

The schedule should be composed by a single convolution. There is a failure in combining the batch norm:

2 no good batch_normalization_1_1/FusedBatchNormV3_1/mul Ops 0 Shape [1, 28, 28, 28] IN ['conv2d_1_1/convolution'] OUT []

I would say that the failure is in the translation from tf to xmodel .... is there a way to confirm ?