[XRT] ERROR: failed to load xclbin: Invalid argument / xocl_read_axlf_helper: interface uuids do not match

[BogdanSorlea]

LE: see my next reply first

I can't seem to be able to run the hello_world example and I'm not sure what I'm holding wrong here. Issues here don't seem to hit anything on the topic (or maybe I'm searching the wrong way(?)

Some details about state and issue:

• instance type: f1.2xlarge
• AMI ID / name: ami-046b65e85f58df5b1 / FPGA Developer AMI - 1.12.2-40257ab5-6688-4c95-97d1-e251a40fd1fc

• git status/log of aws-fpga repo clone is oddly at some commit number although I didn't seem to run any git command myself to put it there (only did the git clone, will paste bash history as well) LE: nevermind, I just realised it's a git submodule and that hash is the commit it's pinned to)

  [centos@ip-10-129-62-160 hello_world]$ git status
  # HEAD detached at f61637e
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #   .ipcache/
  #   _x.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/
  #   _x/
  #   build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/
  #   hello_world
  #   nohup.out
  #   summary.csv
  #   xrt.run_summary
  nothing added to commit but untracked files present (use "git add" to track)
  [centos@ip-10-129-62-160 hello_world]$ git log -n1
  commit f61637e9263ecc1be3df34182ea6c53a0ca10447
  Author: Bhaskar VishnuVardhan Chebrolu <bhaskar.vishnuvardhan.chebrolu@xilinx.com>
  Date:   Wed Dec 22 10:04:22 2021 +0530
  
  updated the aie_adder description.json file
  [centos@ip-10-129-62-160 hello_world]$

• history (trimmed some of the sensitive and/or irelevant (e.g. ll / ls / less) lines)

  7  sudo vi /etc/environment
  17  echo $AWS_FPGA_REPO_DIR
  20  git clone https://github.com/aws/aws-fpga.git $AWS_FPGA_REPO_DIR
  24  cd src/
  26  cd project_data/
  28  cd aws-fpga/
  30  source vitis_setup.sh
  31  systemctl is-active --quiet mpd || sudo systemctl start mpd
  33  cd Vitis/examples/
  35  cd xilinx
  37  cd hello_world/
  39  make
  40  echo $AWS_PLATFORM
  41  echo $AWS_PLATFORM_201920_3
  42  make all
  43  make all PLATFORM=$AWS_PLATFORM
  44  make clean
  45  nohup make all PLATFORM=$AWS_PLATFORM &
  46  less nohup.out
  47  ps aux | grep make
  48  source ../../../../vitis_runtime_setup.sh
  49  dmesg | less
  51  ll build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/
  52  ./hello_world
  53  ./hello_world build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/vadd.xclbin
  54  dmesg > dmesg.log
  55  less dmesg.log
  56  less terminal.log
  57  vi terminal.log
  58  history | less
  59  history > history.log

• can provide the terminal output upon request or any of the logs that might have been generated during make
• while trying to run it, I get:

  [centos@ip-10-129-62-160 hello_world]$ ./hello_world build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/vadd.xclbin
  Found Platform
  Platform Name: Xilinx
  INFO: Reading build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/vadd.xclbin
  Loading: 'build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/vadd.xclbin'
  Trying to program device[0]: xilinx_aws-vu9p-f1_dynamic-shell
  XRT build version: 2.12.0
  Build hash: 723d9e7abbe3a2c374682dbb1a59c47f230f3ee2
  Build date: 2022-01-11 19:12:26
  Git branch: 2021.2
  PID: 2022
  UID: 1000
  [Wed Nov  8 13:09:04 2023 GMT]
  HOST: ip-10-129-62-160.eu-west-1.compute.internal
  EXE: /home/centos/src/project_data/aws-fpga/Vitis/examples/xilinx_2021.2/hello_world/hello_world
  [XRT] ERROR: See dmesg log for details. err=-2
  [XRT] ERROR: failed to load xclbin: Invalid argument
  Failed to program device[0] with xclbin file!
  Failed to program any device found, exit!
  [centos@ip-10-129-62-160 hello_world]$

• dmesg said as follows:

  [ 1654.859445] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 _xocl_drvinst_open: OPEN 1
  [ 1654.859543] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 process_request: received request from peer: 12, passed on
  [ 1654.859550] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_mailbox_srv: received request (12) from peer
  [ 1654.859556] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_mailbox_srv: mgmt driver online
  [ 1654.860681] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_refresh_subdevs: get fdt from peer
  [ 1654.860987] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 _mailbox_request: sending request: 10 via SW
  [ 1654.861096] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 mailbox_read: Software TX msg is too big
  [ 1654.862023] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 _mailbox_request: sending request: 11 via SW
  [ 1654.862085] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 mailbox_read: Software TX msg is too big
  [ 1654.862202] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_mb_connect: ch_state 0x1, ret 0
  [ 9580.445374] xocl 0000:00:1d.0:  ffff94b1b61f7098 _xocl_drvinst_open: OPEN 1
  [ 9580.445395] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_create_client: created KDS client for pid(21857), ret: 0
  [ 9580.459018] xocl 0000:00:1d.0: icap.u.23068672 ffff94cf03084410 icap_read_from_peer: reading from peer
  [ 9580.459033] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 _mailbox_request: sending request: 10 via SW
  [ 9580.459186] xocl 0000:00:1d.0: mailbox.u.9437184 ffff94cefdda0810 mailbox_read: Software TX msg is too big
  [ 9580.478818] xocl 0000:00:1d.0: icap.u.23068672 ffff94cf03084410 icap_cached_ocl_frequency: no cached data for 3
  [ 9580.493540] xocl 0000:00:1d.0: icap.u.23068672 ffff94cf03084410 icap_cached_ocl_frequency: no cached data for 3
  [ 9580.724064] xocl 0000:00:1d.0:  ffff94b1b61f7098 xclbin_downloaded: p2p configure changed
  [ 9580.780521] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_read_axlf_helper: check interface uuid
  [ 9580.780529] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_fdt_check_uuids: invalid blob
  [ 9580.789435] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_read_axlf_helper: interface uuids do not match
  [ 9580.800014] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_read_axlf_helper: Failed to download xclbin, err: -22
  [ 9580.825250] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_destroy_client: client exits pid(21857)
  [ 9580.825258] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_drvinst_close: CLOSE 2
  [ 9580.825261] xocl 0000:00:1d.0:  ffff94b1b61f7098 xocl_drvinst_close: NOTIFY ffff94cf03082c10

Really unsure what is happening, but interface uuids do not match in dmesg and [XRT] ERROR: failed to load xclbin: Invalid argument / Failed to program device[0] with xclbin file! during ./hello_world build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/vadd.xclbin seem like the place where figuring out should start from.

Let me know if there's any other input I should be dropping here.

Would love to know also how get deeper into debugging these types of issues.

[BogdanSorlea]

uhm, so I guess it's because I haven't been doing create_vitis_afi.sh, i.e. have AWS package it and have it stored in S3. doing that I seem to be able to program the FPGA just fine. so I'm wondering - is there no way to do this without pushing to s3 every time? I mean it feels like an unnecessary round-trip - and some of the docs I see around (from various repos/sources) would suggest it's not needed (i.e. loading an .xclbin file, not necessarily an .awsxclbin one)

[AWSjoeluc]

Hi!

Thanks for your question, I'm glad you were able to solve the issue. Customers are required to use the create_vitis_afi.sh to create AFI's and store artifacts in S3. When loading Vitis images, the scripts call fpga-load-local-image <AFI-ID>. To preserve security and health of the FPGA, all AFI's (verified Xilinx bitstreams and .xclbin files) are loaded by AWS after downloading them from S3. All of the aws-fpga workflows utilize S3 to store artifacts and load images to the FPGA.

If you have any more questions, please don't hesitate to ask!

Thanks, -Joe

[BogdanSorlea]

@AWSjoeluc was it the case at some point in the past direct loading of the bitstream was possible - and that's why I was seeing docs mentioning .xclbin (I believe it was some of the reinvent workshop docs from a few years back, but I can't remember exactly which docs I was seeing)? or maybe I'm confusing it with just generic vendor-specific (Xilinx) docs?

just trying to piece together what I've been seeing, not that I remember it that well anymore

[AWScsaralay]

Hi, F1 instances only support .awsxclbin because the customer design needs to be interfaced with the AWS Shell. This process is not done by generic vitis flow which generates .xclbin. Perhaps you may be referring to Xilinx docs and vitis flow for on-prem devices, but for F1 customers are expected to use .awsxclbin for loading their designs into F1 FPGA.

Please reach out to us if you have any questions.

Thanks! Chakra