search

ros-acceleration / acceleration_firmware_kv260

KV260 firmware. Package for enabling hardware acceleration capabilities in ROS 2 with KV260.
Apache License 2.0
10 stars 4 forks source link

download failed with timeout error #1

Closed surender009 closed 2 years ago

surender009 commented 2 years ago

while downloading sources using krs_alpha.repos using vcs command

surender@xhdsurender41:~/krs_ws$ vcs import src --recursive < krs_alpha.repos .....E === src/acceleration/acceleration_firmware (git) ===

HEAD is now at 913fe05 Add CHANGELOG.rst

=== src/acceleration/acceleration_firmware_kv260 (zip) === Invocation of command 'import' on client 'zip' failed: timeout: The read operation timed out (/usr/lib/python3.8/ssl.py:1099) === src/acceleration/ament_vitis (git) ===

HEAD is now at 219c04a Add CHANGELOG

=== src/acceleration/colcon-acceleration (git) ===

HEAD is now at 9ff8ca2 0.2.0

=== src/acceleration/ros2acceleration (git) ===

HEAD is now at a6a1a85 0.2.0 CHANGELOG updates

=== src/acceleration/vitis_common (git) ===

HEAD is now at 8805009 Add utilities library

vmayoral commented 2 years ago

@surender009,

For context, I presume your ask is related to instructions at https://xilinx.github.io/KRS/sphinx/build/html/docs/install.html. Note that acceleration_firmware_kv260 is being fetched directly from GitHub's releases. See the following line in the krs_alpha.repos file:

  acceleration/acceleration_firmware_kv260:
    type: zip
    url: https://github.com/ros-acceleration/acceleration_firmware_kv260/releases/download/v0.6.0/acceleration_firmware_kv260.zip

This is a 1.6 GB file so we might be encountering some issues due to the size itself. Tests I went through to try and reproduce your issues:

  1. ❌ I indeed obtained the same result you did with vcs. Further looking into it, there seems to be a wider issue with operations timing out with big files with Python-based clients, so I believe this is not specific to us. I did a bit of further testing:
  2. ✅ Downloading it from the browser: typing https://github.com/ros-acceleration/acceleration_firmware_kv260/releases/download/v0.6.0/acceleration_firmware_kv260.zip in your browser. I succeed downloading it. You just then need to unzip it and place it under src in your workspace
  3. ✅ wgeting it
wget https://github.com/ros-acceleration/acceleration_firmware_kv260/releases/download/v0.6.0/acceleration_firmware_kv260.zip
--2021-11-16 08:53:02--  https://github.com/ros-acceleration/acceleration_firmware_kv260/releases/download/v0.6.0/acceleration_firmware_kv260.zip
Resolviendo github.com (github.com)... 140.82.121.4
Conectando con github.com (github.com)[140.82.121.4]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Found
Localización: https://objects.githubusercontent.com/github-production-release-asset-2e65be/376909513/ba0e1de1-53b4-4aad-a6fd-a2f5a7c57218?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211116%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211116T075302Z&X-Amz-Expires=300&X-Amz-Signature=04baac28924a42852d2cf6dc600116db32519b0d190c40ba7142169817de9ed0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=376909513&response-content-disposition=attachment%3B%20filename%3Dacceleration_firmware_kv260.zip&response-content-type=application%2Foctet-stream [siguiendo]
--2021-11-16 08:53:02--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/376909513/ba0e1de1-53b4-4aad-a6fd-a2f5a7c57218?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211116%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211116T075302Z&X-Amz-Expires=300&X-Amz-Signature=04baac28924a42852d2cf6dc600116db32519b0d190c40ba7142169817de9ed0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=376909513&response-content-disposition=attachment%3B%20filename%3Dacceleration_firmware_kv260.zip&response-content-type=application%2Foctet-stream
Resolviendo objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Conectando con objects.githubusercontent.com (objects.githubusercontent.com)[185.199.110.133]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 1735682989 (1,6G) [application/octet-stream]
Grabando a: “acceleration_firmware_kv260.zip”

acceleration_firmware_kv 100%[================================>]   1,62G  73,8MB/s    en 22s

2021-11-16 08:53:57 (73,7 MB/s) - “acceleration_firmware_kv260.zip” guardado [1735682989/1735682989]

Short answer to unblock you, please fetch it from any of the other avenues and drop it into your workspace.

vmayoral commented 2 years ago

Further looking into this to isolate potential issues:

vcs functionality validation - fetching it from a local resource ✅

This does not seem to be the issue. The following validation was performed:

mkdir src
cat << 'EOF' > krs_alpha.repos
repositories:
  acceleration/acceleration_firmware:
    type: git
    url: https://github.com/ros-acceleration/acceleration_firmware
    version: 0.4.0
  acceleration/acceleration_firmware_kv260:
    type: zip
    url: file:///Users/xilinx/Downloads/acceleration_firmware_kv260.zip/acceleration_firmware_kv260.zip
  acceleration/colcon-acceleration:
    type: git
    url: https://github.com/ros-acceleration/colcon-acceleration
    version: 0.2.0
  acceleration/ros2acceleration:
    type: git
    url: https://github.com/ros-acceleration/ros2acceleration
    version: 0.2.0
  acceleration/ament_vitis:
    type: git
    url: https://github.com/ros-acceleration/ament_vitis
    version: 0.5.0
  acceleration/vitis_common:
    type: git
    url: https://github.com/ros-acceleration/vitis_common
    version: 0.1.0
EOF

vcs import src --recursive < krs_alpha.repos
......
=== src/acceleration/acceleration_firmware (git) ===

HEAD está ahora en 913fe05 Add CHANGELOG.rst

=== src/acceleration/acceleration_firmware_kv260 (zip) ===
Downloaded zipfile from 'file:///Users/xilinx/Downloads/acceleration_firmware_kv260.zip' and unpacked it
=== src/acceleration/ament_vitis (git) ===

HEAD está ahora en 219c04a Add CHANGELOG

=== src/acceleration/colcon-acceleration (git) ===

HEAD está ahora en 9ff8ca2 0.2.0

=== src/acceleration/ros2acceleration (git) ===

HEAD está ahora en a6a1a85 0.2.0 CHANGELOG updates

=== src/acceleration/vitis_common (git) ===

HEAD está ahora en 8805009 Add utilities library

vcs functionality validation - fetching it from another server ✅

Droped the file acceleration_firmware_kv260.zip in a different server and tried fetching it from there. Particularly, placed it at http://52.211.66.236/acceleration_firmware_kv260.zip:

cat << 'EOF' > krs_alpha.repos
repositories:
  acceleration/acceleration_firmware:
    type: git
    url: https://github.com/ros-acceleration/acceleration_firmware
    version: 0.4.0
  acceleration/acceleration_firmware_kv260:
    type: zip
    url: http://52.211.66.236/acceleration_firmware_kv260.zip
  acceleration/colcon-acceleration:
    type: git
    url: https://github.com/ros-acceleration/colcon-acceleration
    version: 0.2.0
  acceleration/ros2acceleration:
    type: git
    url: https://github.com/ros-acceleration/ros2acceleration
    version: 0.2.0
  acceleration/ament_vitis:
    type: git
    url: https://github.com/ros-acceleration/ament_vitis
    version: 0.5.0
  acceleration/vitis_common:
    type: git
    url: https://github.com/ros-acceleration/vitis_common
    version: 0.1.0
EOF

vcs import src --recursive < krs_alpha.repos
......
=== src/acceleration/acceleration_firmware (git) ===

HEAD está ahora en 913fe05 Add CHANGELOG.rst

=== src/acceleration/acceleration_firmware_kv260 (zip) ===
Downloaded zipfile from 'http://52.211.66.236/acceleration_firmware_kv260.zip' and unpacked it
=== src/acceleration/ament_vitis (git) ===

HEAD está ahora en 219c04a Add CHANGELOG

=== src/acceleration/colcon-acceleration (git) ===

HEAD está ahora en 9ff8ca2 0.2.0

=== src/acceleration/ros2acceleration (git) ===

HEAD está ahora en a6a1a85 0.2.0 CHANGELOG updates

=== src/acceleration/vitis_common (git) ===

HEAD está ahora en 8805009 Add utilities library

Which succeeded.

vmayoral commented 2 years ago

With https://github.com/ros-acceleration/acceleration_firmware_kv260/issues/1#issuecomment-970002827 I believe it becomes clear that the problem's that GitHub's server is interrupting the download.

@jasvinderkhurana and @giuliocorradi, given the previous limitation, can we look for an alternative path whereto deposit firmware releases? We can continue doing public releases at https://github.com/ros-acceleration/acceleration_firmware_kv260/releases (and though it's pretty straightforward to realize the problem in here) it might be worth providing a clean hurdle-less experience.

@jasvinderkhurana and @surender009, I'd be great if you could help with the following (ask): once you figure out where firmware files can be deposited elsewhere (e.g. in a Xilinx-owned server), can you file a Pull Request against KRS to update documentation accordingly?

giuliocorradi commented 2 years ago

@surender009 @vmayoral could you please try vcs import --retry 10 to check if the problem disappears before changing things?

vmayoral commented 2 years ago

@surender009 @vmayoral could you please try vcs import --retry 10 to check if the problem disappears before changing things?

vcs import src --recursive --retry 10 < krs_alpha.repos
.....E
=== src/acceleration/acceleration_firmware (git) ===

HEAD is now at 913fe05 Add CHANGELOG.rst

=== src/acceleration/acceleration_firmware_kv260 (zip) ===
Invocation of command 'import' on client 'zip' failed: timeout: The read operation timed out (/usr/lib/python3.8/ssl.py:1099)
=== src/acceleration/ament_vitis (git) ===

HEAD is now at 219c04a Add CHANGELOG

=== src/acceleration/colcon-acceleration (git) ===

HEAD is now at 9ff8ca2 0.2.0

=== src/acceleration/ros2acceleration (git) ===

HEAD is now at a6a1a85 0.2.0 CHANGELOG updates

=== src/acceleration/vitis_common (git) ===

HEAD is now at 8805009 Add utilities library

Fails with GitHub's servers. They're probably just blocking it. They have an LFS service so likely this is an encouragement to use it 😄. See https://git-lfs.github.com/.

jasvinderkhurana commented 2 years ago

@vmayoral @giuliocorradi , I think we should find a place to host large files in xilinx servers, to give users a seamless experience.

vmayoral commented 2 years ago

ping @giuliocorradi, we need a resolution on this. Currently, it's not possible to install KRS from the documentation since the acceleration_firmware_kv260 package will fail due to GitHub's restrictions. We need either a server wherein we can host the releases or an alternative service for large files.

vmayoral commented 2 years ago

ping @giuliocorradi and @jasvinderkhurana, do we have a solution for this already in a public server?

jasvinderkhurana commented 2 years ago

@vmayoral, currently no progress, but I am discussing with vivado team, for how to they host big files. I will update as soon as I get to know the complete process

vmayoral commented 2 years ago

Thanks for the update @jasvinderkhurana

jasvinderkhurana commented 2 years ago

@vmayoral , I have got the kv260 and zcu102 uploaded on Xilinx server, below are the links

  1. https://www.xilinx.com/bin/public/openDownload?filename=acceleration_firmware_kv260.zip
  2. https://www.xilinx.com/bin/public/openDownload?filename=acceleration_firmware_zcu102.zip

But currently the download speed from this server is very low. I am checking with concerned team and will update once it is fixed.

vmayoral commented 2 years ago

Thanks @jasvinderkhurana,

I gave it a try:

time wget https://www.xilinx.com/bin/public/openDownload?filename=acceleration_firmware_kv260.zip
--2021-12-07 07:41:11--  https://www.xilinx.com/bin/public/openDownload?filename=acceleration_firmware_kv260.zip
Resolviendo www.xilinx.com (www.xilinx.com)... 184.25.39.84
Conectando con www.xilinx.com (www.xilinx.com)[184.25.39.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Moved Temporarily
Localización: https://xilinx-ax-dl.entitlenow.com/dl/ul/2021/12/03/R210492402/acceleration_firmware_kv260.zip?hash=A5I6v3UuqBArDYcdXB9svg&expires=1638880872&filename=acceleration_firmware_kv260.zip [siguiendo]
--2021-12-07 07:41:12--  https://xilinx-ax-dl.entitlenow.com/dl/ul/2021/12/03/R210492402/acceleration_firmware_kv260.zip?hash=A5I6v3UuqBArDYcdXB9svg&expires=1638880872&filename=acceleration_firmware_kv260.zip
Resolviendo xilinx-ax-dl.entitlenow.com (xilinx-ax-dl.entitlenow.com)... 23.210.44.70
Conectando con xilinx-ax-dl.entitlenow.com (xilinx-ax-dl.entitlenow.com)[23.210.44.70]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 1735682989 (1,6G) [application/zip]
Grabando a: “openDownload?filename=acceleration_firmware_kv260.zip”

openDownload?filename=acceleration_firmware_kv260 100%[===========================================================================================================>]   1,62G  20,2MB/s    en 84s

2021-12-07 07:42:39 (19,8 MB/s) - “openDownload?filename=acceleration_firmware_kv260.zip” guardado [1735682989/1735682989]

real    1m28.246s
user    0m3.200s
sys 0m8.552s

It's not too bad I'd say, but if we can improve it, +1!

jasvinderkhurana commented 2 years ago

@vmayoral , yes it is better now. But earlier I was getting 60 KB/s only. So we can go ahead with the changes. I will make the changes in KRS with new links.

vmayoral commented 2 years ago

All right, thanks!

Please paste in here the link to the PR with the changes and let's close with that this ticket.

vmayoral commented 2 years ago

Ping @jasvinderkhurana, can we get the link in here of the firmware for completeness? Thanks!

vmayoral commented 2 years ago

Closing this ticket due to inactivity.

jasvinderkhurana commented 2 years ago

Here are the firmware links. which I provided earlier in the ticket: https://www.xilinx.com/bin/public/openDownload?filename=acceleration_firmware_kv260.zip https://www.xilinx.com/bin/public/openDownload?filename=acceleration_firmware_zcu102.zip

vmayoral commented 2 years ago

Thanks @jasvinderkhurana!