Closed villu164 closed 4 months ago
I've made an issue on the Packetry repo for the drop-down menu failure which I was able to reproduce on an M1 mac: https://github.com/greatscottgadgets/packetry/issues/129
I have not been able to reproduce the problem with packetry --test-cynthion
.
Is the packetry --test-cynthion
failure repeatable? Can you try it with a --release
Cargo build?
cargo build --release
cargo run --release -- --test-cynthion
Finished `release` profile [optimized] target(s) in 0.48s
Running `target/release/packetry --test-cynthion`
Testing at High (480Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: High (480Mbps)
Starting read from test device
Read 4096 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 123/4096 bytes of data read from test device
thread 'main' panicked at src/test_cynthion.rs:127:5:
assertion `left == right` failed: Not all data was captured
left: 123
right: 4096
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Also tried modifying the order to reversed
pub fn run_test(save_captures: bool) {
for (name, speed, ep_addr, length, sof) in [
("FS", Speed::Full, 0x82, 512, Some((995000, 1005000, 50))),
("LS", Speed::Low, 0x83, 64, None),
("HS", Speed::High, 0x81, 4096, Some((124500, 125500, 500)))]
{
test(save_captures, name, speed, ep_addr, length, sof).unwrap();
}
}
And got a bit other info too
cargo run -- --test-cynthion
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.47s
Running `target/debug/packetry --test-cynthion`
Testing at Full (12Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: Full (12Mbps)
Starting read from test device
Read 512 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 512/512 bytes of data read from test device
Checking SOF timestamp intervals
Found 1833 SOF packets with expected interval range
Testing at Low (1.5Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: Low (1.5Mbps)
Starting read from test device
Read 64 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 64/64 bytes of data read from test device
Testing at High (480Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: High (480Mbps)
Starting read from test device
Read 4096 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 506/4096 bytes of data read from test device
thread 'main' panicked at src/test_cynthion.rs:127:5:
assertion `left == right` failed: Not all data was captured
left: 506
right: 4096
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Have you run cynthion update
?
I did run at some point, yes
Did another update, first update failed with something. Then second succeeded, and ran --self-test again. Fails with same issue
(.venv) ➜ luna git:(main) ✗
(.venv) ➜ luna git:(main) ✗ cynthion info --force-offline
Detected a Cynthion device!
Hardware: Cynthion r1.4
Serial number: XO4I6UCMGJJVCIBAEA3DOPIO74
Firmware version: v1.0.6
USB API version: 1.1
Flash UID: 0000000000000000
(.venv) ➜ luna git:(main) ✗ cynthion update
Updating FPGA configuration flash with 253603 bytes...
Operation complete!
Traceback (most recent call last):
File "/Users/villuorav/git/luna/.venv/lib/python3.8/site-packages/fwup/dfu.py", line 139, in __init__
self.device = devices[index]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/villuorav/git/luna/.venv/bin/cynthion", line 8, in <module>
sys.exit(main())
File "/Users/villuorav/git/luna/.venv/lib/python3.8/site-packages/cynthion/commands/cli.py", line 112, in main
args.func(device, args)
File "/Users/villuorav/git/luna/.venv/lib/python3.8/site-packages/cynthion/commands/cynthion_update.py", line 21, in cynthion_update
flash_mcu_firmware(device, find_cynthion_asset("apollo.bin"))
File "/Users/villuorav/git/luna/.venv/lib/python3.8/site-packages/cynthion/commands/util.py", line 127, in flash_mcu_firmware
board = DFUTarget(idVendor=shared.usb.bVendorId.apollo, idProduct=shared.usb.bProductId.apollo)
File "/Users/villuorav/git/luna/.venv/lib/python3.8/site-packages/fwup/dfu.py", line 141, in __init__
raise BoardNotFoundError()
fwup.errors.BoardNotFoundError
(.venv) ➜ luna git:(main) ✗ pip install --upgrade cynthion
Requirement already satisfied: cynthion in ./.venv/lib/python3.8/site-packages (0.1.2)
Requirement already satisfied: amaranth==0.4.1 in ./.venv/lib/python3.8/site-packages (from cynthion) (0.4.1)
Requirement already satisfied: luna-soc~=0.2 in ./.venv/lib/python3.8/site-packages (from cynthion) (0.2.0)
Requirement already satisfied: pygreat~=2024.0 in ./.venv/lib/python3.8/site-packages (from cynthion) (2024.0.1)
Requirement already satisfied: pyserial>=3.5 in ./.venv/lib/python3.8/site-packages (from cynthion) (3.5)
Requirement already satisfied: importlib-resources in ./.venv/lib/python3.8/site-packages (from cynthion) (6.4.0)
Requirement already satisfied: luna-usb~=0.1 in ./.venv/lib/python3.8/site-packages (from cynthion) (0.1.1)
Requirement already satisfied: tomli in ./.venv/lib/python3.8/site-packages (from cynthion) (2.0.1)
Requirement already satisfied: future in ./.venv/lib/python3.8/site-packages (from cynthion) (1.0.0)
Requirement already satisfied: libusb1 in ./.venv/lib/python3.8/site-packages (from cynthion) (3.1.0)
Requirement already satisfied: tabulate in ./.venv/lib/python3.8/site-packages (from cynthion) (0.9.0)
Requirement already satisfied: prompt-toolkit in ./.venv/lib/python3.8/site-packages (from cynthion) (3.0.47)
Requirement already satisfied: pyfwup~=0.4 in ./.venv/lib/python3.8/site-packages (from cynthion) (0.4.0)
Requirement already satisfied: tqdm in ./.venv/lib/python3.8/site-packages (from cynthion) (4.66.4)
Requirement already satisfied: apollo-fpga~=1.0.7 in ./.venv/lib/python3.8/site-packages (from cynthion) (1.0.7)
Requirement already satisfied: pyusb in ./.venv/lib/python3.8/site-packages (from cynthion) (1.2.1)
Requirement already satisfied: pyvcd<0.5,>=0.2.2 in ./.venv/lib/python3.8/site-packages (from amaranth==0.4.1->cynthion) (0.4.0)
Requirement already satisfied: Jinja2~=3.0 in ./.venv/lib/python3.8/site-packages (from amaranth==0.4.1->cynthion) (3.1.4)
Requirement already satisfied: pyxdg>=0.27 in ./.venv/lib/python3.8/site-packages (from apollo-fpga~=1.0.7->cynthion) (0.28)
Requirement already satisfied: usb-protocol~=0.9.1 in ./.venv/lib/python3.8/site-packages (from luna-usb~=0.1->cynthion) (0.9.1)
Requirement already satisfied: wcwidth in ./.venv/lib/python3.8/site-packages (from prompt-toolkit->cynthion) (0.2.13)
Requirement already satisfied: backports.functools-lru-cache in ./.venv/lib/python3.8/site-packages (from pygreat~=2024.0->cynthion) (2.0.0)
Requirement already satisfied: zipp>=3.1.0 in ./.venv/lib/python3.8/site-packages (from importlib-resources->cynthion) (3.19.2)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.8/site-packages (from Jinja2~=3.0->amaranth==0.4.1->cynthion) (2.1.5)
Requirement already satisfied: construct~=2.10 in ./.venv/lib/python3.8/site-packages (from usb-protocol~=0.9.1->luna-usb~=0.1->cynthion) (2.10.70)
[notice] A new release of pip is available: 23.0.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(.venv) ➜ luna git:(main) ✗ cynthion update
Updating FPGA configuration flash with 253603 bytes...
Operation complete!
Updating device firmware with 13816 bytes...
Operation complete!
(.venv) ➜ luna git:(main) ✗ cd ..
(.venv) ➜ git cd packetry
(.venv) ➜ packetry git:(main) ✗ cargo run -- --test-cynthion
Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.03s
Running `target/debug/packetry --test-cynthion`
Testing at Full (12Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: Full (12Mbps)
thread '<unnamed>' panicked at src/test_cynthion.rs:66:63:
called `Result::unwrap()` on an `Err` value: Failure in capture thread
Caused by:
0: Sender was dropped
1: oneshot canceled
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/test_cynthion.rs:27:64:
called `Result::unwrap()` on an `Err` value: Test device not found
(.venv) ➜ packetry git:(main) ✗ cynthion run analyzer
Uploading target bitstream to FPGA with 253603 bytes...
Operation complete!
(.venv) ➜ packetry git:(main) ✗ cargo run -- --test-cynthion
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.78s
Running `target/debug/packetry --test-cynthion`
Testing at Full (12Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: Full (12Mbps)
thread '<unnamed>' panicked at src/test_cynthion.rs:66:63:
called `Result::unwrap()` on an `Err` value: Failure in capture thread
Caused by:
0: Sender was dropped
1: oneshot canceled
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/test_cynthion.rs:27:64:
called `Result::unwrap()` on an `Err` value: Test device not found
(.venv) ➜ packetry git:(main) ✗ echo "Forgot to setup to connections, sorry"
Forgot to setup to connections, sorry
(.venv) ➜ packetry git:(main) ✗ cargo run -- --test-cynthion
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.77s
Running `target/debug/packetry --test-cynthion`
Testing at Full (12Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: Full (12Mbps)
Starting read from test device
Read 512 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 512/512 bytes of data read from test device
Checking SOF timestamp intervals
Found 4858 SOF packets with expected interval range
Testing at Low (1.5Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: Low (1.5Mbps)
Starting read from test device
Read 64 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 64/64 bytes of data read from test device
Testing at High (480Mbps):
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: High (480Mbps)
Starting read from test device
Read 4096 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 156/4096 bytes of data read from test device
thread 'main' panicked at src/test_cynthion.rs:127:5:
assertion `left == right` failed: Not all data was captured
left: 156
right: 4096
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Can you try:
cargo run -- --test-cynthion --save-captures
...and then post the resulting HITL-HS.pcap
file here? (You may have to put it in a zip file to satisfy Github upload rules)
This is very weird.
I don't think the test device endpoint should ever be sending less than a full length data packet (512 bytes); it's being fed by a counter which is always ready.
And the first few bytes of each captured packet are as you'd expect them to be: (00, 01, 02...), which is as expected if the packet length is a multiple of 256.
But after the first 15 bytes, that pattern breaks down.
Something really weird is going on.
I think I figured it out... THE CHEAP USB CABLE, tried another cable...
Opening analyzer device
Disabling test device
Enabling test device
Capture enabled, speed: High (480Mbps)
Starting read from test device
Read 4096 bytes from test device
Requesting capture stop
Capture disabled
Found test device in capture
Captured 4096/4096 bytes of data read from test device
Checking SOF timestamp intervals
Found 14535 SOF packets with expected interval range
Sorry for the troubles...
I wonder if there could be some "test" for the cable quality
🤦♂️
I'm glad you found the solution, but I'd like to make sure we understand this failure mode.
Even with the bad cable, the host was able to read from the test device successfully, and also read from the analyzer successfully. It's only the capture data that was wrong.
So it seems like with the bad cable, the signal integrity was sufficient for host-device communication, but not good enough for the target PHY to capture the signal accurately.
I wonder if there could be some "test" for the cable quality
Seems like the existing test served that purpose in this case.
I have run into similar issues with logic analyzers and marginal cables that worked fine until one really pushed traffic that saturated the bus into them. The lesson; for ones instruments and signal capture do not tolerate bad cables.
@villu164 In my experience, the biggest factor in USB 2.0 cable quality for High Speed signal integrity is shielding. If you have a continuity tester, I would love to know if there is continuity from the shield on one end to the shield on the other end of your bad cable.
Assuming, I understood you correctly, then bad cable - no continuity good cables - continuity
Bad Cable (self tested - 12 Mbps, 1.5 Mbps) - 480 Mbps fails (see above)
Good Cable (self tested - 480 Mbps, 12 Mbps, 1.5 Mbps)
Another Good Cable (self tested - 480 Mbps, 12 Mbps, 1.5 Mbps)
In my defense - I use the cables mostly for charging devices. Where-ever I can, then I move data over SSD-s(which have their own cables) or wirelessly. So I have not had issues (maybe luck?) with them yet, (enthusiast, not a hardware hacker yet)
Nice! Thank for checking that. Yeah, poor cables often work okay until you do something to push them to their limits like running the Packetry self-test or a HackRF at maximum sample rate. This self-test is pretty much the worst case scenario for High Speed USB signal integrity and EMC, so it requires good cables.
Thank you for helping us test these problems! That pcap was quite useful. I'm closing this as the Packetry drop-down issue is tracked separately.
@villu164 thanks again for your efforts to track down the issue here.
I've improved Packetry's handling of malformed packets in https://github.com/greatscottgadgets/packetry/pull/138, which should make it easier to diagnose this issue, when other folks have the same problem in future.
I've added your capture as a test case, to make sure this continues to be handled correctly in future.
I'm suspecting some noob(rust, usb) thing going on here, sorry
On an M2 Pro (13.6.7 Ventura)
Detected a Cynthion device! Bitstream: USB Analyzer (Cynthion Project) Hardware: Cynthion r1.4 Flash UID: 39411ccba3ac60de`
Connected the self-test setup Target C and Control connected to M2 Target A connected to AUX in M2, allowed the "Test USB Device", but it would seem that the test fails
Some things that work Facedancer rubber ducky, was able to issue text typing USB Thumbdrive, was able to view activity with packetry and analyzer
Some more symptoms
I have some issues with the packetry GUI, that doesn't allow changing the speed from 480 to anything else opening the dropdown, generates these lines
https://github.com/user-attachments/assets/fbf4c2b4-f534-4e2f-b332-40ab5c36dda5
Could not get Facedancer usbproxy to work at all (including the thumbdrive, that I saw packets from) (Haven't debugged much further)
Not getting any packets from Full Speed (12 Mbps) Wireless Mouse (even after making a lot of traffic)
dmesg after connecting mouse, everything that included USB