search

tio / tio

A serial device I/O tool
https://tio.github.io
Other
2.03k stars 155 forks source link

XMODEM and YMODEM don't seem to work. #213

Closed Zorbatron closed 7 months ago

Zorbatron commented 1 year ago

I'm running Tio version: v2.7, installed from the non-git package on the AUR using YAY. At the time of writing this I have the latest kernel for Arch Linux (6.5.8-arch1-1) and the rest of my packages are up to date.

The device I'm trying to send data to is a Cisco c3750x switch. These are the steps I took:

  1. Ran tio --baudrate 9600 --flow hard /dev/serial/by-id/usb-Parallax_Inc_USBto232_REV_C_DN05JEFM-if00-port0
  2. copy xmodem: flash:/testfile and hit enter at Destination filename [testfile]?
  3. Hit ctrl+t x
  4. Type /tmp/testfile

The next line after Press any key to abort transfer shows this: C.|||||||||||||||||||||| with each vertical line being printed about once a second. I can see the TX and RX LEDs blink each time a vertical line is printed too. Without pressing any key, Tio reports Aborted and checking dir flash: shows that there's no testfile. The testfile's content was just Hello World! but trying with a larger file such as one generated by dd if=/dev/random of=./testfile bs=1k count=16 is the same. With the larger file the USB serial port's tx/rx LEDs flashed and it from the output it seemed like it was successfully transferring, but it failed at the end.

If I run tio with the listen socket option and the run lrzsz seperately, then it works. socat unix:tio.sock exec:'lrzsz-sx /tmp/testfile',pty,ctty and I can see testfile in the switch's flash card. SecureCRT also transfers the file fine.

Just to make sure that there wasn't anything wrong with my switch, I got another USB serial port and ran a getty on it with systemctl start getty@ttyUSB1 connected it to the other USB serial port with a null modem cable. Running lrzsz-rx and trying to use Tio's xmodem transfer resulted in nothing happening and Tio eventually reporting Aborted. I can see the RX LED light up every few seconds too. If I start lrzsz-rx without starting xmodem from Tio, I get this text when the RX LED flashes: **B0100000023be50.

I'm not sure if I'm not using Tio's XMODEM mode correctly, but I can't seem to get it to work. The YMODEM mode also behaves similarly.

lundmar commented 1 year ago

@Zorbatron Thanks for reporting the issue.

I see that you are testing lrzsz which I think defaults to a block size of 128. Our newly added xymodem implementation defaults to a block size of 1024. Perhaps this is the reason it fails?

Zorbatron commented 1 year ago

Does lrzsz have an option to change the block size? I can't seem to find one. I don't think that's the issue because the switch says Begin the Xmodem or Xmodem-1K transfer now... which seems like it used a 1k block size. But you never know...

lundmar commented 1 year ago

According to the man page there is a '-k' option to use 1024 byte blocks.

Zorbatron commented 1 year ago

Okay, so it seems that -k is only for YMODEM mode. Also, even though there's three commands, sx, sb, and sz for sending and rx, rb, and rz for receiving (all prefixed with lrzsz-). They *should* correspond to XMODEM, YMODEM, and ZMODEM. I'm not sure why, but they actually all have the same sha256 hash, meaning that they're the same binary, just named differently.

gabri@PortPC: ~$ sha256sum $(which lrzsz-rx) $(which lrzsz-rb) $(which lrzsz-rz)
ca7c948ee670ba40f8dd42c386dbf277db8e9f61a803edd957d2c6e2a704fab9  /sbin/lrzsz-rx
ca7c948ee670ba40f8dd42c386dbf277db8e9f61a803edd957d2c6e2a704fab9  /sbin/lrzsz-rb
ca7c948ee670ba40f8dd42c386dbf277db8e9f61a803edd957d2c6e2a704fab9  /sbin/lrzsz-rz

So when I was running lrzsz-rx and assuming that it would be XMODEM, I was wrong. All three default to ZMODEM! You have to specify which mode you'd like them to be in. Running any of the three receive commands with -X works similar to how the switch did with XMODEM, by sending a C which I assume is the character to initiate communication and Tio does seem to start to send data. Upon closed inspection of the man page, -k is only for sending in 1k blocks in XMODEM mode and it doesn't work for any of the receiving commands.

Any of the three commands with --ymodem and YMODEM mode with tio doesn't seem to work. At first is seems like it's transferring data, and then it hangs and eventually times out. When in XMODEM mode -X, you have to specify a destination file name. When I do something like lrzsz-rx -X testfile, I get lrzsz-rz: ready to receive testfile but also sh: line 1: zsz-rz: command not found and when initiating a transfer, nothing happens. Putting Tio in hex mode I see the occasional 0x15 but else.

lundmar commented 1 year ago

I believe that @pnrhub tested the xymodem (#208) feature with lrzsz suite (rx/ex etc.). Maybe he used some specific options. I'm not sure.

pnrhub commented 1 year ago

I did test with lrzsz, so we need to find out what the difference is. I am not sure I fully understand the test scenario that is not working. From what I understand of the original cisco scenario, it would seem that the cisco router is not ending the exchange with a final ACK: https://github.com/tio/tio/blob/master/src/xymodem.c#L152-L171

Maybe you could run a test with a modified tio that accepts the transmission if it did not get the ACK within five tries of sending EOT?

lundmar commented 1 year ago

There is a guy here trying to do an xmodem transfer via teraterm with a Cisco switch in the same switch family. https://community.cisco.com/t5/networking-blogs/loading-an-ios-on-a-switch-via-xmodem/ba-p/3103557

Looks like xmodem with crc and I think teraterm defaults to a block size of 128 bytes: https://ttssh2.osdn.jp/manual/4/en/macro/command/xmodemsend.html

lundmar commented 1 year ago

@Zorbatron At what baud rate did you successfully transfer using lrzsz-sx/tio ? Was there any retransmission attempts during the transfer?

Zorbatron commented 1 year ago

I wasn't able to get lrzsz to work. I think there's something wrong with Arch Linux's version of it because when I run lrzsz-rx -X {filename}, it says sh: line 1: zsz-rz: command not found. Doing the same but with lrzsz-rb --ymodem and Tio's ymodem mode doesn't work either. Compiling https://github.com/UweOhse/lrzsz from source gives a working version where xmodem receive works and running socat file:/dev/ttyUSB0,b9600,raw,echo=0 exec:'lrzsz-rx -X {filename},pty,ctty' works perfectly with checksums matching. Tio also still didn't work with the newer version, x or y modem mode.

Changing speed with stty speed {baudrate} then exiting Tio and changing the baudrate doesn't change the outcome either. I'm not sure what Tio looks for before starting the data transfer, but when I start an xmodem receive, I see hex 0x15 as what I assume the "start sending"/"Initiate transfer" message is.

TLDR: I haven't gotten Tio's x/y modem to work in any situation, regardless of baudrate.

lundmar commented 1 year ago

Just to clarify, so you are saying that tio xmodem transfer fails with Cisco switch (rommon) but you can get it working but only with latest version of the lrzsz-sx suite (v0.13alpha as maintained by UweOhse)?

Zorbatron commented 1 year ago

No. I can't get Tio to work *at all*.

pnrhub commented 1 year ago

I tested the xymodem routines against lrzsz as built by home-brew on MacOS, using a pseudo terminal pair as transport and this worked fine. I noticed that the README for https://github.com/UweOhse/lrzsz says "NOTE: Do not use the repository until further notice. It contains work in progress and at least one change i am unhappy with. Use either http://ohse.de/uwe/testing/lrzsz-0.12.21rc.tar.gz (over 20 years in rc state, so it's good enough), or 0.12.20 from https://ohse.de/uwe/software/lrzsz.html"

I will repeat the tests on Debian using 0.12.20. and see if I get the same results.

pnrhub commented 1 year ago

By the way, in more detail my test is as follows:

(a) in an xterm run "screen /dev/ttys0" (b) in a second xterm run "tio /dev/ptys0" (c) type a few characters in screen and in tio to make sure the connection works as expected (d) in screen, type Ctrl-A : to enter a command filter; enter the command exec !! lrz -X -c test

This will print "lrz: ready to receive test" in the screen window, and "C" will appear in tio once per second. Notes: "-X" means use xmodem and "-c" means use a CRC instead of a simple checksum (it is also the difference between seeing "C" instead of 0x15 NAK)

(e) now in tio, send a test file by typing Ctrl-T x, and entering the filename.

The file will now be sent and both screen and tio revert back to normal terminal mode.

For ymodem it is almost the same, but the command in screen is exec !! lrz --ymodem Note that for ymodem the destination file must not already exist.

pnrhub commented 1 year ago

Just did the test on Debian Bookworm. It works there too. The test is slightly different on Linux, as it uses Unix98 pseudo terminals instead of BSD-style pseudo terminals and it would seem that neither screen nor tio can open a Unix98 pseudo tty slave device. Luckily, socat comes to the rescue:

(i) open an xterm, and run socat pty,rawer,echo=0,link=/tmp/portA pty,rawer,echo=0,link=/tmp/portB

This creates two symlinks, "/tmp/portA" and "/tmp/portB" that together behave like a BSD-style pseudo terminal pair. Modify step (a) and (b) above into:

(a) in a second xterm run "screen /tmp/portB" (b) in a third xterm run "tio /tmp/portA"

Then proceed as per the above. For me, results are the same. @Zorbatron: can you replicate these results on Arch? Or does it give different results there?

For avoidance of doubt: I'm using lrzsz 0.12.20 as downloaded from https://ohse.de/uwe/software/lrzsz.html and tio using today's head.

Zorbatron commented 1 year ago

I can't test right now, but you said that you used the -c option for CRC in lrzsz. In all of my tests I didn't use that and the Cisco switch probably doesn't do that either. I'll try it out when I can.

pnrhub commented 1 year ago

I'll try it out when I can.

@Zorbatron That is fine. Maybe we should also focus on the original issue: the Cisco switch. I don't have access to one to run tests with, but carefully reading your initial report gives some clues.

(1) I do think that the Cisco switch expects CRC checks, that is what the "C" prompt signifies -- otherwise it would have sent a series of NAK's (0x15). You don't have to provide CRC-mode as a command line switch to lsz because it can see what is expected from the prompt character.

(2) You use hardware flow control. I don't think this is the core issue, but it is worth checking. I noticed on this page about uploading to a Cisco 3750 that Xon/Xoff was used instead.

(3) It seems that most of the tio xmodem transfer actually works. Each dot after the C signifies a 1KB block having been sent to the switch and the switch acknowledging correct receipt. It fails at the end sequence, which are the bars.

My understanding of the Xmodem protocol is that a transmission is terminated by sending a series of up to 10 "EOT" characters. If the receiver sees two in a row (instead of the next packet) it must respond with an "ACK" character (or a "CAN" character if the file transfer failed for some reason). The tio xmodem code prints a bar for each EOT sent out.

You write that you see both the RX and TX leds flashing during this phase, i.e. tio sends out the bar chars and the switch responds with something, but apparently it is not "ACK" nor "CAN".

After 22 seconds (I counted 22 bars in your message) the switch apparently gives up and sends a "CAN" character after all, which causes tio to terminate the transmission with an abort condition.

(4) I've posted a gist with some test code to see what the switch is actually responding with if it is not "ACK" or "CAN". The gist can be found here: https://gist.github.com/pnrhub/9a99207da29b145984bb33b3248012f2

It would also be helpful to repeat the tio socket test with the 0.12.20 version of lrzsz as per the below: socat unix:tio.sock exec:'lsz -X -k /tmp/testfile',pty,ctty If this works as it should, at least I have a defined bit of sender source code to work with.

Zorbatron commented 1 year ago

Okay, I've rebuilt Tio with the gist's code. These are the steps I took. Cloned Tio from master. replaced src/xymodem.c with the code from the gist. Built but not installed and ran it from build/src/tio. Repeating the same test on the secondary PC, there was no change. I saw the RX LED flash but nothing happened on PC1 and it eventually timed out. Am I supposed to use the code in a different way, maybe standalone instead of integrated into Tio? Just to make sure we're on the same page, this is the topology:

 PC1 --- USB to serial --- null modem cable --- USB to serial --- PC2
Sender                                                          Receiver

For receiving, getty@ttyUSB0 is running for the serial terminal. I have to use lrzsz version 0.13.0-alpha because version 0.12.20's xmodem receive mode reports this error: sh: line 1: zsz-rx: command not found. 0.12.20 also fails and spews the content of the sending file all over the terminal. I assume that Tio doesn't receive a proper stop signal so it keeps going after lrzsz ends.

The next test used Tio in socket listening mode as to use lrzsz to send instead.

pnrhub commented 1 year ago

Hi, thanks for that feedback! Unfortunately, I am all confused now:

1) We have two different test scenarios now, which makes it difficult to replicate the issue. I will try your setup with two PC's and two usb-serials connected via a null modem connection. Could you in parallel try the test with a pseudo-tty pair as described above?

2) Could you describe your test in more detail? In particular the error report sh: line 1: zsz-rx: command not found is difficult to place. In my build of lrzsz 0.12.20 from the canonical source, I get only two binaries lsz and lrz. Is this different in your build of the same? Are you sure the freshly build lsz and lrz get used (i.e. no other version earlier in the search path)?

3) The gist was intended specifically to test the feedback from the cisco switch. Did you try tio with the gist patch on the cisco and what happened? It should have stopped after 10 bar chars, but after each bar chart there should have been a hexadecimal number (one byte) showing what response came back from the cisco. Is that what happened?

Zorbatron commented 1 year ago
  1. Looking back at you comment, I can't get it to work. socat pty,rawer,echo=0,link=/tmp/portA pty,rawer,echo=0,link=/tmp/portB puts the symlinks into /tmp and I can confirm that it transfers data between the two pseudo ttys. But screen /tmp/portB gives Cannot exec '/tmp/portB': Permission denied. So I'll ignore GNU screen and use socat instead. Modifying your command to socat pty,raw,echo=0,link=/tmp/portA exec:'lrz -X /tmp/archinstall.tar',pty,ctty gives a pty that lrz is connected to. tio /tmp/portA and trying to send a file yields no results and lrz eventually time out with this message: 
    Blocks received: -1
lrz: /tmp/archinstall.tar removed.

    If I instead run socat file:/tmp/portA,raw,echo=0 exec:'lsz -X /home/gabri/archinstall.tar',pty,ctty as the sender, it works correctly.

  2. I'm using Arch Linux's package of lrzsz, which must be built differently so I'll uninstall that. I downloaded the source for 0.12.20 from the same place as you and use that instead. After it built I ran sudo make install and I can confirm that the correct binaries are on my PATH now.
  3. I forgot about that. This is the output of the modified Tio binary attempting to transfer with the Cisco switch (Timestamp enabled so I don't have to guess how long it took): 
    
[07:32:10.286] Switch#copy xmodem: flash:/archinstall.tar
[07:32:13.567] Destination filename [archinstall.tar]? 
[07:32:15.312] Begin the Xmodem or Xmodem-1K transfer now...
[07:32:15.360] C
[07:32:15.801] Send file with XMODEM
[07:32:17.369] Sending file '/home/gabri/archinstall.tar'  
[07:32:17.369] Press any key to abort transfer
[07:32:47.534] Done..|||0x15|0x15|0x15|0x15|0x15|0x15|0x15|0x15

[07:32:53.059] %Error reading xmodem: (I/O error) [07:32:53.315] Switch#dir flash: [07:32:58.149] Directory of flash:/

[07:32:58.181] 2 -rwx 676 Jan 2 2006 00:10:30 +00:00 vlan.dat [07:32:58.261] 3 -rwx 4120 Jan 2 2006 00:24:12 +00:00 multiple-fs [07:32:58.325] 4 drwx 512 Mar 1 1993 01:21:49 +00:00 c3750e-universalk9-mz.152-4.E10

[07:32:58.437] 57671680 bytes total (26700288 bytes free) [07:32:58.485] Switch#


`archinstall.tar` is just a tarball of some json files. The `Done..` starts on the same line as the debug output of the xmodem status, but it looked like this before being covered: `C....................|||0x15|0x15|0x15|0x15|0x15|0x15|0x15|0x15`
During the periods being printed and the first three `|`s, the TX LED was on. After that, the TX and RX LEDs would flash in this pattern: RX LED flashes, 0.5s, TX and RX LEDs flash and `|0x15` is printed (presumably the | and TX flash and the RX + 0x15 follows too fast for me too see the delay), another 0.5 seconds and then it repeats. There are 10 `|s` and 8 `0x15`s.
pnrhub commented 1 year ago

Thank you for those tests.

About 1:

Modifying your command to socat pty,raw,echo=0,link=/tmp/portA exec:'lrz -X /tmp/archinstall.tar',pty,ctty gives a pty that lrz is connected to. tio /tmp/portA and trying to send a file yields no results and lrz eventually time out

Could you please retry that using

socat pty,raw,echo=0,link=/tmp/portA exec:'lrz -X -c /tmp/archinstall.tar',pty,ctty

to enable the CRC check? The tio xymodem code does not understand the old simple checksum method and it keeps waiting for a 'C' that never comes.

About 4:

Most helpful test! It would seem that the Cisco is silent after the first two EOT's and then sends a NAK for each following EOT. If I understand the spec correctly, there should have been an ACK response after the second EOT. The later NAK's make sense in a way, because apparently the Cisco expects a new packet and EOT is not a valid packet start.

Maybe I should send two EOT's in quick succesion instead of separated by 1 second -- possibly the Cisco times out before receiving the second EOT and does not recognise it as a valid termination sequence.

===

Update: I've updated the gist to terminate with a double EOT each second, for up to 10 seconds: https://gist.github.com/pnrhub/9a99207da29b145984bb33b3248012f2

Zorbatron commented 1 year ago
  1. Redoing the receive with -c for CRC works and the checksum matches! This is what the transfer status looks like: ....................|| . The archinstall.tar file is 20480 bytes. So I'm not sure how, but I can't replicate the one time it worked. I'm getting this now. lrzsz output: 
    
gabri@Laptop: ~$ socat pty,raw,echo=0,link=/tmp/portA exec:'lrz -X -c /tmp/archinstall.tar',pty,ctty

lrz: ready to receive /tmp/archinstall.tar Retry 0: Got 04 sector header Retry 1: Got 04 sector header Retry 2: Got 04 sector header Retry 3: Got 04 sector header Retry 4: TIMEOUT Retry 5: TIMEOUT Retry 6: TIMEOUT Retry 7: TIMEOUT Retry 8: TIMEOUT Retry 9: TIMEOUT Blocks received: -1 lrz: /tmp/archinstall.tar removed.

Transfer incomplete 2023/11/02 17:36:17 socat[3085] E waitpid(): child 3086 exited with status 128 gabri@Laptop: ~$ 

Tio output: `C...................|||0x15|||0x15|||0x15|`
If I ctrl+c socat so kill lrzsz before it removes the file, the checksum does match. But it must time out and delete it.
2. I'm not sure why, but doing a transfer between two PCs doesn't work. Even though I'm using the proper lrzsz and the modified Tio.

gabri@PortPC: ~/Documents/git/lrzsz-0.12.20/src$ ./lrz -c -X /tmp/archinstall.tar [16:52:12.242] [16:52:12.253] lrz: ready to receive /tmp/archinstall.tar [16:52:12.315] C [16:52:13.517] Send file with XMODEM [16:52:14.446] Sending file '/home/gabri/archinstall.tar'
[16:52:14.446] Press any key to abort transfer C....................|| [16:52:46.527] Done

[16:53:34.858] lrz: /tmp/archinstall.tar removed. gabri@PortPC: ~/Documents/git/lrzsz-0.12.20/src$

3. Rebuilding Tio with the updated gist it seems like it works:

Switch#copy xmodem: flash:/archinstall.tar Destination filename [archinstall.tar]? Begin the Xmodem or Xmodem-1K transfer now... C [16:27:26.178] Send file with XMODEM [16:27:27.482] Sending file '/home/gabri/archinstall.tar'
[16:27:27.482] Press any key to abort transfer C....................|||0x15|0x15| [16:27:52.638] Done

Switch#


But when I check the flash, the `archinstall.tar` file is only 1024 bytes and the check sums obviously didn't match. FTPing the file over and viewing it in a hex editor, I can see that the 1024 bytes that were transferred do match the source file. Interesting to note is that, during the transfer (when the periods were being printed), the RX LED didn't flash at all, which I guess must mean the switch isn't doing CRC.
pnrhub commented 1 year ago

It seems we are closing in on the issue. I will try to replicate your results under 1. and 2.

For 3. the result is encouraging, at least something is getting through. The switch did respond, even if RX did not (visibly) light up: each dot means that an ACK character was received back in response to a block being sent. ACK means that the switch accepted the block, i.e. the CRC was recognised and found to be correct. 20 dots means 20 blocks and that matches expectation (1KB blocks and sending a 20KB file). Why the file as received is then only the first [?] block is still a bit of a mystery.

It may be a couple of days before I have a time slot to run more tests.

eshattow commented 10 months ago

tio is useful for sending SPL to the MaskROM ZSBL on JH7110 CPU ; there's a bug in the XMODEM implementation on JH7110 MaskROM and tio does not seem to hit this bug in its XMODEM implementation. However, I do note that it takes a few attempts from tio before the transfer starts to sync up and when it's not working it's just as described in this issue (vertical bars status indication; cancel and retry until some period dot character indicators show up then it's good to go).

sly74fr commented 10 months ago

This could also be useful for HP48G and so on owners to connect to their favorite toy 😉

HiFiPhile commented 7 months ago

I'm working to add XMODEM-CRC support which is very popular.

I saw resp variable is not cleared before the receiving loop, after 1s timeout if it has ACK from last transfer then it simple continues. https://github.com/tio/tio/blob/83f826349bbce456f471311595479aa58e0101a9/src/xymodem.c#L120

eshattow commented 7 months ago

I'm working to add XMODEM-CRC support which is very popular.

I saw resp variable is not cleared before the receiving loop, after 1s timeout if it has ACK from last transfer then it simple continues.

https://github.com/tio/tio/blob/83f826349bbce456f471311595479aa58e0101a9/src/xymodem.c#L120

tio version 2.8 entered Debian unstable recently so I happened to try this. Either method XMODEM-1K and XMODEM-CRC are successful for transferring secondary program loader (144kb) to the JH7110 CPU MaskROM UART loader.

edit: I do not like that it will insert an additional question into the workflow, but I am not sure what to suggest - should this really be another level of menu, or can it be a new shortcut sequence? I would suggest ctrl-t x for the way it was before the new feature, and ctrl-t X for the new XMODEM variant? Reading the pull request #236 this was already discussed and I did not participate there... nevermind it.

lundmar commented 7 months ago

tio version 2.8 entered Debian unstable recently so I happened to try this. Either method XMODEM-1K and XMODEM-CRC are successful for transferring secondary program loader (144kb) to the JH7110 CPU MaskROM UART loader.

That is good, nothing broken then. 👍🏻

edit: I do not like that it will insert an additional question into the workflow, but I am not sure what to suggest - should this really be another level of menu, or can it be a new shortcut sequence? I would suggest ctrl-t x for the way it was before the new feature, and ctrl-t X for the new XMODEM variant? Reading the pull request #236 this was already discussed and I did not participate there... nevermind it.

I added the x-modem sub menu in an effort to try avoid cluttering the top level commands. The way things are going we will likely see more xmodem commands added such as the original xmodem that uses 128 byte block size. Also, in terms of key presses it is the exact same.

eshattow commented 7 months ago

tio version 2.8 entered Debian unstable recently so I happened to try this. Either method XMODEM-1K and XMODEM-CRC are successful for transferring secondary program loader (144kb) to the JH7110 CPU MaskROM UART loader.

That is good, nothing broken then. 👍🏻

edit: I do not like that it will insert an additional question into the workflow, but I am not sure what to suggest - should this really be another level of menu, or can it be a new shortcut sequence? I would suggest ctrl-t x for the way it was before the new feature, and ctrl-t X for the new XMODEM variant? Reading the pull request #236 this was already discussed and I did not participate there... nevermind it.

I added the x-modem sub menu in an effort to try avoid cluttering the top level commands. The way things are going we will likely see more xmodem commands added such as the original xmodem that uses 128 byte block size. Also, in terms of key presses it is the exact same.

Thanks. I wrote what ideas I could at #243 - maybe it stays as-is but anyone with suggestion how this could be, it is not any problem with the functional implementation of the XMODEM-CRC :-)

lundmar commented 7 months ago

@Zorbatron Maybe you could try the new tio xmodem-crc feature to see if that solves your transfer issue with your cisco switch.

Zorbatron commented 7 months ago

After updating to Tio 2.8, I can confirm that transferring files works with the CRC mode!