search

Lanchon / openwrt-rt4230w-rev6

OpenWrt installation instructions for Askey RT4230W REV6 / RAC2V1K
Other
9 stars 1 forks source link

Scripts are incompatible with sock firmware version 1.2.3 #1

Open jturn08 opened 1 year ago

jturn08 commented 1 year ago

I followed all the steps in the instructions, but I'm stuck on step #10. When running ./stock4230w ssh '$HOST "cat /proc/version"', terminal displays an error message ash: cat: not found

I'm able to successfully ssh to the router from terminal using ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -oHostKeyAlgorithms=+ssh-rsa technician@192.168.1.1 and run thecat /proc/version command and see Linux version 3.14.77 (jenkins@ci-server)... response, but the stock4230w script doesn't work.

Any ideas what I'm doing wrong?

Lanchon commented 1 year ago

hmmm.... that means ash (the shell) is not finding the cat command. let's debug.

try: ./stock4230w ssh '$HOST' you should have ssh'd into the router with that

try: /stock4230w ssh '$HOST "pwd"' /stock4230w ssh '$HOST pwd'

which platform and shell are you using?

jturn08 commented 1 year ago

Here's what I saw after your suggestions

Running ./stock4230w ssh '$HOST' successfully ssh'd into the router. Terminal displays a working remote shell prompt, although ash: cat: not found error message is displayed even though I didn't run a cat command yet.

BusyBox v1.28.3 () built-in shell (ash)

-ash: cat: not found
technician@rt4230w:~#

Running /stock4230w ssh '$HOST "pwd"' and /stock4230w ssh '$HOST pwd', terminal displays the correct remote response.

Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
/root

Running ./stock4230w ssh '$HOST "cat /proc/version"' still displays ash: cat: not found error. Running ./stock4230w ssh '$HOST sh' <mtd-backup.sh displays ash: sh: not found error.

The RAC2V1K router I'm using has a serial number beginning with A922* and is running stock FW 1.2.3, and I restored the BusyBox config from 1.1.28. I'm running Xubuntu 22.04 and using bash shell based on running echo $SHELL which returns /bin/bash.

Appreciate your help, sorry, I'm not sure what to do next to troubleshoot.

Lanchon commented 1 year ago

hi,

these are the FWs that i've seen: https://github.com/Lanchon/rt4230w-rev6-stock-firmware never seen 1.2.x yet.

locally i'm using an ubuntu jammy base as well.

i'm guessing here... it'd seem that the remote command passed to ssh is being is executed in an incorrectly configured environment, while if shh invokes the shell the env is ok then (because you can cat /proc/version there).

Running ./stock4230w ssh '$HOST' successfully ssh'd into the router. Terminal displays a working remote shell prompt, although ash: cat: not found error message is displayed even though I didn't run a cat command yet.

that could be the login script trying to cat the login welcome message.

ssh, then try which cat, which sh, which rm. EDIT: please take note of the outputs, so i can update my scripts if applicable. let's assume output was /bin/cat, etc.

then try: ./stock4230w ssh '$HOST "/bin/cat /proc/version"'

if it works, continue with the installation but adding full paths to the passed commands.

otherwise you could scp the scripts to /tmp, then ssh, then run them from there.

please tell me how it goes, thanks!

jturn08 commented 1 year ago

Not sure what happened, but my RAC2V1K router broke before I could try running which cat script. The router displays a solid red light after 2-3 seconds after powering on (hardware failure???), can't connect to it by ssh or through 192.168.1.1 web page anymore :(

Thanks for your help though.

Lanchon commented 1 year ago

you can get a TTL/3V usb serial adapter on ebay for maybe $2, then open the router and connect to the serial port. you'll see it booting and you might be able to rescue it.

jturn08 commented 1 year ago

You were right, I was able to rescue it. I was able to successfully flash OpenWrt after disassembling the router, accessing serial console using a USB adapter, and flashing OpenWrt using tftp method.

Can you share instructions for installing dual OS (main & recovery) from a serial console or converting a RAC2V1K with OpenWrt installed using tftp method? Wasn't sure if there was a way to run your scripts from a serial console.

Lanchon commented 1 year ago

https://github.com/Lanchon/openwrt-rt4230w-rev6#install-a-recovery-image-if-you-installed-openwrt-through-any-other-method

jturn08 commented 1 year ago

hi,

these are the FWs that i've seen: https://github.com/Lanchon/rt4230w-rev6-stock-firmware never seen 1.2.x yet.

locally i'm using an ubuntu jammy base as well.

i'm guessing here... it'd seem that the remote command passed to ssh is being is executed in an incorrectly configured environment, while if shh invokes the shell the env is ok then (because you can cat /proc/version there).

Running ./stock4230w ssh '$HOST' successfully ssh'd into the router. Terminal displays a working remote shell prompt, although ash: cat: not found error message is displayed even though I didn't run a cat command yet.

that could be the login script trying to cat the login welcome message.

ssh, then try which cat, which sh, which rm. EDIT: please take note of the outputs, so i can update my scripts if applicable. let's assume output was /bin/cat, etc.

then try: ./stock4230w ssh '$HOST "/bin/cat /proc/version"'

if it works, continue with the installation but adding full paths to the passed commands.

otherwise you could scp the scripts to /tmp, then ssh, then run them from there.

please tell me how it goes, thanks!

I got another RAC2V1K device and had the same issue, so was able to run the which commands you suggested, but still got stuck. Good news is ./stock4230w ssh '$HOST "/bin/cat /proc/version"' worked, but I was not able to get the next step (backup stock firmware) ./stock4230w ssh '$HOST /bin/sh' <mtd-backup.sh to work. Any further suggestions?

Really appreciate your help, I will try your instructions for installing a recovery image on a device with OpenWrt installed through another method soon.

Sharing logs below

jturn08@jturn08-pc:~$ ssh -o KexAlgorithms=diffie-hellman-group1-sha1 -o HostKeyAlgorithms=ssh-rsa -o StrictHostKeyChecking=no technician@192.168.1.1
technician@192.168.1.1's password: 

BusyBox v1.28.3 () built-in shell (ash)

-ash: cat: not found
technician@rt4230w:~# which cat
/bin/cat
technician@rt4230w:~# which sh
/bin/sh
technician@rt4230w:~# which rm
/bin/rm

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w ssh '$HOST "/bin/cat /proc/version"'
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
Linux version 3.14.77 (jenkins@ci-server) (gcc version 5.2.0 (OpenWrt GCC 5.2.0 v1.2.3.2+r49254) ) #1 SMP PREEMPT Mon Aug 31 09:45:14 UTC 2020
jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w ssh '$HOST /bin/sh' <mtd-backup.sh
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
/bin/sh: mkdir: not found
Lanchon commented 1 year ago

well i suppose you could edit my script and add the full paths all throughout or you could add /bin to the path at the start of the script with something like: export PATH=/bin:$PATH

what version is that new fw?

EDIT: i think the version info is here: /etc/system_version.info

jturn08 commented 1 year ago

This device has FW_VERSION="1.2.3"

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w ssh '$HOST "/bin/cat /etc/system_version.info"'
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
FW_VERSION="1.2.3"
FULL_FW_VERSION="1.2.3.00831.2-r116"
BUILD_DATE="2020-08-31 09:01 +0000"

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w ssh '$HOST "/bin/cat /etc/openwrt_release"'
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='Chaos Calmer'
DISTRIB_REVISION='v1.2.3.2+r49254'
DISTRIB_CODENAME='chaos_calmer'
DISTRIB_TARGET='ipq806x/generic'
DISTRIB_DESCRIPTION='OpenWrt Chaos Calmer 15.05.1'
DISTRIB_TAINTS='no-all busybox override'

I was able to successfully run mtd-backup.sh and ubi-backup.sh after adding export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/sbin to the beginning of the scripts like you suggested. Is my backup complete even if I got a few Device or resource busy errors? I'm happy to share the stock backup files for FW 1.2.3 to your repo.

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w ssh '$HOST /bin/sh' < mtd-backup.sh
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.

creating backup files:
backing up mtd0 to 'mtd0-0:SBL1.gz'
backing up mtd1 to 'mtd1-0:MIBIB.gz'
backing up mtd2 to 'mtd2-0:SBL2.gz'
backing up mtd3 to 'mtd3-0:SBL3.gz'
backing up mtd4 to 'mtd4-0:DDRCONFIG.gz'
backing up mtd5 to 'mtd5-0:SSD.gz'
backing up mtd6 to 'mtd6-0:TZ.gz'
backing up mtd7 to 'mtd7-0:RPM.gz'
backing up mtd8 to 'mtd8-0:APPSBL_1.gz'
backing up mtd9 to 'mtd9-0:APPSBLENV.gz'
backing up mtd10 to 'mtd10-0:ART.gz'
backing up mtd11 to 'mtd11-0:BOOTCONFIG.gz'
backing up mtd12 to 'mtd12-0:SBL2_1.gz'
backing up mtd13 to 'mtd13-0:SBL3_1.gz'
backing up mtd14 to 'mtd14-0:DDRCONFIG_1.gz'
backing up mtd15 to 'mtd15-0:SSD_1.gz'
backing up mtd16 to 'mtd16-0:TZ_1.gz'
backing up mtd17 to 'mtd17-0:RPM_1.gz'
backing up mtd18 to 'mtd18-0:BOOTCONFIG1.gz'
backing up mtd19 to 'mtd19-0:APPSBL.gz'
backing up mtd20 to 'mtd20-rootfs_1.gz'
backing up mtd21 to 'mtd21-rootfs.gz'
backing up mtd22 to 'mtd22-ubifs.gz'
backing up mtd23 to 'mtd23-spi32766.0.gz'
backing up mtd24 to 'mtd24-kernel.gz'
backing up mtd25 to 'mtd25-ubi_rootfs.gz'
backing up mtd26 to 'mtd26-rootfs_data.gz'
gzip: can't open '/dev/mtd26': Device or resource busy
 -> mtd26: backup failed
backing up mtd27 to 'mtd27-log.gz'
gzip: can't open '/dev/mtd27': Device or resource busy
 -> mtd27: backup failed
backing up mtd28 to 'mtd28-vendor.gz'
gzip: can't open '/dev/mtd28': Device or resource busy
 -> mtd28: backup failed
backing up mtd29 to 'mtd29-user_data.gz'

tarring backup files to '/tmp/lanchon/mtd-backup.tar'

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w ssh '$HOST /bin/sh' < ubi-backup.sh
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
UBI device number 2, total 1024 LEBs (130023424 bytes, 124.0 MiB), available 0 LEBs (0 bytes), LEB size 126976 bytes (124.0 KiB)
ubiattach: error!: cannot attach mtd21
           error 17 (File exists)

creating backup files:
backing up ubi0_0 to 'ubi0_0-kernel.gz'
backing up ubi0_1 to 'ubi0_1-ubi_rootfs.gz'
backing up ubi0_2 to 'ubi0_2-rootfs_data.gz'
backing up ubi1_0 to 'ubi1_0-log.gz'
backing up ubi1_1 to 'ubi1_1-vendor.gz'
backing up ubi1_2 to 'ubi1_2-user_data.gz'
backing up ubi2_0 to 'ubi2_0-kernel.gz'
backing up ubi2_1 to 'ubi2_1-ubi_rootfs.gz'
backing up ubi2_2 to 'ubi2_2-rootfs_data.gz'

tarring backup files to '/tmp/lanchon/ubi-backup.tar'

ubidetach: error!: cannot detach mtd21
           error 16 (Device or resource busy)
ubidetach: error!: cannot detach mtd20
           error 22 (Invalid argument)

I was not able to use scp to transfer the backup files. I worked around the scp problem by using ssh to run cp command to copy the backup files to a USB drive plugged into the USB port of the RAC2V1K router, but maybe you have suggestions on how to fix the scp script? Otherwise, I'll try copying the recovery.bin from USB drive as well, and run the install-recovery.sh script next.

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w scp '$HOST:/tmp/lanchon/mtd-backup.tar .'
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
ash: scp: not found

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ssh -o KexAlgorithms=diffie-hellman-group1-sha1 -o HostKeyAlgorithms=ssh-rsa -o StrictHostKeyChecking=no technician@192.168.1.1
technician@192.168.1.1's password: 

BusyBox v1.28.3 () built-in shell (ash)

-ash: cat: not found
technician@rt4230w:~# which scp
/usr/bin/scp
technician@rt4230w:~# exit
Connection to 192.168.1.1 closed.
jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ ./stock4230w /usr/bin/scp '$HOST:/tmp/lanchon/mtd-backup.tar .'
Warning: Permanently added '192.168.1.1' (RSA) to the list of known hosts.
ash: scp: not found

I tried manually running scp command but haven't figured out how to fix the ash: scp: not found error message.

jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ scp -o KexAlgorithms=diffie-hellman-group1-sha1 -o HostKeyAlgorithms=ssh-rsa -o StrictHostKeyChecking=no technician@192.168.1.1:/tmp/lanchon/mtd-backup.tar .
technician@192.168.1.1's password: 
ash: scp: not found
jturn08@jturn08-pc:~/github/openwrt-rt4230w/openwrt-rt4230w-rev6$ /usr/bin/scp -o KexAlgorithms=diffie-hellman-group1-sha1 -o HostKeyAlgorithms=ssh-rsa -o StrictHostKeyChecking=no technician@192.168.1.1:/tmp/lanchon/mtd-backup.tar .
technician@192.168.1.1's password: 
ash: scp: not found
Lanchon commented 1 year ago

hey thanks so much, very informative! glad you made progress and i'm sorry i've been too busy to reply earlier.

Is my backup complete even if I got a few Device or resource busy errors?

yes it is. if you check the actual partition table of the large NAND flash you'll see it goes from mtd0 all the way to mtd22 (ubifs). you can clearly see this in the bootlog of your kernel (try dmesg on the router).

the NAND contains the complete system (in the 4230w devices i've seen). the NAND can contain defects and develop new ones with use. defect management on read-only partitions is done by skipping bad blocks. the mtd layers of uboot and linux do this when you read or write a complete partition from start to end without any seeks. the boot rom inside the SoC also does this. this allows booting the device in as many stages as required, and some devices load the linux kernel (stored as an mtd partition) in this way (but not this device). wear-leveling on read-only partitions is not needed as these partitions are written fully and seldomly.

defect management and wear-leveling on the large UBI partitions is handled by the UBI layers of uboot and linux. due to this non-trivial layer, linux userland can use the UBI partitions mostly without regard for the various issues NAND flash implies.

however the hardware designers did not envision the use of a complex layer such as UBI and thus expected the NAND contents to remain mostly constant, only to be modified during a firmware update. they felt the need to provide a way to store variable configuration outside of the NAND, and thus added a small NOR flash. NOR can be written in small chunks, is expected to be free of defects, and has a much larger write endurance than NAND. no defect management or wear-leveling is needed (NOR can be used simply as a non-volatile RAM) but NOR sizes are very small compared to NAND's.

the included NOR flash is connected via a SPI bus to the SoC and is exposed by the kernel as mtd23 (spi32766.0). i think it is 32 KiB in size and goes completely unused on your device. you can check your dump, it should be all 0xFFs.

next comes a series of UBI volumes reflected as mtd partitions. but these volumes actually reside inside their containing UBI partitions. they are already backed up as a unit as part of UBI partitions mtd20, mtd21 and mtd22. they will also be backed up separately later during the ubi-backup.sh run.

during this mtd-backup.sh run though, the UBI volumes mounted read-only (kernel is raw, rootfs is sqshfs) will back up yet one more time, while volumes mounted read/write (which are UBIFS) will fail to lock for reading. you can completely disregard these errors.

phewww that was long... sorry.

I'm happy to share the stock backup files for FW 1.2.3 to your repo.

thanks! i'll take a look when i can.

I was not able to use scp...

yep, all is part of the same thing. on your firmware the sshd environment is misconfigured, and sshd is unable to even find the scp executable to which it needs to delegate. if the environment is fixed, everything should work without changes. (one hopes that the scp binary is actually there... it probably is.) but i can't do it unless i can ssh into your router somehow to investigate and experiment :(.

I'll try copying the recovery.bin from USB drive as well, and run the install-recovery.sh script

that should work!

but if you haven't done it yet, please wait a bit. if you run the script you'll wipe the only platform we have to find a solution to this problem. let me look into you system backup a for a bit. although i can't test anything, maybe i can suggest a workaround for you to try.

thanks!

Lanchon commented 1 year ago

i looked at your firmware and unfortunately could not find any obvious issues with the sshd (dropbear) configuration.

jturn08 commented 1 year ago

Appreciate the detailed explanation of how the different backup scripts work with MTD and UBI partitions.

yep, all is part of the same thing. on your firmware the sshd environment is misconfigured, and sshd is unable to even find the scp executable to which it needs to delegate. if the environment is fixed, everything should work without changes. (one hopes that the scp binary is actually there... it probably is.) but i can't do it unless i can ssh into your router somehow to investigate and experiment :(.

I ssh'd into the router and ran which scp and got /usr/bin/scp as the result, so looks like scp is installed.

i looked at your firmware and unfortunately could not find any obvious issues with the sshd (dropbear) configuration.

If I ssh into the router using interactive shell, the $PATH looks correct and looks like it's set in in /etc/profile. I suspect dropbear may not be using /etc/profile if using a non-interactive shell based on this discussion.

I also tried debugging if $PATH is defined incorrectly. I ran ./stock4230w ssh '$HOST "/bin/echo $PATH"' to try to discover the default $PATH when using sshpass in the stock4203w script, but got stuck because the output was the local PC $PATH, not the router's $PATH.

technician@rt4230w:~# env
USER=technician
SSH_CLIENT=192.168.1.2 37786 22
SHLVL=1
HOME=/root
SSH_TTY=/dev/pts/0
PS1=${USER}@\h:\w\$ 
LOGNAME=technician
TERM=xterm-256color
PATH=/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/ash
PWD=/root
SSH_CONNECTION=192.168.1.2 37786 192.168.1.1 22

technician@rt4230w:~# cat /etc/profile
#!/bin/sh
[ -f /etc/banner ] && cat /etc/banner
[ -e /tmp/.failsafe ] && cat /etc/banner.failsafe

export PATH=/usr/sbin:/usr/bin:/sbin:/bin
export HOME=$(grep -e "^${USER:-root}:" /etc/passwd | cut -d ":" -f 6)
export HOME=${HOME:-/root}
export PS1='${USER}@\h:\w\$ '
[ -f /tmp/BTHOST_BD_ADDR ] && export BTHOST_BD_ADDR=$(cat /tmp/BTHOST_BD_ADDR)
[ -f /tmp/BTHOST_XCAL_TRIM ] && export BTHOST_XCAL_TRIM=$(cat /tmp/BTHOST_XCAL_TRIM)

[ -x /bin/more ] || alias more=less
[ -x /usr/bin/vim ] && alias vi=vim || alias vim=vi

[ -z "$KSH_VERSION" -o \! -s /etc/mkshrc ] || . /etc/mkshrc

[ -x /usr/bin/arp ] || arp() { cat /proc/net/arp; }
[ -x /usr/bin/ldd ] || ldd() { LD_TRACE_LOADED_OBJECTS=1 $*; }

[ -n "$FAILSAFE" ] || {
    for FILE in /etc/profile.d/*.sh; do
        [ -e "$FILE" ] && . "$FILE"
    done
    unset FILE
}

but if you haven't done it yet, please wait a bit.

I can wait, so I haven't installed OpenWrt yet. Let me know if you've got any other ideas.

Lanchon commented 1 year ago

I ran ./stock4230w ssh '$HOST "/bin/echo $PATH"' [...] got stuck because the output was the local PC $PATH

my script uses eval to expand $HOST. try \$PATH or \\\$PATH or ssh without using ./stock4230w.

dropbear may not be using /etc/config

(you mean /etc/profile.)

the thing is, it works on the previous FW version and i didn't know what's changed.

dropbear seems to have the PATH hardwired (yuck): https://github.com/mkj/dropbear/blob/9defeb477aebf0eb575885eb1fd4a4330ce52531/default_options.h#L353-L355 and should contain /usr/bin and /bin. could it be a misconfigured binary?

I can wait

naaah, go ahead and run the install script using a pendrive to transport the files.

peparadis commented 1 year ago

Hey there, WORKAROUND below

First of all, THANKS for your work. Much appreciated.

Can confirm ALL of jturn08's observations, I have a RAC2V1K with firmware v1.2.4

Also, the SSH hack config file for v1.1.28 is functional.

The problem seems to be that in these more recent firmwares, dropbear (or something ...) does not set the PATH env variable (it is empty, therefor NONE of the commands are 'found').

I did a quick "sanity" check / browsed some changelogs and found that some months back, devs changed the PATH assignment behavior to address root vs non-root users .... Unfortunately I do not have the time to dig any further. I do think your script could be modified to parse the CLI arguments and insert the forced PATH in the proper position, OR figure out how to tell dropbear to export it.

The workaround that let me successfully finish the flashing of current openwrt 22.03.3, was to prepend some of the commands with the missing PATH (the env PATH vs the utility's absolute path), ie: ./stock4230w ssh '$HOST PATH=/usr/sbin:/usr/bin:/sbin:/bin sh' <mtd-backup.sh

Then manually executing the copy/delete of the files, in an SSH session (using the credentials for v1.1.28 hack) directly into the router.

Then manually again, copy the openwrt-rt4230w-rev6-main script and downloaded openwrt recovery.bin to /tmp : scp -r user@192.168.1.14:openwrt-rt4230w-rev6-main/ /tmp/ scp user@192.168.1.14:recovery.bin /tmp/

And finally, to start the install, run: cd /tmp ; sh openwrt-rt4230w-rev6-main/install-recovery.sh

Then finish with the sysupgrade ...

Nice hardware, absolutely great with openWRT.

Thanks again.