fail on R6220 'brick' - "sendto: Operation not permitted"

antipodes5 commented 4 months ago

Hi

I have a Netgear R6220 with an unknown state of installation - its effectively bricked, probably from a bad install of openWRT a few years ago. I am now trying to go back to stock firmware.

It has shown a number of behaviours, but now seems to have a bootloop of random duration:

power light on, number 4 on -> Internet light on, (no power light), number 4 port light -> lights out then up all the lights except the ports -> back to the start.

Setup is: Linux (Lubuntu) box -> switch port 2 AND R6220 -> switch port 1

I can't figure out exactly which part of the cycle is going to happen, but this is the return when I launch the nmrpflash command:

$ sudo ./nmrpflash -i enp0s25 -f R6220-V1.1.0.114_1.0.1.img
Advertising NMRP server on enp0s25 ... \
Received configuration request from b0:39:56:1a:bc:00.
Sending configuration: 10.164.183.253/24.
Received upload request without filename.
Uploading R6220-V1.1.0.114_1.0.1.img ... sendto: Operation not permitted

This has happened many many times. Only once, this happened:

$ sudo ./nmrpflash -i enp0s25 -f R6220-V1.1.0.114_1.0.1.img
Advertising NMRP server on enp0s25 ... \
Received TFTP_UL_REQ while waiting for CONF_REQ!
Received upload request without filename.
Uploading R6220-V1.1.0.114_1.0.1.img ... sendto: Operation not permitted

...but I'm not sure that's important.

I have also changed the filename to the shorter R6220.img and got the same result.

What does the sendto: Operation not permitted mean? (I have no luck in websearches.)

Most importantly, what do I do?

EDIT

See also discussion on OpenWRT forum.

jclehner commented 4 months ago

According to this thread, a firewall could cause the sendto function to fail with Operation not permitted.

Btw, because it was mentioned on the OpenWRT forum thread you referenced: The "Received upload request without filename." message is not an issue.

antipodes5 commented 4 months ago

Thanks for your response.

Please note that ufw was disabled, and yet the above behaviour was still observed. Many times.

Good to know about the "Received upload..." message.

Going for a full reinstallation of this OS, (upgrade), will try again after. Will make brief report here either way.

antipodes5 commented 4 months ago

Tried it again. All from a fresh installation of the OS, new copies of all files. Still getting the same behavior, despite ufw (Uncomplicated FireWall) being set to disable.

Error message is still Operation not permitted. Still unsure what is not permitting what.

jclehner commented 4 months ago

Weird. Which Lubuntu version is it?

Please run the following commands and post the output:

$ sudo ufw disable
$ sudo iptables -L

On Sun, May 12, 2024, 06:44 antipodes5 @.***> wrote:

Tried it again. All from a fresh installation of the OS, new copies of all files. Still getting the same behavior, despite ufw (Uncomplicated FireWall) being set to disable.

Error message is still Operation not permitted. Still unsure what is not permitting what.

— Reply to this email directly, view it on GitHub https://github.com/jclehner/nmrpflash/issues/148#issuecomment-2106116860, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXPWSJT2VQSWLRRT3CVWLZB3XSFAVCNFSM6AAAAABHMKY72SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBWGEYTMOBWGA . You are receiving this because you commented.Message ID: @.***>

antipodes5 commented 4 months ago

Tried it again. All from a fresh installation of the OS, new copies of all files. Still getting the same behavior, despite ufw (Uncomplicated FireWall) being set to disable.

Error message is still Operation not permitted. Still unsure what is not permitting what.

antipodes5 commented 4 months ago

Originally I was using Lubuntu 22.04.4, but I just bust back down the the 'stable' release of 22.04. Makes no difference, apparently.

Also attempted with Fedora 40 live usb, but neither nmrpflash nor the dependencies are found by dnf, so I left it.

Might make one more try with a live Ubuntu usb, but not sure. Leaning towards abandoning the 6220 and using an old Rasp. Pi instead.

Again, to make clear, ufw has been disabled all along.

For your request:

$ sudo ufw disable
[sudo] password for x: 
Firewall stopped and disabled on system startup

and

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ufw-before-logging-input  all  --  anywhere             anywhere            
ufw-before-input  all  --  anywhere             anywhere            
ufw-after-input  all  --  anywhere             anywhere            
ufw-after-logging-input  all  --  anywhere             anywhere            
ufw-reject-input  all  --  anywhere             anywhere            
ufw-track-input  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ufw-before-logging-forward  all  --  anywhere             anywhere            
ufw-before-forward  all  --  anywhere             anywhere            
ufw-after-forward  all  --  anywhere             anywhere            
ufw-after-logging-forward  all  --  anywhere             anywhere            
ufw-reject-forward  all  --  anywhere             anywhere            
ufw-track-forward  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ufw-before-logging-output  all  --  anywhere             anywhere            
ufw-before-output  all  --  anywhere             anywhere            
ufw-after-output  all  --  anywhere             anywhere            
ufw-after-logging-output  all  --  anywhere             anywhere            
ufw-reject-output  all  --  anywhere             anywhere            
ufw-track-output  all  --  anywhere             anywhere            

Chain ufw-after-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-input (1 references)
target     prot opt source               destination         

Chain ufw-after-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-after-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-after-output (1 references)
target     prot opt source               destination         

Chain ufw-before-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-before-output (1 references)
target     prot opt source               destination         

Chain ufw-reject-forward (1 references)
target     prot opt source               destination         

Chain ufw-reject-input (1 references)
target     prot opt source               destination         

Chain ufw-reject-output (1 references)
target     prot opt source               destination         

Chain ufw-track-forward (1 references)
target     prot opt source               destination         

Chain ufw-track-input (1 references)
target     prot opt source               destination         

Chain ufw-track-output (1 references)
target     prot opt source               destination

All seems very permissive.

antipodes5 commented 4 months ago

And if it makes a difference, I am running it through a switch (the PC and the router are the only things connected).

antipodes5 commented 4 months ago

Have changed distros to the more mainstream Ubuntu. Tried with and without the switch. Yes, ufw is off.

-> no change in behavior.

I guess its one of three things:

hardware problem. Maybe???
a debian-based or Ubuntu-based linux problem.
something on the router is still active and looking for permissions

If I can get VirtualBox to run Windows and bridge to the router, I'll try there. I'll report success or failure here. But I have to be honest, my enthusiasm is waning.

antipodes5 commented 4 months ago

I should note though, I am grateful for your work @jclehner .

antipodes5 commented 4 months ago

If I can get VirtualBox to run Windows and bridge to the router, I'll try there. I'll report success or failure here. But I have to be honest, my enthusiasm is waning.

Fail. Couldn't get Windows running in qemu/kvm-virt-mang or bootable USB.

Had enough.

antipodes5 commented 4 months ago

One final point @jclehner :

Someone has noticed a difference between Windows and Linux:

On windows, this is the purpose of the message Received keep-alive request (253). which I didn't had using Linux. This message increments itself while flashing.

jclehner commented 4 months ago

Had enough.

Bummer. Could you try running nmrpflash with strace one last time. And include the dmesg output immediately after running the command.

$ sudo strace -tt -o strace.log -- ./nmrpflash -i enp0s25 -f R6220-V1.1.0.114_1.0.1.img
$ sudo dmesg > dmesg.log

and attach the resulting strace.log and dmesg.log files.

Someone has noticed a difference between Windows and Linux:

Weird. I'll investigate...

antipodes5 commented 4 months ago

Going to take a little time.

Have a complete failure to boot on that machine - just after I installed fuse to get your program up and running. I thought it might be that (still might), but apparently its not uncommon.

-> Full reinstall.

antipodes5 commented 4 months ago

ok, second attempt.

New system in place (Ubuntu jammy), your latest version and tried again. Same results.

libpcap0.8/jammy,now 1.10.1-4build1 amd64 [installed,automatic]
libnl-3-200/jammy,now 3.5.0-0.1 amd64 [installed,automatic]

$ sudo ufw disable
[sudo] password for x: 
Firewall stopped and disabled on system startup
...
$ sudo strace -tt -o strace.log -- ./nmrpflash -i enp0s25 -f R6220-V1.1.0.114_1.0.1.img
[sudo] password for x: 
Waiting for Ethernet connection (Ctrl-C to skip).
Advertising NMRP server on enp0s25 ... /
Received configuration request from b0:39:56:1a:bc:00.
Sending configuration: 10.164.183.253/24.
Received upload request without filename.
Uploading R6220-V1.1.0.114_1.0.1.img ... sendto: Operation not permitted

strace.log.zip dmesg-extract.log

Hope it helps!

jclehner commented 3 months ago

Hmm... I'm kinda stumped by this one.

It's not a firewall issue
It's neither a hardware nor driver issue (same results with internal and external Ethernet, as reported by you on the OpenWRT forums)
It's not an Ubuntu 22.04 issue (flashing one of my devices from that Ubuntu release worked without issue)

sendto is the function that sends the TFTP packets, but it isn't even supposed to fail with that specific error code (EPERM corresponds to "Operation not permitted"). The fact that the function fails must be related to something locally, not the router. It fails to send even the first TFTP packet.

antipodes5 commented 3 months ago

This is interesting, (in a depressing kind of way).

Could this be hardware related? Its a battered old second-hand Levono x230i, ex-corporate. Its been great for what it is, but I've never tried to use it for any work on a second device, like the router.

I've just had a fail on a serial connection to a malfunctioning Pi, but in reality I think that's a problem with the connector, not the computer. Still makes me wonder.

Could a corporate security policy shut down serial or other (e.g. TFPT) connections at a hardware level as a safety measure?

jclehner commented 2 months ago

Hope it's not too late. You could try another TFTP client, since the issue might be nmrpflash's TFTP implementation:

$ sudo apt-get install -y tftp-hpa
$ sudo nmrpflash -i enp0s25 -c 'tftp -v -m binary $IP $PORT -c put R6220-V1.1.0.114_1.0.1.img'

Note that the current version of nmrpflash will exit immedtiately after running the tftp command, so wait for at least 5-10 minutes before rebooting the router.

Could this be hardware related? Could a corporate security policy shut down serial or other (e.g. TFPT) connections at a hardware level as a safety measure?

No. Something in the linux kernel tells the sendto function to fail with that error code, before the packet is even sent. Could be AppArmor related.

jclehner / nmrpflash

fail on R6220 'brick' - "sendto: Operation not permitted" #148