transmission-daemon TCP uploads capped to ~ 300 kBps per peer

[ghost]

Maintainer: @dangowrt Environment: ARMV7, IPQ806x, 23.05.2 Version: 4.0.4-1 Description: TCP uploads from Transmission seem to be capped at around 300 kBps (most commonly between 270-320 kBps). μTP uploads, as well as all downloads, are unaffected and seem to only be capped at my installed bandwidth of 500 mbps (symmetric).

The uploads are from a USB 3.0 SSD that is attached to the device. Read speeds from the USB device, and the attached network are all otherwise nominal, as tested locally within the device and over the LAN (using SMBD).

Here's a redacted copy of my /etc/config/transmission.conf file:

config transmission
        option config_overwrite '1'
        option mem_percentage '70'
        option nice '5'
        option blocklist_enabled 'true'
        option blocklist_url 'https://github.com/Naunter/BT_BlockLists/raw/master/bt_blocklists.gz'
        option peer_port 'xxxx'
        option peer_port_random_on_start 'false'
        option preallocation '0'
        option rename_partial_files 'true'
        option rpc_enabled 'true'
        option upload_slots_per_torrent '5'
        option ratio_limit_enabled 'true'
        option umask '"000"'
        option port_forwarding_enabled 'false'
        option config_dir '/mnt/sda1/Transmission'
        option user 'root'
        option group 'root'
        option encryption '0'
        option torrent_added_verify_mode 'fast'
        option trash_original_torrent_files 'false'
        option download_dir '/mnt/sda1/'
        option scrape_paused_torrents_enabled 'false'
        option enabled '1'
        option peer_limit_global '190'
        option peer_limit_per_torrent '5'
        option message_level '2'
        option start_added_torrents 'true'
        option download_queue_enabled 'false'
        option queue_stalled_enabled 'false'
        option ratio_limit '1.03'
        option cache_size_mb '4'
        option web_home '/mnt/sda1/xxxx'
        option rpc_host_whitelist_enabled 'false'
        option rpc_whitelist_enabled 'true'
        option rpc_whitelist '127.0.0.*,192.168.xxxx.*'
        option rpc_authentication_required 'false'
        option rpc_port 'xxxx'
        option default_trackers 'udp://93.158.213.92:1337/announce\n\nudp://102.223.180.235:6969/announce\n\nudp://23.134.88.6:1337/announce\n\nudp://193.189.100.187:6969/announce\n\nudp://185.243.218.213:80/announce\n\nudp://91.216.110.52:451/announce\n\nudp://208.83.20.20:6969/announce\n\nudp://23.157.120.14:6969/announce\n\nudp://156.234.201.18:80/announce\n\nudp://185.102.219.163:6969/announce\n\nudp://185.50.159.149:6969/announce\n\nudp://209.141.59.16:6969/announce\n\nudp://38.7.201.142:6969/announce\n\nudp://73.170.204.100:6969/announce\n\nudp://176.31.250.174:6969/announce\n\nudp://82.156.24.219:6969/announce\n\nudp://83.102.180.21:80/announce\n\nudp://185.230.4.150:1337/announce\n\nudp://tracker.opentrackr.org:1337/announce\n\nudp://opentracker.i2p.rocks:6969/announce\n\nudp://open.demonii.com:1337/announce\n\nhttp://tracker.openbittorrent.com:80/announce\n\nudp://tracker.openbittorrent.com:6969/announce\n\nudp://open.stealth.si:80/announce\n\nudp://tracker.torrent.eu.org:451/announce\n\nudp://exodus.desync.com:6969/announce\n\nudp://explodie.org:6969/announce\n\nudp://tracker1.bt.moack.co.kr:80/announce\n\nhttps://tracker.tamersunion.org:443/announce\n\nudp://uploads.gamecoast.net:6969/announce\n\nudp://tracker.tiny-vps.com:6969/announce\n\nudp://tracker.theoks.net:6969/announce\n\nudp://tracker.cubonegro.lol:6969/announce\n\nudp://thinking.duckdns.org:6969/announce\n\nudp://tamas3.ynh.fr:6969/announce\n\nudp://ryjer.com:6969/announce\n\nudp://retracker01-msk-virt.corbina.net:80/announce\n\nudp://p4p.arenabg.com:1337/announce'

TCP send buffers don't seem to affect the problem one way or another, and I've tried tweaking all the other TCP settings too. Here's my current /etc/sysctl.d/20-transmission.conf file:

# Transmission requests large buffers by default
net.core.rmem_max = 8388608
net.core.wmem_max = 20971520
net.ipv4.tcp_rmem = 4096 16384 8388608
net.ipv4.tcp_wmem = 4096 16384 20971520
net.ipv4.tcp_no_metrics_save = 1

# net.ipv4.tcp_mem = 8919 11892 17838
net.ipv4.tcp_mem = 8919 23784 26757

# Some firewalls block SYN packets that are too small
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_reflect_tos = 1

The TCP congestion control algorithm doesn't seem to matter. I've tried cubic, reno, bbr, and scalable.

When a TCP upload is in progress, ss shows a large Send-Q for the connection (as expected), but it doesn't show any memory information even with the -m option set.

Given all the steps that I've described above, I currently suspect this to be a bug in the package. I'm open to trying out any other troubleshooting ideas to help narrow down the cause.

[ghost]

I have also tried sqm-scripts package with fq_codel and cake qdiscs, neither of which seemed to affect this issue one way or another.

[brada4]

You need to check you provider's traffic management policies. Especially if normal connections like speed.cloudflare.com shows reasonable bw.

Your sysctls wildly increase network memory usage just get rid of all of them. And dont change unless nstat and/or ethtool -S show actual resorce shortage.

[ghost]

Thanks @brada4 , I'll follow your suggestion to rule out any traffic management issues. Will report back here in a few days with results.

[ghost]

I just installed qBittorrent on my Windows laptop and started seeding the very same torrent (using SMBD) that was going at ~260kbps on transmission-daemon for the last several days. Same connection, same files from the same filesystem, same tracker, same peer.

Also, I found that my ISP publicly declares that they are net-neutral. In fact they are actively lobbying for policy reform to make this the law for all ISPs in my region.

[ghost]

Tweaked some parameters in the fileshare and now I'm up to this:

So in just this one example I'm getting 16250 kBps, about 50 times the best upload speed I've ever seen come out of transmission-daemon

[brada4]

You can try disabling various netcard transmission offloads via ethtool -k/-K So far we figured out your sysctls are not of fault as one big macro block. nstat | grep -e Err -e Over -e Fail -e Drop Probably search web on how to prevent particular anomaly.

[ghost]

I'm not an OpenWRT/Linux developer so I don't fully comprehend what you're suggesting. In any case here's the output of the commands you suggested:

root@OpenWRT:~# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: on [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

root@OpenWRT:~# nstat | grep -e Err -e Over -e Fail -e Drop
IcmpInErrors                    563                0.0
TcpAttemptFails                 157                0.0
TcpExtSyncookiesFailed          89                 0.0
TcpExtTCPSackFailures           49                 0.0
TcpExtTCPLossFailures           16                 0.0
TcpExtTCPSackRecoveryFail       140                0.0
TcpExtTCPAbortFailed            19                 0.0
TcpExtTCPRetransFail            295                0.0

Is there something in particular you would recommend I do to help troubleshoot further. It seems unlikely to me that this issue is unique to my setup. Seems more likely that others came across bugs like this and simply moved on to other platforms/solutions.

[brada4]

Try one at the time and measure (kernel CPU consumption too)

• gso off
• tx off
• sg off

If none helps do same with rx tunables. If nothing helps revert.

TcpExtTCPRetransFail -> What does ethtool -S say? i.e if drop was due to soft buffers or to the wire. Format is nonstandard, do guesses around small-ish numbers.

Another would be to add tbf qdisc with bandwidth x latency plus fq_codel sub-qdisc as to voluntarily drop line speed under subscribed.

[ghost]

Sorry, but I'm going to have to ask you to please go into a bit more detail to explain what you mean by each of those 3 things you mentioned. Right now I'm guessing you may be referring to some features of that device that I could experiment with turning off, but the only ones I'm seeing from the ethtool results above that match your brief description are "tx-gso-robust", "tx-gso-partial", and "tx-gso-list", each of which is already off.

Here are the device stats from ethtool as you'd suggested:

root@OpenWRT:~# ethtool -S eth0
NIC statistics:
     mmc_tx_octetcount_gb: 263133207
     mmc_tx_framecount_gb: 17595688
     mmc_tx_broadcastframe_g: 3
     mmc_tx_multicastframe_g: 223
     mmc_tx_64_octets_gb: 867089
     mmc_tx_65_to_127_octets_gb: 1936085
     mmc_tx_128_to_255_octets_gb: 437620
     mmc_tx_256_to_511_octets_gb: 218925
     mmc_tx_512_to_1023_octets_gb: 125659
     mmc_tx_1024_to_max_octets_gb: 14010310
     mmc_tx_unicast_gb: 17595462
     mmc_tx_multicast_gb: 223
     mmc_tx_broadcast_gb: 3
     mmc_tx_underflow_error: 0
     mmc_tx_singlecol_g: 0
     mmc_tx_multicol_g: 0
     mmc_tx_deferred: 0
     mmc_tx_latecol: 0
     mmc_tx_exesscol: 0
     mmc_tx_carrier_error: 0
     mmc_tx_octetcount_g: 263133207
     mmc_tx_framecount_g: 17595688
     mmc_tx_excessdef: 0
     mmc_tx_pause_frame: 0
     mmc_tx_vlan_frame_g: 17595681
     mmc_rx_framecount_gb: 11128147
     mmc_rx_octetcount_gb: 3495424753
     mmc_rx_octetcount_g: 3495424753
     mmc_rx_broadcastframe_g: 0
     mmc_rx_multicastframe_g: 0
     mmc_rx_crc_error: 0
     mmc_rx_align_error: 0
     mmc_rx_run_error: 0
     mmc_rx_jabber_error: 0
     mmc_rx_undersize_g: 0
     mmc_rx_oversize_g: 0
     mmc_rx_64_octets_gb: 0
     mmc_rx_65_to_127_octets_gb: 5071913
     mmc_rx_128_to_255_octets_gb: 777898
     mmc_rx_256_to_511_octets_gb: 257000
     mmc_rx_512_to_1023_octets_gb: 111256
     mmc_rx_1024_to_max_octets_gb: 4910080
     mmc_rx_unicast_g: 11128147
     mmc_rx_length_error: 0
     mmc_rx_autofrangetype: 0
     mmc_rx_pause_frames: 0
     mmc_rx_fifo_overflow: 0
     mmc_rx_vlan_frames_gb: 11128147
     mmc_rx_watchdog_error: 0
     mmc_rx_ipc_intr_mask: 2147385342
     mmc_rx_ipc_intr: 0
     mmc_rx_ipv4_gd: 11128023
     mmc_rx_ipv4_hderr: 0
     mmc_rx_ipv4_nopay: 1
     mmc_rx_ipv4_frag: 5
     mmc_rx_ipv4_udsbl: 186
     mmc_rx_ipv4_gd_octets: 3229359075
     mmc_rx_ipv4_hderr_octets: 0
     mmc_rx_ipv4_nopay_octets: 52
     mmc_rx_ipv4_frag_octets: 6593
     mmc_rx_ipv4_udsbl_octets: 25663
     mmc_rx_ipv6_gd_octets: 0
     mmc_rx_ipv6_hderr_octets: 0
     mmc_rx_ipv6_nopay_octets: 0
     mmc_rx_ipv6_gd: 0
     mmc_rx_ipv6_hderr: 0
     mmc_rx_ipv6_nopay: 0
     mmc_rx_udp_gd: 2138037
     mmc_rx_udp_err: 0
     mmc_rx_tcp_gd: 8965811
     mmc_rx_tcp_err: 29
     mmc_rx_icmp_gd: 23989
     mmc_rx_icmp_err: 6
     mmc_rx_udp_gd_octets: 1827701026
     mmc_rx_udp_err_octets: 0
     mmc_rx_tcp_gd_octets: 1175050771
     mmc_rx_tcp_err_octets: 1297
     mmc_rx_icmp_gd_octets: 4020646
     mmc_rx_icmp_err_octets: 274
     mmc_tx_fpe_fragment_cntr: 0
     mmc_tx_hold_req_cntr: 0
     mmc_rx_packet_assembly_err_cntr: 0
     mmc_rx_packet_smd_err_cntr: 0
     mmc_rx_packet_assembly_ok_cntr: 0
     mmc_rx_fpe_fragment_cntr: 0
     tx_underflow: 0
     tx_carrier: 0
     tx_losscarrier: 0
     vlan_tag: 0
     tx_deferred: 0
     tx_vlan: 17595673
     tx_jabber: 0
     tx_frame_flushed: 0
     tx_payload_error: 0
     tx_ip_header_error: 0
     rx_desc: 0
     sa_filter_fail: 0
     overflow_error: 0
     ipc_csum_error: 0
     rx_collision: 0
     rx_crc_errors: 0
     dribbling_bit: 0
     rx_length: 0
     rx_mii: 0
     rx_multicast: 0
     rx_gmac_overflow: 0
     rx_watchdog: 0
     da_rx_filter_fail: 0
     sa_rx_filter_fail: 0
     rx_missed_cntr: 0
     rx_overflow_cntr: 0
     rx_vlan: 11127091
     rx_split_hdr_pkt_n: 0
     tx_undeflow_irq: 0
     tx_process_stopped_irq: 0
     tx_jabber_irq: 0
     rx_overflow_irq: 0
     rx_buf_unav_irq: 0
     rx_process_stopped_irq: 0
     rx_watchdog_irq: 0
     tx_early_irq: 0
     fatal_bus_error_irq: 0
     rx_early_irq: 88015
     threshold: 1
     tx_pkt_n: 17595680
     rx_pkt_n: 11127091
     normal_irq_n: 5320840
     rx_normal_irq_n: 4641691
     napi_poll: 6770549
     tx_normal_irq_n: 698400
     tx_clean: 2123891
     tx_set_ic_bit: 703827
     irq_receive_pmt_irq_n: 0
     mmc_tx_irq_n: 0
     mmc_rx_irq_n: 0
     mmc_rx_csum_offload_irq_n: 0
     irq_tx_path_in_lpi_mode_n: 0
     irq_tx_path_exit_lpi_mode_n: 0
     irq_rx_path_in_lpi_mode_n: 0
     irq_rx_path_exit_lpi_mode_n: 0
     phy_eee_wakeup_error_n: 0
     ip_hdr_err: 0
     ip_payload_err: 0
     ip_csum_bypassed: 6
     ipv4_pkt_rcvd: 11127008
     ipv6_pkt_rcvd: 0
     no_ptp_rx_msg_type_ext: 11127008
     ptp_rx_msg_type_sync: 0
     ptp_rx_msg_type_follow_up: 0
     ptp_rx_msg_type_delay_req: 0
     ptp_rx_msg_type_delay_resp: 0
     ptp_rx_msg_type_pdelay_req: 0
     ptp_rx_msg_type_pdelay_resp: 0
     ptp_rx_msg_type_pdelay_follow_up: 0
     ptp_rx_msg_type_announce: 0
     ptp_rx_msg_type_management: 0
     ptp_rx_msg_pkt_reserved_type: 0
     ptp_frame_type: 0
     ptp_ver: 0
     timestamp_dropped: 0
     av_pkt_rcvd: 0
     av_tagged_pkt_rcvd: 0
     vlan_tag_priority_val: 0
     l3_filter_match: 0
     l4_filter_match: 0
     l3_l4_filter_no_match: 0
     irq_pcs_ane_n: 0
     irq_pcs_link_n: 0
     irq_rgmii_n: 0
     mtl_tx_status_fifo_full: 0
     mtl_tx_fifo_not_empty: 0
     mmtl_fifo_ctrl: 0
     mtl_tx_fifo_read_ctrl_write: 0
     mtl_tx_fifo_read_ctrl_wait: 0
     mtl_tx_fifo_read_ctrl_read: 0
     mtl_tx_fifo_read_ctrl_idle: 0
     mac_tx_in_pause: 0
     mac_tx_frame_ctrl_xfer: 0
     mac_tx_frame_ctrl_idle: 0
     mac_tx_frame_ctrl_wait: 0
     mac_tx_frame_ctrl_pause: 0
     mac_gmii_tx_proto_engine: 0
     mtl_rx_fifo_fill_level_full: 0
     mtl_rx_fifo_fill_above_thresh: 0
     mtl_rx_fifo_fill_below_thresh: 0
     mtl_rx_fifo_fill_level_empty: 0
     mtl_rx_fifo_read_ctrl_flush: 0
     mtl_rx_fifo_read_ctrl_read_data: 0
     mtl_rx_fifo_read_ctrl_status: 0
     mtl_rx_fifo_read_ctrl_idle: 0
     mtl_rx_fifo_ctrl_active: 0
     mac_rx_frame_ctrl_fifo: 0
     mac_gmii_rx_proto_engine: 0
     tx_tso_frames: 0
     tx_tso_nfrags: 0
     mtl_est_cgce: 0
     mtl_est_hlbs: 0
     mtl_est_hlbf: 0
     mtl_est_btre: 0
     mtl_est_btrlm: 0
     q0_tx_pkt_n: 17595680
     q0_tx_irq_n: 0
     q0_rx_pkt_n: 11127091
     q0_rx_irq_n: 0

I'm also not entirely sure how to interpret your suggestions about qdiscs. I'm guessing "tbf" is Token Bucket Filter. How do I go about getting/setting that on this device?

Finally, based on the discsusion so far I'd like to share the results of another test that I ran. This seems to indicate that the problem may not be with the transmission-daemon package as I had originally thought. Please let me know if this issue is best moved to another place for further discussion?

root@OpenWRT:~# speedtest-netperf.sh -c -t 180 -H netperf-eu.bufferbloat.net -n 1
2024-01-16 19:53:19 Starting speedtest for 180 seconds per transfer session.
Measure speed to netperf-eu.bufferbloat.net (IPv4) while pinging gstatic.com.
Download and upload sessions are concurrent, each with 1 simultaneous streams.
........................................................................................................................................................................................
 Download: 174.41 Mbps
   Upload:   3.16 Mbps
  Latency: [in msec, 184 pings, 0.00% packet loss]
      Min:  15.054
    10pct:  15.152
   Median:  15.529
      Avg:  15.606
    90pct:  16.119
      Max:  16.591
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 179 samples]
     cpu0:  16.1 +/-  5.7  @ 1725 MHz
     cpu1:   7.0 +/-  6.3  @ 1725 MHz
 Overhead: [in % used of total CPU available]
  netperf:   6.7

[brada4]

The experiment plan: Stop torrent activity and keep lan-side activity low run netperf disable one of tx offloads run netperf disable another tx offload .... i.e attempt to identify damaging offload in network card. 300kB/s is way too slow even for 10Mbps ethernet.

[ghost]

Okay two questions:

(1) I've found the following features in ethtool that I think pertain to tx offloads. Can you please confirm if these are the correct, and the only ones pertaining to data transmission?

tx-vlan-offload tx-gso-robust tx-gso-partial tx-gso-list esp-tx-csum-hw-offload tls-hw-tx-offload

(2) Each of these is already off in the ethtool -k results shown above. Would it be worthwhile to try enabling these offloads instead to see if performance improves?

[brada4]

those "fixed" can not be changed.

generic-segmentation-offload gso off tx-checksumming tx off scatter-gather sg off

I used 10 year old still supported short aliases for those features i spotted in the list. tcp-segmentation-offload tso is already off.

[brada4]

Also ethtool -a/A may help, if you get too eager pause frames ypur netcard just queues up packets without sending. ethtool eth0 to see pause status.

[ghost]

Oh, I see now! Looks like your suggested changes could all be accomplished by just running the following commands!

ethtool -K eth0 gso off ethtool -K eth0 tx off ethtool -K eth0 sg off

Unfortunately the issue is completely unaffected by these changes.

root@OpenWRT:~# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: on [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

root@OpenWRT:~# speedtest-netperf.sh -c -H netperf-eu.bufferbloat.net -n 1
2024-01-18 12:11:45 Starting speedtest for 60 seconds per transfer session.
Measure speed to netperf-eu.bufferbloat.net (IPv4) while pinging gstatic.com.
Download and upload sessions are concurrent, each with 1 simultaneous streams.
...............................................................
 Download: 119.17 Mbps
   Upload:   2.40 Mbps
  Latency: [in msec, 63 pings, 0.00% packet loss]
      Min:  14.843
    10pct:  15.009
   Median:  15.217
      Avg:  15.407
    90pct:  15.825
      Max:  18.903
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 60 samples]
     cpu0:  15.5 +/- 10.7  @ 1725 MHz
     cpu1:   5.7 +/-  4.5  @ 1725 MHz
 Overhead: [in % used of total CPU available]
  netperf:   5.7

[brada4]

Run ethtool eth0 And try to disable pause via ethtool -a/-A I measured a bit above subscribed speed using your command on RT-AX54 (different and slower platform) (there is no pause indicated in ethtool -S, might be another red herring though, but i am out of more ideas)

[ghost]

Looks like maybe pause frames are already disabled for transmit?

root@OpenWRT:~# ethtool eth0
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: No
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: ug
        Wake-on: d
        Current message level: 0x0000003f (63)
                               drv probe link timer ifdown ifup
        Link detected: yes

[ghost]

root@OpenWRT:~# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             off
TX:             off
RX negotiated:  off
TX negotiated:  off

[brada4]

pause frames were not negotiated. I am out of ideas. In principle this issue is generic for the device or target with that network card, transmission was just a tool for initial discovery, and netperf confirmed same issue. i.e not qdisc, not pause frames, not tcp buffer sizes, try to disable more offloads, some will rise CPU usage for downloads, but we need to find if there is one broken offload.

[ghost]

@Ansuel Do you have any thoughts on this?

[brada4]

https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/stmicro/stmmac.html You need to set module parameter via modules.conf/add to modules.d (sort of ethtool reported all off)

[ghost]

https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/stmicro/stmmac.html You need to set module parameter via modules.conf/add to modules.d (sort of ethtool reported all off)

What would be the parameter that I would need to add to /etc/modules.conf please? Unfortunately I lack even a basic understanding of the IPQ806x drivers so I am quite clueless in this area.

[brada4]

options stmmac_pci flow_ctrl=0 

(check with ethtool -i interface for correct driver name)

[ghost]

Even before setting the driver option, flow control shows as off for that device in the system logs:

kern.info kernel: [   34.193755] ipq806x-gmac-dwmac 37000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

But just to make, I tried it out anyway, but in vain:

root@OpenWRT:~# cat /etc/modules.conf
# examples:
# options mod1 option=val
# blacklist mod2
options st_gmac flow_ctrl=0

And after a reboot:

root@OpenWRT:~# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: on [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

root@OpenWRT:~# speedtest-netperf.sh -c -H netperf-eu.bufferbloat.net -n 1
2024-01-23 17:43:49 Starting speedtest for 60 seconds per transfer session.
Measure speed to netperf-eu.bufferbloat.net (IPv4) while pinging gstatic.com.
Download and upload sessions are concurrent, each with 1 simultaneous streams.
...............................................................
 Download: 148.67 Mbps
   Upload:   2.76 Mbps
  Latency: [in msec, 63 pings, 0.00% packet loss]
      Min:  14.972
    10pct:  15.087
   Median:  15.400
      Avg:  15.536
    90pct:  16.057
      Max:  16.826
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 60 samples]
     cpu0:  16.3 +/-  6.4  @ 1725 MHz
     cpu1:   4.8 +/-  3.1  @ 1725 MHz
 Overhead: [in % used of total CPU available]
  netperf:   6.0

What do we do now?

[brada4]

It must be transferred to ../openwrt. You can try to read kernel documentation, tso is disabled, they say it is supported on fresh chips. i.e ethtool -K ethX tso on. [*] https://www.kernel.org/doc/html/latest/networking/segmentation-offloads.html

[ghost]

root@OpenWRT:~# ethtool -K eth0 tso on
Cannot change tcp-segmentation-offload
Could not change any device features

How does one transfer an issue between repos?

[brada4]

Wait for a maintainer to do that. Probably re-phrase issue title to tragic upload performance on router X openwrt v Y

[ghost]

This issue seems to still persist in Openwrt v23.05.3

[brada4]

Obviously kernel issue, not transmission, easily replicated with speedtest-netperf. I thnk worth following up in the other issue. No magic transmission can do to bypass kernel network driver.

[ghost]

Thanks @brada4, does this seem like the same issue as https://github.com/openwrt/openwrt/issues/10879 ?

[brada4]

Yes it looks like that. ipq... kernel is only thing common to netperf, torrent and the other issue.

[ghost]

Thanks for the confirmation @brada4