search

home-assistant / supervisor

:house_with_garden: Home Assistant Supervisor
https://home-assistant.io/hassio/
Apache License 2.0
1.72k stars 627 forks source link

Ingress makes videos from Frigate download 4 times slower #5209

Open Chaoscontrol opened 1 month ago

Chaoscontrol commented 1 month ago

Describe the issue you are experiencing

I noticed when playing recorded videos from Frigate via the Frigate UI, it was taking a good while to load each clip (8MB clips in 3s). After some debugging with Frigate devs on Discord, we figured out it was Ingress fault.

When accessing Frigate UI directly with its port, the same clips download 4 times faster (8MB clips in 700ms). This confirmed it wasn't a Frigate issue but a HA Ingress issue.

See here the comparison: image

I use HAOS, and use an internal M2 SSD. Recordings are stored there, same as HAOS. Host is connected via Eth, and the laptop I'm using to access is too using Eth.

Not certain when this started. I don't think it was due to recent updates (either Frigate or HA). I just committed to ask about the issue now.

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

  1. Install Frigate addon (beta4)
  2. Set up cameras and configuration addon, let it save some recordings
  3. Run the addon via Ingress, go to the Review tab, Motion, open your camera
  4. Scrub through the footage using the timeline on the right hand side
  5. The download of this saved files is very slow, about 3-4 times slower than if using Frigate directly (out of Ingress)
  6. You can check the download in browser Dev tools, Network, filter ts files.

Anything in the Supervisor logs that might be useful for us?

2024-07-27 10:03:00.942 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-07-27 10:03:00.942 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-07-27 10:03:00.942 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-07-27 10:03:17.048 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:04:06.083 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-07-27 10:08:17.043 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:13:17.050 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:16:16.071 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/adamoutler/HassOSConfigurator repository
2024-07-27 10:16:16.076 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/bropat/hassio-eufy-security-ws repository
2024-07-27 10:16:16.078 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/Rbillon59/home-assistant-addons repository
2024-07-27 10:16:16.085 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/einschmidt/hassio-addons repository
2024-07-27 10:16:16.089 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/cnorick/ha-auto-guest-login repository
2024-07-27 10:16:16.092 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/brenner-tobias/ha-addons repository
2024-07-27 10:16:16.097 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/blakeblackshear/frigate-hass-addons repository
2024-07-27 10:16:16.101 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/ihrapsa/hassio-addons.git repository
2024-07-27 10:16:16.110 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/matt8707/addon-ha-fusion repository
2024-07-27 10:16:16.120 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt repository
2024-07-27 10:16:16.124 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/homewayio/addon repository
2024-07-27 10:16:16.126 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/sabeechen/hassio-google-drive-backup repository
2024-07-27 10:16:16.132 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/music-assistant/home-assistant-addon repository
2024-07-27 10:16:16.135 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
2024-07-27 10:16:16.138 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
2024-07-27 10:16:16.142 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
2024-07-27 10:16:16.144 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/alexbelgium/hassio-addons repository
2024-07-27 10:16:16.147 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/koush/scrypted repository
2024-07-27 10:16:16.149 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/Poeschl/Hassio-Addons repository
2024-07-27 10:16:16.150 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/petersendev/hassio-addons repository
2024-07-27 10:16:17.429 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/TekniskSupport/homeassistant-addons repository
2024-07-27 10:16:19.657 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access)
2024-07-27 10:16:19.664 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access) Beta
2024-07-27 10:16:19.900 WARNING (SyncWorker_6) [supervisor.store.data] Can't read /data/addons/git/09e60fb6/external/ring-client-api/packages/homebridge-ring/config.schema.json: Invalid Add-on config!. Got {'pluginAlias': 'Ring', 'pluginType': 'platform', 'singular': True, 'customUi': True, 'customUiPath': './lib/homebridge-ui', 'customUiDevServer': 'http://localhost:3000/api/plugins/settings-ui/homebridge-ring', 'headerDisplay': 'See [the wiki](https://github.com/dgreif/ring/wiki/Refresh-Tokens) for details on generating a refresh token', 'accountLinkingHeader': 'Please supply your Ring credentials.  These will be used to contact the Ring servers and generate a unique access token for your acc...
2024-07-27 10:16:20.030 WARNING (SyncWorker_6) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Signalk Server
2024-07-27 10:16:20.391 WARNING (SyncWorker_2) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access)
2024-07-27 10:16:20.398 WARNING (SyncWorker_2) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access) Beta
2024-07-27 10:16:20.506 WARNING (SyncWorker_0) [supervisor.store.data] Can't read /data/addons/git/09e60fb6/external/ring-client-api/packages/homebridge-ring/config.schema.json: Invalid Add-on config!. Got {'pluginAlias': 'Ring', 'pluginType': 'platform', 'singular': True, 'customUi': True, 'customUiPath': './lib/homebridge-ui', 'customUiDevServer': 'http://localhost:3000/api/plugins/settings-ui/homebridge-ring', 'headerDisplay': 'See [the wiki](https://github.com/dgreif/ring/wiki/Refresh-Tokens) for details on generating a refresh token', 'accountLinkingHeader': 'Please supply your Ring credentials.  These will be used to contact the Ring servers and generate a unique access token for your acc...
2024-07-27 10:16:20.623 WARNING (SyncWorker_0) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Signalk Server
2024-07-27 10:16:20.700 INFO (MainThread) [supervisor.store] Loading add-ons from store: 236 all - 0 new - 0 remove
2024-07-27 10:16:20.700 INFO (MainThread) [supervisor.store] Loading add-ons from store: 236 all - 0 new - 0 remove
2024-07-27 10:18:17.048 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:23:17.046 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:28:17.050 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:33:17.048 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:34:06.198 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-07-27 10:38:17.048 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:39:57.907 INFO (MainThread) [supervisor.backups.manager] Backup 290c60d2 starting stage addon_repositories
2024-07-27 10:39:57.907 INFO (MainThread) [supervisor.backups.manager] Backup 290c60d2 starting stage docker_config
2024-07-27 10:39:57.907 INFO (MainThread) [supervisor.backups.manager] Creating new partial backup with slug 290c60d2
2024-07-27 10:39:57.909 INFO (MainThread) [supervisor.backups.manager] Backup 290c60d2 starting stage addons
2024-07-27 10:39:57.958 INFO (MainThread) [supervisor.addons.addon] Building backup for add-on db21ed7f_immich
2024-07-27 10:39:57.961 INFO (MainThread) [supervisor.addons.addon] Finish backup for addon db21ed7f_immich
2024-07-27 10:39:57.962 INFO (MainThread) [supervisor.backups.manager] Backup 290c60d2 starting stage finishing_file
2024-07-27 10:39:57.965 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 290c60d2 completed
2024-07-27 10:39:57.966 INFO (MainThread) [supervisor.docker.addon] Updating image ghcr.io/alexbelgium/immich-amd64:1.109.2 to ghcr.io/alexbelgium/immich-amd64:1.110.0
2024-07-27 10:39:57.967 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/alexbelgium/immich-amd64 with tag 1.110.0.
2024-07-27 10:40:08.377 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
2024-07-27 10:40:08.382 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
2024-07-27 10:40:08.395 INFO (MainThread) [supervisor.api.middleware.security] /backups/290c60d2/info access from cebe7a76_hassio_google_drive_backup
2024-07-27 10:40:08.410 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
2024-07-27 10:40:08.414 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
2024-07-27 10:43:17.061 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:45:52.166 INFO (MainThread) [supervisor.addons.addon] Add-on 'db21ed7f_immich' successfully updated
2024-07-27 10:45:52.284 INFO (SyncWorker_2) [supervisor.docker.manager] Cleanup images: ['ghcr.io/alexbelgium/immich-amd64:1.109.2']
2024-07-27 10:46:02.126 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: db21ed7f_immich
2024-07-27 10:46:02.232 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/TekniskSupport/homeassistant-addons repository
2024-07-27 10:46:02.234 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/adamoutler/HassOSConfigurator repository
2024-07-27 10:46:02.236 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/bropat/hassio-eufy-security-ws repository
2024-07-27 10:46:02.240 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/Rbillon59/home-assistant-addons repository
2024-07-27 10:46:02.241 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/einschmidt/hassio-addons repository
2024-07-27 10:46:02.243 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/cnorick/ha-auto-guest-login repository
2024-07-27 10:46:02.252 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/brenner-tobias/ha-addons repository
2024-07-27 10:46:02.257 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/blakeblackshear/frigate-hass-addons repository
2024-07-27 10:46:02.261 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/ihrapsa/hassio-addons.git repository
2024-07-27 10:46:02.265 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/matt8707/addon-ha-fusion repository
2024-07-27 10:46:02.266 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt repository
2024-07-27 10:46:02.271 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/homewayio/addon repository
2024-07-27 10:46:02.274 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/sabeechen/hassio-google-drive-backup repository
2024-07-27 10:46:02.277 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/music-assistant/home-assistant-addon repository
2024-07-27 10:46:02.279 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
2024-07-27 10:46:02.281 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
2024-07-27 10:46:02.284 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
2024-07-27 10:46:02.286 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/alexbelgium/hassio-addons repository
2024-07-27 10:46:02.288 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/koush/scrypted repository
2024-07-27 10:46:02.290 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/Poeschl/Hassio-Addons repository
2024-07-27 10:46:02.292 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/petersendev/hassio-addons repository
2024-07-27 10:46:02.434 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
2024-07-27 10:46:06.576 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access)
2024-07-27 10:46:06.583 WARNING (SyncWorker_5) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access) Beta
2024-07-27 10:46:06.683 WARNING (SyncWorker_7) [supervisor.store.data] Can't read /data/addons/git/09e60fb6/external/ring-client-api/packages/homebridge-ring/config.schema.json: Invalid Add-on config!. Got {'pluginAlias': 'Ring', 'pluginType': 'platform', 'singular': True, 'customUi': True, 'customUiPath': './lib/homebridge-ui', 'customUiDevServer': 'http://localhost:3000/api/plugins/settings-ui/homebridge-ring', 'headerDisplay': 'See [the wiki](https://github.com/dgreif/ring/wiki/Refresh-Tokens) for details on generating a refresh token', 'accountLinkingHeader': 'Please supply your Ring credentials.  These will be used to contact the Ring servers and generate a unique access token for your acc...
2024-07-27 10:46:06.792 WARNING (SyncWorker_7) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Signalk Server
2024-07-27 10:46:07.153 WARNING (SyncWorker_3) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access)
2024-07-27 10:46:07.162 WARNING (SyncWorker_3) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate (Full Access) Beta
2024-07-27 10:46:07.262 WARNING (SyncWorker_6) [supervisor.store.data] Can't read /data/addons/git/09e60fb6/external/ring-client-api/packages/homebridge-ring/config.schema.json: Invalid Add-on config!. Got {'pluginAlias': 'Ring', 'pluginType': 'platform', 'singular': True, 'customUi': True, 'customUiPath': './lib/homebridge-ui', 'customUiDevServer': 'http://localhost:3000/api/plugins/settings-ui/homebridge-ring', 'headerDisplay': 'See [the wiki](https://github.com/dgreif/ring/wiki/Refresh-Tokens) for details on generating a refresh token', 'accountLinkingHeader': 'Please supply your Ring credentials.  These will be used to contact the Ring servers and generate a unique access token for your acc...
2024-07-27 10:46:07.385 WARNING (SyncWorker_7) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Signalk Server
2024-07-27 10:46:07.453 INFO (MainThread) [supervisor.store] Loading add-ons from store: 236 all - 0 new - 0 remove
2024-07-27 10:46:07.453 INFO (MainThread) [supervisor.store] Loading add-ons from store: 236 all - 0 new - 0 remove
2024-07-27 10:46:07.495 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:51:07.049 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:56:07.031 WARNING (MainThread) [supervisor.addons.options] Option 'webhook_tunnel_url' does not exist in the schema for Hass n8n (3cfc8f0f_hass-n8n)
2024-07-27 10:59:48.499 INFO (MainThread) [supervisor.backups.manager] Found 112 backup files
2024-07-27 10:59:58.649 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
2024-07-27 10:59:58.652 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup


### System Health information

## System Information

version | core-2024.7.3
-- | --
installation_type | Home Assistant OS
dev | false
hassio | true
docker | true
user | root
virtualenv | false
python_version | 3.12.4
os_name | Linux
os_version | 6.6.33-haos
arch | x86_64
timezone | Europe/London
config_dir | /config

<details><summary>Home Assistant Community Store</summary>

GitHub API | ok
-- | --
GitHub Content | ok
GitHub Web | ok
GitHub API Calls Remaining | 5000
Installed Version | 1.34.0
Stage | running
Available Repositories | 1462
Downloaded Repositories | 31

</details>

<details><summary>Home Assistant Cloud</summary>

logged_in | false
-- | --
can_reach_cert_server | ok
can_reach_cloud_auth | ok
can_reach_cloud | ok

</details>

<details><summary>Home Assistant Supervisor</summary>

host_os | Home Assistant OS 12.4
-- | --
update_channel | stable
supervisor_version | supervisor-2024.06.2
agent_version | 1.6.0
docker_version | 26.1.4
disk_total | 234.0 GB
disk_used | 142.8 GB
healthy | true
supported | true
host_connectivity | true
supervisor_connectivity | true
ntp_synchronized | true
virtualization | 
board | generic-x86-64
supervisor_api | ok
version_api | ok
installed_addons | Mosquitto broker (6.4.1), AirCast (4.2.1), Samba share (12.3.1), Zigbee2MQTT (1.39.0-1), Hass n8n (1.0.43), Home Assistant Google Drive Backup (0.112.1), Studio Code Server (5.15.0), Filebrowser (2.23.0_14), Frigate (Full Access) Beta (0.14.0-beta4), HyperHDR (19.0.0.0), Changedetection.io (0.46.1), Cloudflared (5.1.15), Advanced SSH & Web Terminal (18.0.0), SQLite Web (4.2.0), InfluxDB (5.0.0), Immich (1.110.0), Whatsapper (1.0.1)

</details>

<details><summary>Dashboards</summary>

dashboards | 8
-- | --
resources | 17
views | 12
mode | storage

</details>

<details><summary>Recorder</summary>

oldest_recorder_run | 17 July 2024 at 09:54
-- | --
current_recorder_run | 22 July 2024 at 09:36
estimated_db_size | 421.27 MiB
database_engine | sqlite
database_version | 3.45.3

</details>

### Supervisor diagnostics

[config_entry-hassio-1d4e9a0d489ccab7d923ba12156d2d67.json](https://github.com/user-attachments/files/16398949/config_entry-hassio-1d4e9a0d489ccab7d923ba12156d2d67.json)

### Additional information

_No response_
github-actions[bot] commented 2 weeks ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

Chaoscontrol commented 2 weeks ago

Still happening with last version of HA.

Same video elements of 6.5MB, ~600ms when using Frigate directly, ~2700ms when using it via Ingress with HAOS.

image

agners commented 2 weeks ago

Hm, so things did improve a bit. This is most likely due to faster aiohttp WS processing with the 3.10.x releases.

I guess a slight slowdown is kinda expected since we add some proxying with ingress. But the amount of slowdown does seem a bit excessive. Especially since this can be rather big transfers.

@bdraco do you have thoughts on this?

bdraco commented 2 weeks ago

aiohttp isn't well suited to be a proxy for high volume data. It simply isn't optimized for this use case. Someone would need to spend a few days/weeks optimizing it to improve the throughput, but even than it will always be slower since there is a cost to proxy the data. It will probably get a little bit better over time but a major improvement isn't likely without someone spending the time to optimize for this use case.

agners commented 2 weeks ago

Hm, long term, maybe it would be a better solution to setup a Nginx or something to handle ingress? :thinking:

bdraco commented 2 weeks ago

Hm, long term, maybe it would be a better solution to setup a Nginx or something to handle ingress? 🤔

Yes that would be a lot faster solution for sure

bdraco commented 2 weeks ago

There is also a chance the .ts files are being incorrectly compressed by ingress which would make everything a bit slower

bdraco commented 2 weeks ago

Also on the core side we use https://github.com/home-assistant/core/blob/68d6f1c1aab2cf80b51191f2c617c9a7fc81aa7a/homeassistant/components/hassio/ingress.py#L212

on supervisor its 4096 chunks https://github.com/home-assistant/supervisor/blob/91a8fae9b5f2441d138d762ba799e33fb7ff2c10/supervisor/api/ingress.py#L281 which is going to be a lot slower for larger payloads