Improve handling of 429 status code from playerdb API

[Ahptt]

Describe the problem

My docker-compose.yml was working on 1.19.2. Upon attempting to update to version 1.19.3, I receive an error with docker adding users to the whitelist. I tried again on 1.19.2 after discovering this and the whitelist fails to compose properly. I do use a firewall, but I've verified that there isn't anything being blocked, and per the error message, it seems that I'm sending too many requests?

Container definition

version: "3.8"
services:
  mc:
    image: itzg/minecraft-server
    environment:
      EULA: "true"
      TYPE: "quilt"
      VERSION: "1.19.3"
      MEMORY: 64G
#      SEED:
      OPS: "Ahpt"
      DIFFICULTY: "hard"
      MAX_PLAYERS: "1487"
      VIEW_DISTANCE: "12"
      SIMULATION_DISTANCE: "6"
      ENABLE_WHITELIST: "true"
#      ENFORCE_WHITELIST: "true"
      ENFORCE_SECURE_PROFILE: "false"
      WHITELIST: "Ahpt"
#      MOTD: ""
      SPAWN_PROTECTION: "-1"
#      NETWORK_COMPRESSION_THRESHOLD: "256"
      DATAPACKS: /datapacks
#      MODRINTH_PROJECTS: alternate-current,c2me-fabric,debugify,ferrite-core,krypton,lazydfu,lithium,memoryleakfix,no-shield-delay,servercore,smoothboot-fabric,simple-voice-chat,sound-physics-remastered,spark,starlight,voice-chat-interaction
#      MODRINTH_DOWNLOAD_OPTIONAL_DEPENDENCIES: "FALSE"
#      MODRINTH_ALLOWED_VERSION_TYPE: "release"
      RCON_CMDS_STARTUP:  |-
        gamerule doFireTick false
        gamerule disableElytraMovementCheck true
        gamerule doImmediateRespawn true
        gamerule showDeathMessages true
      ENABLE_AUTOPAUSE: "true"
      MAX_TICK_TIME: "-1"
    ports:
      - "25565:25565"
      - "24454:24454/udp"
    volumes:
      - ./skysmp:/data
      - ./mods:/mods
      - ./datapacks:/datapacks
      - ./config:/config
    restart: unless-stopped
volumes:
  data: {}

Container logs

Recreating skysmp_mc_1 ... done
Attaching to skysmp_mc_1
mc_1  | [init] Changing ownership of /data to 1000 ...
mc_1  | [init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x 1 1000 1000 0 Dec 14 23:20 /data'
mc_1  | [init] Resolved version given 1.19.3 into 1.19.3 and major version 1.19
mc_1  | [init] Autopause functionality enabled
mc_1  | [init] Starting RCON commands
mc_1  | [init] Rcon cmds functionality enabled
mc_1  | [init] Resolving type given quilt
mc_1  | [init] Checking Quilt Installer version information.
mc_1  | [init] Checking Quilt Loader version information.
mc_1  | [init] Downloading and installing https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-installer/0.5.0/quilt-installer-0.5.0.jar ...
mc_1  | Installing server launcher at: .
mc_1  | Installing server launcher for 1.19.3 with loader 0.17.8
mc_1  | Installing libraries
mc_1  | Downloading library at: https://maven.fabricmc.net/net/fabricmc/tiny-mappings-parser/0.3.0+build.17/tiny-mappings-parser-0.3.0+build.17.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/net/fabricmc/sponge-mixin/0.11.4+mixin.0.8.5/sponge-mixin-0.11.4+mixin.0.8.5.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/net/fabricmc/tiny-remapper/0.8.6/tiny-remapper-0.8.6.jar
mc_1  | Downloading library at: https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-loader/0.17.8/quilt-loader-0.17.8.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/net/fabricmc/access-widener/2.1.0/access-widener-2.1.0.jar
mc_1  | Downloading library at: https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-json5/1.0.2/quilt-json5-1.0.2.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/org/ow2/asm/asm/9.3/asm-9.3.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/org/ow2/asm/asm-analysis/9.3/asm-analysis-9.3.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/org/ow2/asm/asm-commons/9.3/asm-commons-9.3.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/org/ow2/asm/asm-tree/9.3/asm-tree-9.3.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/org/ow2/asm/asm-util/9.3/asm-util-9.3.jar
mc_1  | Downloading library at: https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-config/1.0.0-beta.6/quilt-config-1.0.0-beta.6.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/net/fabricmc/intermediary/1.19.3/intermediary-1.19.3.jar
mc_1  | Downloading library at: https://maven.fabricmc.net/net/fabricmc/intermediary/1.19.3/intermediary-1.19.3.jar
mc_1  | Downloading server
mc_1  | Downloading 1.19.3 server jar from https://piston-data.mojang.com/v1/objects/c9df48efed58511cdd0213c56b9013a7b5c9ac1f/server.jar
mc_1  | [init] Copying datapacks from /datapacks ...
mc_1  | [init] Copying any mods over...
mc_1  | [mc-image-helper] 23:20:43.993 INFO  : Copying /mods/no-shield-delay-1.0.0.jar -> /data/mods/no-shield-delay-1.0.0.jar
mc_1  | [mc-image-helper] 23:20:44.003 INFO  : Copying /mods/voicechat-quilt-1.19.3-2.3.23.jar -> /data/mods/voicechat-quilt-1.19.3-2.3.23.jar
mc_1  | [mc-image-helper] 23:20:44.007 INFO  : Copying /mods/lazydfu-0.1.3.jar -> /data/mods/lazydfu-0.1.3.jar
mc_1  | [mc-image-helper] 23:20:44.007 INFO  : Copying /mods/qfapi-5.0.0-alpha.5_qsl-4.0.0-beta.2_fapi-0.68.1_mc-1.19.3.jar -> /data/mods/qfapi-5.0.0-alpha.5_qsl-4.0.0-beta.2_fapi-0.68.1_mc-1.19.3.jar
mc_1  | [mc-image-helper] 23:20:44.008 INFO  : Copying /mods/ferritecore-5.1.0-fabric.jar -> /data/mods/ferritecore-5.1.0-fabric.jar
mc_1  | [mc-image-helper] 23:20:44.009 INFO  : Copying /mods/anti-xray-1.2.7-Fabric-1.19.3.jar -> /data/mods/anti-xray-1.2.7-Fabric-1.19.3.jar
mc_1  | [mc-image-helper] 23:20:44.009 INFO  : Copying /mods/starlight-1.1.1+fabric.ae22326.jar -> /data/mods/starlight-1.1.1+fabric.ae22326.jar
mc_1  | [mc-image-helper] 23:20:44.010 INFO  : Copying /mods/memoryleakfix-1.19.3-0.7.0.jar -> /data/mods/memoryleakfix-1.19.3-0.7.0.jar
mc_1  | [mc-image-helper] 23:20:44.010 INFO  : Copying /mods/c2me-fabric-mc1.19.3-0.2.0+alpha.9.12.jar -> /data/mods/c2me-fabric-mc1.19.3-0.2.0+alpha.9.12.jar
mc_1  | [mc-image-helper] 23:20:44.011 INFO  : Copying /mods/krypton-0.2.1.jar -> /data/mods/krypton-0.2.1.jar
mc_1  | [mc-image-helper] 23:20:44.011 INFO  : Copying /mods/Debugify-1.19.3+1.0.jar -> /data/mods/Debugify-1.19.3+1.0.jar
mc_1  | [mc-image-helper] 23:20:44.012 INFO  : Copying /mods/smoothboot-fabric-1.19-1.7.1.jar -> /data/mods/smoothboot-fabric-1.19-1.7.1.jar
mc_1  | [mc-image-helper] 23:20:44.012 INFO  : Copying /mods/lithium-fabric-mc1.19.3-0.10.4.jar -> /data/mods/lithium-fabric-mc1.19.3-0.10.4.jar
mc_1  | [mc-image-helper] 23:20:44.013 INFO  : Copying /mods/vcinteraction-1.19.3-1.0.3.jar -> /data/mods/vcinteraction-1.19.3-1.0.3.jar
mc_1  | [mc-image-helper] 23:20:44.013 INFO  : Copying /mods/spark-1.10.17-fabric.jar -> /data/mods/spark-1.10.17-fabric.jar
mc_1  | [mc-image-helper] 23:20:44.015 INFO  : Copying /mods/alternate-current-mc1.19-1.4.0.jar -> /data/mods/alternate-current-mc1.19-1.4.0.jar
mc_1  | [init] Copying any configs from /config to /data/config
mc_1  | [init] Creating server properties in /data/server.properties
mc_1  | [init] Enabling whitelist functionality
mc_1  | [init] Updating ops
mc_1  | [mc-image-helper] 23:20:57.447 ERROR : 'get' command failed: Failed to download https://playerdb.co/api/player/minecraft/Ahpt: status code: 429, reason phrase: Too Many Requests caused by status code: 429, reason phrase: Too Many Requests
mc_1  | [init] Updating whitelist
mc_1  | [mc-image-helper] 23:21:09.712 ERROR : 'get' command failed: Failed to download https://playerdb.co/api/player/minecraft/Ahpt: status code: 429, reason phrase: Too Many Requests caused by status code: 429, reason phrase: Too Many Requests

[apfohl]

I'm also having the issue.

[apfohl]

I found out that the request to playerdb.co fails for some users on my server, not all:

if ! playerData=$(get "https://playerdb.co/api/player/minecraft/$i" | jq -re ".data.player"); then
  log "WARNING: Could not lookup user $i for ${output} addition"
else
  playerDataList=$playerDataList$(echo "$playerData" | jq -r "$userData")
fi

Therefore the error handling here is maybe not sufficient.

[itzg]

@Cherry, just wanted to let you know that users here are getting 429's (too many requests) from your service.

@apfohl agreed in principle, something should be improved on error handling, but 429's are a little tricky:

• Retry-After might be present, but what if it says try again in an hour. Would you like the container startup to be paused for an hour?
• If the Retry-After isn't present...how long should it wait to retry?
• Bailing out the startup with an error code and assuming container restart is enabled may not be best since the container startup would continue to hammer on an already overloaded endpoint

[Cherry]

@Cherry, just wanted to let you know that users here are getting 429's (too many requests) from your service.

Yeah 😢 Mojang are heavily rate limiting requests right now, and PlayerDB uses Cloudflare Workers so all of the requests come from the same IP. We're in contact with Mojang and are trying to find a solution that'll work for everyone, and are also looking at hosting a few instances elsewhere to utilize more IPs, but for the time being we're seeing an increasing number of 429s unfortunately.

[itzg]

Thanks for the super quick reply @Cherry . If you don't mind, can you post an update here as appropriate?

[Cherry]

Absolutely, I'll be sure to update here as soon as I have anything additional to share. Sorry for any inconvenience folks are experiencing because of this - it's impacting some of our internal tools too so we know the pain 😅

[Cherry]

We've made some changes which should drastically reduce the number of 429s seen for the time-being. We're continuing to try and work with Mojang to find a long-term solution.

[itzg]

@Cherry I have been getting several reports over last few days of 429 responses. One was to https://playerdb.co/api/player/minecraft/GnU , if that helps.

[Cherry]

Apologies for this, we've too been seeing the increase in 429s recently and are working to resolve it. The rate limits Mojang have in place are making this harder and harder 😢

[Cherry]

We've made some more changes that'll improve things again. Hopefully this will help a lot moving forward, thanks for your patience. #2342 seems like a great addition too!