API Key is rate-limited of there are manual mod packs to install

[camalot]

Describe the problem

If there are manual files that need to be downloaded, and it fails to start over and over due to this, eventually CurseForge ends up complaining about the API key.

This is likely due to API throttling.

Container definition

minecraft-curseforge:
    hostname: minecraft-curseforge
    container_name: minecraft-curseforge
    image: itzg/minecraft-server:java17-alpine
    networks:
    - internal
    ports: 
    - 25565:25565
    - 25575:25575
    restart: unless-stopped
    dns:
    - 192.168.2.7
    - 192.168.2.1
    - 1.1.1.1
    - 1.0.0.1
    - 8.8.8.8
    - 8.8.4.4
    dns_search: xxxxxxxxxxxx

    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    deploy:
      resources:
        limits:
          memory: 24g
        reservations:
          memory: 24g
    environment:

      TYPE: AUTO_CURSEFORGE
      CF_API_KEY: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
      CF_SLUG: all-the-mods-9
      VERSION: 1.20.1
      FORGEVERSION: 47.1.3
      CF_FORCE_SYNCHRONIZE: true

      ENABLE_AUTOPAUSE: "TRUE"
      AUTOPAUSE_KNOCK_INTERFACE: 'eth0'

      SEED: '6960347577965979123'
      VIEW_DISTANCE: 6

      ICON: "https://i.imgur.com/RV1PBPD.png"
      OVERRIDE_ICON: 'TRUE'

      OPS_FILE: 'http://xxxxxxxxxxxxxxxxxxxxx/minecraft/ops.json'
      WHITELIST_FILE: 'http://xxxxxxxxxxxxxxxxxxx/minecraft/whitelist.json'
      ENFORCE_WHITELIST: 'true'
      OVERRIDE_WHITELIST: 'true'

      REMOVE_OLD_MODS: 'false'

      LEVEL: taco_atm9

      MEMORY: 24G

      ENABLE_RCON: 'true'
      RCON_PASSWORD: 'xxxxxxxxxxxxxxxxxxxxxxxxxx'
      BROADCAST_RCON_TO_OPS: 'true'

      ALLOW_NETHER: 'true'
      ANNOUNCE_PLAYER_ACHIEVEMENTS: 'true'
      STOP_SERVER_ANNOUNCE_DELAY: 60
      STOP_DURATION: 60

      MOTD: The §l§6TACO§r §cMinecraft§r Server
      MAX_PLAYERS: 20
      SNOOPER_ENABLED: 'false'
      HARDCORE: 'false'
      DIFFICULTY: easy
      GENERATE_STRUCTURES: 'true'
      MAX_BUILD_HEIGHT: 256
      SPAWN_ANIMALS: 'true'
      SPAWN_NPCS: 'true'
      SPAWN_MONSTERS: 'true'
      SPAWN_PROTECTION: 128

      MODE: survival
      PVP: 'false'

      LEVEL_TYPE: minecraft:large_biomes
      ALLOW_FLIGHT: 'true'
      USE_MODPACK_START_SCRIPT: 'true'
      ENABLE_ROLLING_LOGS: 'true'
      MAX_TICK_TIME: "-1"
      GUI: 'FALSE'
      UID: 1000
      GID: 1000
      TZ: 'America/Chicago'
      EULA: 'true'
      tty: 'true'
      stdin_open: 'true'
    volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /mnt/container_data/minecraft/curseforge/modpacks:/modpacks
    - /mnt/container_data/minecraft/curseforge/data:/data
    - /mnt/data/Backup/minecraft/backups:/data/simplebackups/
    - /mnt/data/Backup/minecraft/downloads/atm9:/downloads

Container logs

2024-02-08T21:15:25.986079005Z Mod           Version Name              Filename                            Download page                                                             
2024-02-08T21:15:25.986084653Z ============  ========================  ==================================  ==========================================================================
2024-02-08T21:15:25.986086531Z Packet Fixer  Packet Fixer Forge 1.2.1  PacketFixer-forge-1.2.1-1.20.1.jar  https://www.curseforge.com/minecraft/mc-mods/packet-fixer/download/4884233
2024-02-08T21:15:26.314650518Z [init] ERROR failed to auto-install CurseForge modpack
2024-02-08T21:15:29.112547209Z [init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x 26 1000 1000 4096 Feb  8 11:56 /data'
2024-02-08T21:15:29.114804836Z [init] Autopause functionality enabled
2024-02-08T21:15:30.375082613Z [mc-image-helper] 15:15:30.373 INFO  : Requested force synchronize of All the Mods 9-0.2.43
2024-02-08T21:15:31.397842079Z [mc-image-helper] 15:15:31.397 INFO  : Processing modpack 'All the Mods 9-0.2.43' (all-the-mods-9) @ 715572:5089637
2024-02-08T21:16:01.507659608Z [mc-image-helper] 15:16:01.507 ERROR : Invalid parameter provided for 'install-curseforge' command: Access to https://api.curseforge.com is forbidden or rate-limit has been exceeded. Ensure CF_API_KEY is set to a valid API key from https://console.curseforge.com/ or allow rate-limit to reset.
2024-02-08T21:16:01.835918037Z [init] ERROR failed to auto-install CurseForge modpack
2024-02-08T21:16:03.299143347Z [init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x 26 1000 1000 4096 Feb  8 11:56 /data'
2024-02-08T21:16:03.301516152Z [init] Autopause functionality enabled

[CsiPA0723]

Having the same issue with a private modpack.

Container definition

version: "3.8"

services:
  mc_helltime:
    image: itzg/minecraft-server
    container_name: mc_helltime
    tty: true
    stdin_open: true
    ports:
      - "25565:25565"
    environment:
      EULA: "TRUE"
      # allocate from https://console.curseforge.com/ and set in .env file
      CF_API_KEY: ${CF_API_KEY}
      TYPE: AUTO_CURSEFORGE
      CF_MODPACK_MANIFEST: /manifests/manifest.json
      CF_SLUG: "custom"
      REMOVE_OLD_MODS: "TRUE"
      ENABLE_ROLLING_LOGS: "true"
      MEMORY: "4G"
      OVERRIDE_SERVER_PROPERTIES: "true"
      DIFFICULTY: "normal"
      SPAWN_PROTECTION: "4"
      VIEW_DISTANCE: "10"
      ALLOW_FLIGHT: "TRUE"
      USE_NATIVE_TRANSPORT: "TRUE"
      EXEC_DIRECTLY: "TRUE"
      MOTD: "A not so hard modpack"
      CF_EXCLUDE_MODS: |
        ding
        controlling
        tips
        entity-model-features
        entity-texture-features-fabric
        extreme-sound-muffler
        ambientsounds
        overflowing-bars
        better-fps-render-distance
        yeetusexperimentus
        embeddium
        embeddiumplus
        textrues-embeddium-options
        tool-stats
        journeymap-integration
        inventory-hud-forge
        toast-control
        clean-tooltips
        bad-wither-no-cookie-reloaded
        seasonhud
        model-gap-fix
        entityculling
        distant-horizons
        betterf3
        fix-gpu-memory-leak
      CF_FORCE_SYNCHRONIZE: "TRUE"
    restart: unless-stopped
    volumes:
      # attach the relative directory 'data' to the container's /data path
      - ./data:/data
      - ./manifests:/manifests:ro

volumes:
  data: {}

Container logs

mc_helltime  | [init] Running as uid=1000 gid=1000 with /data as 'drwxr-xr-x 2 1000 1000 4096 Feb  8 21:23 /data'
mc_helltime  | [mc-image-helper] 21:23:56.268 ERROR : Invalid parameter provided for 'install-curseforge' command: Access to https://api.curseforge.com is forbidden or rate-limit has been exceeded. Ensure CF_API_KEY is set to a valid API key from https://console.curseforge.com/ or allow rate-limit to reset.
mc_helltime  | [init] ERROR failed to auto-install CurseForge modpack

[itzg]

I highly recommend you remove the restart policy parameter until you have your container setup initially. Otherwise, you will exceed the rate limit that CurseForge imposes.

[Shawak]

Pretty sure its sends it's rate limit in the http response, so maybe we can add delays so we dont run into the timeout?

[itzg]

Pretty sure its sends it's rate limit in the http response, so maybe we can add delays so we dont run into the timeout?

@Shawak How are you pretty sure about that? Can you provide a network capture?

[itzg]

...OK, I grabbed a new capture and only the body seems to indicate the rate limit. I was previously looking only at the headers, since that's where APIs like Github's put rate limit information:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: gj-mv1BaNzWG0vrGDCer8ewJ4DH8vHbz_sbrjfewGW6SZ23aulBcKw==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

[Shawak]

Ye.. checked it myself. Can't find any rate limit headers in the response, its pretty weird to not send them there. Probably they just set a rate limit via cloudfront, guess we have to guess then, kinda wild ngl

[itzg]

I'll add a conditional on a phrase in the response body. It's a little cheesy and brittle, but better than being totally vague with a failure reason.

[TrueOsiris]

Just my 2 cents, ignore if unimportant: make sure your .env file is in this format: CF_API_KEY='yourapikeywith$^""specialsymbols' single quotes, or dollar signs will be interpreted as variable-in-variable.

[itzg]

Thanks @TrueOsiris that would be easier than the instructions I currently provide to double up the dollar signs. I hadn't thought about looking here instead of here.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Please add a comment describing the reason to keep this issue open.