[BUG] DB downloading failed (new MaxMind policy 2024)

[cwilson1776]

Is there an existing issue for this?

• [X] I have searched the existing issues

833

but it was closed without resolution

Name of mod

swag-maxmind

Name of base container

swag

Current Behavior

swag | Downloading GeoIP2 City database. swag | tar: short read

Expected Behavior

swag | Downloading GeoIP2 City database. (success)

Steps To Reproduce

Configured maxmind as directed in compose.yml: DOCKER_MODS: linuxserver/mods:swag-maxmind MAXMINDDB_LICENSE_KEY: xxxxxx docker compose up -d swag docker compose logs -f swag shows that the download failed. It appears that MaxMind has recently (2024.02.07) changed their download policy, at least for new users(?):

We will begin enforcing our API endpoint policies which could cause breaking changes if you you have not configured your integration correct. Read more on our release note

From there, an additional link says that you now need to use a curl command similar to this:

curl -O -J -L -u YOUR_ACCOUNT_ID:YOUR_LICENSE_KEY 'https://download.maxmind.com/geoip/databases/GeoIP2-City-CSV/download?suffix=zip'

rather than what swag-maxmind uses:

curl --silent https://download.maxmind.com/app/geoip_download?edition_id=${EDITION_ID}&license_key=YOUR_LICENSE_KEY&suffix=tar.gz" -o "$TMPDIR/${filename}.tar.gz"

I'm not sure how to update the repo here to effect the following changes, but by making these within the container I was able to successfully download:

1. Modified the following lines in /etc/libmaxminddb.cron.conf

# MAXMINDDB_URL="https://download.maxmind.com/geoip/databases"

MAXMINDDB_USER_ID="..."

2. Modified /etc/periodic/weekly/libmaxminddb

[ -z "$MAXMINDDB_URL" ] && MAXMINDDB_URL="https://download.maxmind.com/geoip/databases"

[ -z "$MAXMINDDB_USER_ID" ] && { echo "No MaxMind user id found; exiting. Please enter your user id into /etc/libmaxminddb.cron.conf"; exit 1; }

...

curl --silent -L -J -u "${MAXMINDDB_USER_ID}:${MAXMINDDB_LICENSE_KEY}" "${MAXMINDDB_URL}/${EDITION_ID}/download?suffix=tar.gz" -o "$TMPDIR/${filename}.tar.gz"

3. Then, running /etc/periodic/weekly/libmaxminddb from within the container successfully downloaded the db.

Environment

- OS: VERSION="22.04.4 LTS (Jammy Jellyfish)"
- How docker service was installed: distro package manager

CPU architecture

x86-64

Docker creation

services:
  swag:
    image: lscr.io/linuxserver/swag:latest
    container_name: swag
    networks:
      - frontend
      - frontend_ip6
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    environment:
      <<: *default-tz-puid-pgid
      URL: XXXXXXXXXXXXXXX
      SUBDOMAINS: XXXXXXXXXXXXX
      VALIDATION: dns
      DNSPLUGIN: cloudflare
      PROPAGATION: 60
      DOCKER_MODS: linuxserver/mods:swag-maxmind
      MAXMINDDB_LICENSE_KEY: XXXXXXXXXXXXXXXXXXXXXXXX
    volumes:
      - $DOCKERDIR/appdata/swag/config:/config
    ports:
      - "443:443"
      - "80:80"
      - "25565:25565/tcp"

Container logs

[mod-init] Running Docker Modification Logic
[mod-init] Adding linuxserver/mods:swag-maxmind to container
[mod-init] Downloading linuxserver/mods:swag-maxmind from lscr.io
[mod-init] Installing linuxserver/mods:swag-maxmind
[mod-init] linuxserver/mods:swag-maxmind applied to container
[migrations] started
[migrations] 01-nginx-site-confs-default: skipped
[migrations] done
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support the app dev(s) visit:
Certbot: https://supporters.eff.org/donate/support-work-on-certbot

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1000
User GID:    1000
───────────────────────────────────────

using keys found in /config/keys
Variables set:
PUID=1000
PGID=1000
TZ=US/Eastern
URL=XXXXXXXXX
SUBDOMAINS=XXXXXXX
EXTRA_DOMAINS=
ONLY_SUBDOMAINS=false
VALIDATION=dns
CERTPROVIDER=
DNSPLUGIN=cloudflare
EMAIL=
STAGING=

Using Let's Encrypt as the cert provider
SUBDOMAINS entered, processing
Sub-domains processed are: XXXXXXXXXXXXXXXXXXXXXXXXX
No e-mail address entered or address invalid
dns validation via cloudflare plugin is selected
Certificate exists; parameters unchanged; starting nginx
The cert does not expire within the next day. Letting the cron script handle the renewal attempts overnight (2:08am).
**** adding libmaxminddb to package install list ****
[mod-init] **** Installing all mod packages ****
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/1) Installing libmaxminddb (1.7.1-r2)
Executing busybox-1.36.1-r15.trigger
OK: 201 MiB in 216 packages
Applying the maxmind mod...
Downloading GeoIP2 City database.
tar: short read
Applied the maxmind mod
[custom-init] No custom files found, skipping...
[ls.io-init] done.
Server ready

[aptalca]

Please report upstream.

Alpine controls the db updates

https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/main/libmaxminddb/libmaxminddb.cron?ref_type=heads

[cwilson1776]

Reported: aports#15813

[cwilson1776]

PR committed upstream !61320

I'm not sure what the process is from here: how long until a package with this fix is available on the alpine mirrors, how the swag-maxmind mod picks up that new package... and if any documentation updates are needed in this README.md...

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[DougEdey]

I hit this too, as a workaround, I manually downloaded the GeoLite2-City database to the shared config directory (/config/geoip2db) and made sure it was extracted as GeoLite2-City.mmdb

That at least got my swag-maxmind setup working

[tanmaychimurkar]

I hit this too, as a workaround, I manually downloaded the GeoLite2-City database to the shared config directory (/config/geoip2db) and made sure it was extracted as GeoLite2-City.mmdb

Hey I did the same, however, I noticed that the GeoLite2-City.mmdb does not auto reload after the duration mentioned in auto_reload parameter of maxmind.conf. Can you please confirm if this is the case for you as well?

Thanks

[tanmaychimurkar]

Hey just found out from Maxmind's documentation that they have put up a section for automated updating of their databases: https://dev.maxmind.com/geoip/updating-databases

Linking maxmind's GitHub repository for automatic database updates if anyone stumbles upon the same issue: https://github.com/maxmind/geoipupdate

We can map the volume to wherever swag maps /config/geoip2db/ folder on host machine. Hope this helps!!

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[Crash1602]

With the current swag container 2.10.0 and maxmind-mod, the problem (tar: short read) still exists. The only thing that helps is the workaround, the manual download.

libmaxminddb goaccess already installed, skipping swag-auto-reload deps already installed, skipping libmaxminddb already installed, skipping libmaxminddb already installed, skipping Applied the SWAG dashboard mod Applying the maxmind mod... Downloading GeoIP2 City database. tar: short read Applied the maxmind mod [custom-init] No custom files found, skipping... MOD Auto-reload: Watching the following files/folders for changes (excluding .sample and .md files): /config/nginx [ls.io-init] done. nginx: [emerg] MMDB_open("/config/geoip2db/GeoLite2-City.mmdb") failed - Error opening the specified MaxMind DB file in /config/nginx/maxmind.conf:1

Server ready nginx: [emerg] MMDB_open("/config/geoip2db/GeoLite2-City.mmdb") failed - Error opening the specified MaxMind DB file in /config/nginx/maxmind.conf:1

[aptalca]

Alpine's libmaxmind package was only fixed in 3.20. The issue will persist until SWAG is rebased on alpine 3.20, which will be soon

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[IBeTheBlueCat]

I can confirm that this issue is resolved in the latest version of SWAG as they rebased to Alpine 3.2, this resolved my issue with the exact same symptoms as above

[Crash1602]

I can also confirm that the error has been fixed with version 2.11.0-ls312 (Alpine 3.20).

[tanmaychimurkar]

Can confirm the same.

[DougEdey]

@cwilson1776 can you confirm this works for you too and close this issue?