Meilisearch rejects gpx file due to size

[garbast]

I have a gpx file that is 528,6 kB large. When i try to upload it, i get the following error. In the browser i only get an "Internal Error"

web-1     | SvelteKitError: Content-length of 528846 exceeds limit of 524288 bytes.
web-1     |     at Object.start (file:///app/build/handler.js:984:19)
web-1     |     at setupReadableStreamDefaultController (node:internal/webstreams/readablestream:2333:23)
web-1     |     at setupReadableStreamDefaultControllerFromSource (node:internal/webstreams/readablestream:2366:3)
web-1     |     at new ReadableStream (node:internal/webstreams/readablestream:289:7)
web-1     |     at get_raw_body (file:///app/build/handler.js:973:9)
web-1     |     at getRequest (file:///app/build/handler.js:1052:7)
web-1     |     at Array.ssr (file:///app/build/handler.js:1230:19)
web-1     |     at handle (file:///app/build/handler.js:1300:23)
web-1     |     at file:///app/build/handler.js:1300:40
web-1     |     at Array.<anonymous> (file:///app/build/handler.js:1219:4) {
web-1     |   status: 413,
web-1     |   text: 'Payload Too Large'
web-1     | }

Rename to .gpx without .txt to test with. 2021-10-24_536064034_Essen-Mitte nach Bochum-Hauptbahnhof.gpx.txt

[Flomp]

Turns out SvelteKit has a body size limit that I didn't know about: https://kit.svelte.dev/docs/adapter-node#environment-variables-bodysizelimit

You should be able to add a BODY_SIZE_LIMIT: Infinity to the environment section of the wanderer-web section in your docker-compose.yml.

I'm currently not at my computer to test it. I'll add it to the docs later tonight if it works.

Thanks for testing!

[garbast]

With the env parameter set i still get an error

{
    "url": "http://db:8090/api/collections/trails/records/4skbnicy838t48a",
    "status": 400,
    "response": {
        "code": 400,
        "message": "Something went wrong while processing your request.",
        "data": {
            "photos": {
                "code": "validation_file_size_limit",
                "message": "Failed to upload \"20240309_143022.jpg\" - the maximum allowed file size is 5242880 bytes."
            }
        }
    },
    "isAbort": false,
    "originalError": {
        "url": "http://db:8090/api/collections/trails/records/4skbnicy838t48a",
        "status": 400,
        "data": {
            "code": 400,
            "message": "Something went wrong while processing your request.",
            "data": {
                "photos": {
                    "code": "validation_file_size_limit",
                    "message": "Failed to upload \"20240309_143022.jpg\" - the maximum allowed file size is 5242880 bytes."
                }
            }
        }
    },
    "name": "ClientResponseError 400"
}

[asitemade4u]

Same here but BODY_SIZE_LIMIT: Infinity fixed it.

[garbast]

Updated to latest version and executed a shell in the web container. There i have these env variables present:

UPLOAD_PASSWORD=
PUBLIC_DISABLE_SIGNUP=true
NODE_VERSION=18.20.1
HOSTNAME=28cc2c31f9cb
ORIGIN=http://localhost:3000
YARN_VERSION=1.22.19
UPLOAD_USER=
SHLVL=2
HOME=/root
PUBLIC_POCKETBASE_URL=http://db:8090
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BODY_SIZE_LIMIT=Infinity
MEILI_MASTER_KEY=<replaced>
MEILI_URL=http://search:7700
PWD=/app
NODE_ENV=production

But still the upload failes with the error validation_file_size_limit. Where does the BODY_SIZE_LIMIT need to be set in the docker-compose.yaml?

[Flomp]

Ah, I see what's happening. The good news is you solved the frontend file size issue. The bad news is you are running into the backend file size limit. Currently, uploads are capped at 5MB for both photos and gpx files in PocketBase.

You can adapt the limit by making the PocketBase container publically available and navigating to the PocketBase admin panel at http://:8090/_/. From there you can edit the respective fields in the trails collection to increase the file size.

Unfortunately, I currently cannot find a way to make this limit into an environment variable without rewriting a considerable amount of PocketBase's code. So I'm not sure if the update will stick when you update the wanderer-db container.

[garbast]

For now i wouldnt go so fare to modify PocketBase. That's something for Version 2.x ;)

But this is the ideal moment for me to suggest some paradigm change. Would it hurt to much to store images and gpx files individually in the file system?

This would have the benefit, that the database would not get out of hand. Imagine you have 200 trails after some years and have added a lot of images too. Then every backup would be very large because all of the images get backuped again and again. Having the images separate, they only get backuped once.

[Flomp]

I politely disagree with the suggested paradigm change. PocketBase stores the database and the stored data separately. You can check that for yourself: in your pb_data folder you will find a data.db file containing the database and a storage folder with your gpx files and images. Consequently, it is easy to backup the data separately from the database. Incremental backups should also be unproblematic as the data in the storage folder does not change even if the associated record in the database is updated (unless you swap the file, of course).

[garbast]

I'm sorry. I was wrong. Didn't knew that PocketBase is storing files like that. So this is sorted but the upload limit problem remains and needs to be fixed on a later day.

I found in an ticket (https://github.com/pocketbase/pocketbase/issues/4067) that Migrations can change the MaxSize. Would it be possible to use environment variables in the migrations?

I cant see the flomp/wanderer-db repository, so i can only make assumptions.