Docker-compose first setup SyntaxError Unexpected token < in JSON at position 0

EinAeffchen commented 6 months ago

Hey, so I tried setting up your system using the docker-compose guide. And I got the system running for a short while and can open it in the browser, however there is nothing to see and checking the logs I get the following error:


> [2024-05-08 08:32:47.631]: index Updating file index for directory /data/Pictures
> [2024-05-08 08:32:47.641]: index.filter.maxFilesize Limit files up to 20M (20971520 bytes)
> [2024-05-08 08:32:47.642]: index.filter.limit Index has 0 entries. Set index limit to max 10 new entries
> [2024-05-08 08:32:47.673]: index.filter.limit Index limit of 10 exceeded. No more new files are added to the file index.
> [2024-05-08 08:32:47.675]: index.create Read 10 files in /data/Pictures
> [2024-05-08 08:32:47.679]: index.update Initiate index with 10 entries
> [2024-05-08 08:32:47.687]: index.checksum Calculating ids for 9 entries with 15.1MB of total size 15.1MB (100.0%)
> [2024-05-08 08:32:47.994]: index.checksum All ids of 15.1MB are calculated. Calculated ids of 15.1MB (100.0%)
> [2024-05-08 08:32:48.004]: index File index was saved to /data/config/Pictures.idx and 9 entries have new checkums/ids
> [2024-05-08 08:32:48.009]: index.journal Journal /data/config/Pictures.idx.0508-3aBB.journal created
> [2024-05-08 08:32:48.009]: index Updated file index for directory /data/Pictures
> [2024-05-08 08:32:48.028]: cli.spawn Run cli with extract --index /data/config/Pictures.idx --journal 0508-3aBB
> [2024-05-08 08:32:49.495]: index.readStream Reading file index journal /data/config/Pictures.idx.0508-3aBB.journal
> [2024-05-08 08:32:49.942]: extractor.image.resize Use sharp to resize images
> [2024-05-08 08:32:51.478]: server.api.events Add new client 6b7c0677-06ca-4c7c-917f-c803792574eb
> [2024-05-08 08:32:51.747]: extractor.image.preview Created 4 image previews from c6a146b:Pictures:2024/DSC01171.png with sizes of 1280,800,320,128
> [2024-05-08 08:32:52.068]: server.request 200 POST /embeddings 21ms
> undefined:1
> <!DOCTYPE html>

SyntaxError: Unexpected token < in JSON at position 0 at JSON.parse () at /app/node_modules/@home-gallery/storage/dist/write-storage-file.js:16:30 at /app/node_modules/@home-gallery/common/dist/fs/write-safe.js:15:7 at FSReqCallback.oncomplete (node:fs:192:23)

Node.js v18.20.2`

Logs from the gallery.log:

{"level":30,"time":"2024-05-08T08:32:52.068Z","pid":17,"hostname":"82615049baf7","module":"server.request","req":{"id":2,"method":"POST","url":"/embeddings","query":{},"params":{},"headers":{"host":"<censored>","content-length":"45167","content-type":"image/jpeg","x-forwarded-for":"172.18.0.1","x-forwarded-host":"<censored>","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"47cd60ed823f","x-real-ip":"172.18.0.1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.6","remotePort":55940},"levelName":"info","res":{"statusCode":200,"headers":{"access-control-allow-origin":"*","accept-ranges":"bytes","cache-control":"public, max-age=0","last-modified":"Tue, 23 Apr 2024 03:50:15 GMT","etag":"W/\"13bd-18f09135cd8\"","content-type":"text/html; charset=UTF-8","vary":"Accept-Encoding","content-encoding":"gzip"}},"responseTime":20,"msg":"200 POST /embeddings 21ms"}
{"level":20,"time":"2024-05-08T08:32:52.209Z","pid":28,"hostname":"82615049baf7","module":"cli.spawn","levelName":"debug","spawn":{"env":{"GALLERY_CONFIG":"/data/config/gallery.config.yml","GALLERY_LOG_LEVEL":"trace","GALLERY_LOG_JSON_FORMAT":"true"},"command":"/usr/local/bin/node","args":["/app/gallery.js","extract","--index","/data/config/Pictures.idx","--journal","0508-3aBB"],"pid":50,"code":1,"signal":null,"cmd":"GALLERY_CONFIG=/data/config/gallery.config.yml GALLERY_LOG_LEVEL=trace GALLERY_LOG_JSON_FORMAT=true /usr/local/bin/node /app/gallery.js extract --index /data/config/Pictures.idx --journal 0508-3aBB"},"duration":4181,"msg":"Executed cmd GALLERY_CONFIG=/data/config/gallery.config.yml GALLERY_LOG_LEVEL=trace GALLERY_LOG_JSON_FORMAT=true /usr/local/bin/node /app/gallery.js extract --index /data/config/Pictures.idx --journal 0508-3aBB"}
{"level":50,"time":"2024-05-08T08:32:52.210Z","pid":28,"hostname":"82615049baf7","module":"cli.task.import","levelName":"error","code":1,"signal":null,"msg":"Failed to import: [object Object]"}
{"level":30,"time":"2024-05-08T08:32:52.210Z","pid":28,"hostname":"82615049baf7","module":"cli.task.import","levelName":"info","msg":"Stop watching files due error"}
{"level":50,"time":"2024-05-08T08:32:52.227Z","pid":28,"hostname":"82615049baf7","module":"cli.run","levelName":"error","code":1,"signal":null,"msg":"Error: [object Object]"}
{"level":20,"time":"2024-05-08T08:32:52.257Z","pid":17,"hostname":"82615049baf7","module":"server.cli","levelName":"debug","spawn":{"env":{"GALLERY_CONFIG":"/data/config/gallery.config.yml","GALLERY_LOG_LEVEL":"trace","GALLERY_LOG_JSON_FORMAT":"true"},"command":"/usr/local/bin/node","args":["/app/gallery.js","run","import","--initial","--update","--watch"],"pid":28,"code":0,"signal":null,"cmd":"/usr/local/bin/node /app/gallery.js run import --initial --update --watch"},"duration":5657,"msg":"Cli run import --initial --update --watch exited with code 0"}

I am mounting a samba volume onto my machine and then mount it into docker. Though I don't assume that this is causing the problem as its able to see and process the images for a bit. My docker-compose for reference:


version: "3.6"
services:
  api:
    # custom build via
    #build: packages/api-server
    image: xemle/home-gallery-api-server
    environment:
      # TensorflowJS backends
      # - cpu: slowest and best support
      # - wasm: good perfromance for arm64 and amd64 platforms
      # - node: best performance on amd64 platform
      #- BACKEND=cpu
      - BACKEND=wasm
      #- BACKEND=node
  gallery:
    # custom build via
    #build: .
    image: xemle/home-gallery
    environment:
      - GALLERY_API_SERVER=<censored>
      #- GALLERY_API_SERVER_CONCURRENT=1 # On low powered devices
      #- GALLERY_API_SERVER_TIMEOUT=60 # On low powered devices
      - GALLERY_OPEN_BROWSER=false
      # Use polling for safety of possible network mounts. Try 0 to use inotify via fs.watch
      - GALLERY_WATCH_POLL_INTERVAL=300
      - PID=1000
      - GID=1000
    volumes:
      - ./data:/data
      # Mount your media directories below /data
      - /mnt/hdd/photos:/data/Pictures
    ports:
      - "3000:3000"
    entrypoint: ['node', '/app/gallery.js']
    command: ['run', 'server']

xemle commented 6 months ago

Hi @EinAeffchen

Thank you for using HomeGallery and reporting your issue here with detailed logs.

It seems that the api server has an invalid response which is not a json format.

What is your api server settings, which is apparently <censored> in the provided info. It should be http://api:3000 which than uses the api container of your docker compose setting.

If you use a different api server, please check that the responses are valid.

EinAeffchen commented 6 months ago

Thanks for the quick help! You're right I mistakenly put my url in there instead of the api url. However, I realized I can't get the API Image to run, it errors with the following logs, independent of the backend I set via env:

node[1]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xb87bc0 node::Abort() [node]
 2: 0xb87c3e  [node]
 3: 0xbf50fe  [node]
 4: 0xbf51e1 node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xb458f3 node::InitializeOncePerProcess(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, node::ProcessFlags::Flags) [node]
 6: 0xb45f4b node::Start(int, char**) [node]
 7: 0x7fd56f3a01ca  [/lib/x86_64-linux-gnu/libc.so.6]
 8: 0x7fd56f3a0285 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0xac3f4e _start [node]

Running the container manually via docker run and adding --priviliged makes it look like non of the Backends are known (same error independent of set backend:

[eval]:1
BACKEND=node
^

[eval]:1
BACKEND=cpu
^

ReferenceError: cpu is not defined
    at [eval]:1:1
    at Script.runInThisContext (node:vm:123:12)
    at Object.runInThisContext (node:vm:299:38)
    at node:internal/process/execution:82:21
    at [eval]-wrapper:6:24
    at runScript (node:internal/process/execution:81:62)
    at evalScript (node:internal/process/execution:103:10)
    at node:internal/main/eval_string:29:3

Node.js v18.18.0

Command used: docker run --privileged -it xemle/home-gallery-api-server -e BACKEND=cpu

xemle commented 6 months ago

Hi @EinAeffchen

IMHO --privileged does not need to be set. Is there a reason not to run the default wasm backend for the moment?

Since the errors are provided with some context, could you provide more context such host machine, architecture, OS and your docker commands to trigger these errors? Otherwise it is difficult to follow, to validate and provide further help.

EinAeffchen commented 6 months ago

Hi @xemle, I assumed so too, but didn't want to leave anything untried. With independent of the backend, I mean that no matter which one I choose (wasm, node, cpu) it's always the same error.

The machine I'm running this on:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

docker-compose --version
Docker Compose version v2.27.0

Docker version 20.10.7, build f0df350

Otherwise I started the system with the above docker-compose running the commands as described in your guide:

mkdir -p data

echo "CURRENT_USER=$(id -u):$(id -g)" >> .env

docker compose run gallery run init --source /data/Pictures

docker compose up -d

xemle commented 6 months ago

Hi @EinAeffchen

thank you for the details. Currently I do not have a clue why the gallery on your side stuck. Which os architecture you are using? Do you run it on your bare metal or do you use any cloud provider? How much RAM does the ubuntu machine has?

Have you tried a subset with disabled or the public api-server?

I ran

mkdir -p data
echo "CURRENT_USER=$(id -u):$(id -g)" >> .env
docker compose run gallery run init --source /data/Pictures
docker compose up -d

from my Ubuntu 24.04 amd64 laptop with docker 26.1.2 and docker compose 26.1.2. It worked as expected.

How can we proceed? Do you have further ideas?

Anyway: thank you so much for your time and reports

EinAeffchen commented 6 months ago

Hi @xemle,

thanks for following up with me. I tried running the api on docker desktop for Windows and run into the exact same problem. However, removing the env variable completely worked on windows to get the api running. Though on my ubuntu machine this doesn't seem to work.

I sadly never really worked with node.js to know anything about what might be going wrong here, though I'd assume the fact that it's running in docker should make it pretty much independent of where the docker installation itself is running. What's interesting however is that I seem to get different node.js versions depending on if I run the image on windows or on ubuntu:

[eval]:1
BACKEND=cpu
       ^

ReferenceError: cpu is not defined
at [eval]:1:1                 
at runScriptInThisContext (node:internal/vm:209:10) at node:internal/process/execution:118:14           at [eval]-wrapper:6:24
at runScript (node:internal/process/execution:101:62)

at evalScript (node:internal/process/execution:133:3)

at node:internal/main/eval_string:51:3

Node.js v20.13.1

Before when I posted this error it stated at the bottom that it was using Node.js 18.18? Did you update the image since I started this issue?

EDIT: So updating my docker version from ~20 to ~26 and using the new api image without setting an env seems to work!

xemle commented 6 months ago

I tried running the api on docker desktop for Windows

So your target host would be Windows? Have you tried the binary distribution of the gallery? ... and you would like to have your own api server locally running via WSL and docker?

Before when I posted this error it stated at the bottom that it was using Node.js 18.18? Did you update the image since I started this issue?

Yes. I am trying to fix the CI jobs to build the docker images via github actions for #136. However somehow the jobs stucks while locally I can build the images. So now the node version is 20 LTS for the docker images if you build them from source.

EinAeffchen commented 6 months ago

So your target host would be Windows?

No no I just tried to run it on windows to see if the problem is my linux system. The target is linux. Yes preferably I'd like to have my local api Server running to be able to run the system as independent as possible.

But as I wrote, with the new image and removing the BACKEND env variable from the api everything is running! The api crashes from time to time without any error, but a simple restart fixes that, which is enough for me for now.

Thank you for your help!

xemle commented 3 months ago

Closing this issue due inactivity.

Please reopen it, if it becomes active again.

xemle / home-gallery

Docker-compose first setup SyntaxError Unexpected token < in JSON at position 0 #138