online docker image + docker-compose

[darrepac]

Hi

Your projet sounds nice and I will give it a try. Why not publishing the docker image into docker hub and proposing a docker-compose file...sounds to me easier than the steps you propose...but will give a try anyway ;)

[xemle]

Thank you for your feedback.

Recently I was focusing on providing binaries for Linux, Mac and Windows for a more common usecase than docker. See installation section in the docs.

The status of docker is older and with the tar.gz version of the new bundling, the docker steps could be simplified further.

If there is a need for docker, I could provide images to docker hub, true.

So do not hesitate to try the gallery and I appriciate your feedback and your needs

[xemle]

Maybe the multi-stage build would help to simplify the steps and resulting image size using new bundle tar.gz files...

@darrepac what kind of services would you like to include into the docker compose stack?

[darrepac]

Hi

I miss the documentation link (only read the github readme). Will definitely take a look next week. Distributing a binary seems quite odd nowadays: everyone is proposing (because people ask) docker image… easier, safer, I can see only advantages particularly to test a new stuff. Then, having a docker compose stack with all the services needed and the volume creation etc is also going for the easy route for new user My 2 cents

[xemle]

Thank you for your feedback.

I think the format of docker or binary is a question of use case and flavor. I prefer more the docker way on my Linux, too. However my friends are desktop end users on Windows or Mac. So a decision is difficult and mostly subjective at the current state.

Let's discuss this topic when you are convinced by HomeGallery and you prefer a docker way.

I am also happy about a PR or can give some support to create one.

[darrepac]

Sure! Not a bad idea to do the test first on windows as a binary is easier to deploy here

Le 29 mai 2021 à 10:34, Sebastian @.***> a écrit :

 Thank you for your feedback.

I think the format of docker or binary is a question of use case and flavor. I prefer more the docker way on my Linux, too. However my friends are desktop end users on Windows or Mac. So a decision is difficult and mostly subjective at the current state.

Let's discuss this topic when you are convinced by HomeGallery and you prefer a docker way.

I am also happy about a PR or can give some support to create one.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[igomezl]

Hi!

I'm also discovering your project, and I found it very interesting and useful. I also prefer using Docker for my selfhosted apps. The current way to run it in interactive mode served me for testing, but in order to use it for "production" the image should be able to run in non-interactive mode, maybe passing an argument to the gallery.js to start the server (same action as using the menu). That would be good enough. Running in non-interactive would allow to prepare a compose file, even without the need to publish an image

Thanks for sharing this with the community!

[xemle]

Welcome @igomezl! Glad you found and like HomeGallery.

Meanwhile I pushed new changes to the master including updated non-interactive mode and improved Dockerfile. Please have a look to the updated CHANGELOG.md.

I was also playing with public docker images, please do not hesitate to test the image from the hub xemle/home-gallery. I am still struggling to support both amd64 and arm64 architecture and could need some help here. I was following Building Multi-Architecture Docker Images With Buildx. My docker buildx command failed on arm64 image on a amd64 host.

For non-interative mode you can access them via run command. See details calling the docker image with -h argument. E.g. docker run ... xemle/home-gallery run server or docker run ... xemle/home-gallery run import.

My next week will be busy. So do not expect further changes soon. If you find time to test the new features I appreciate your feedback.

[darrepac]

Hi!

I'm also discovering your project, and I found it very interesting and useful. I also prefer using Docker for my selfhosted apps. The current way to run it in interactive mode served me for testing, but in order to use it for "production" the image should be able to run in non-interactive mode, maybe passing an argument to the gallery.js to start the server (same action as using the menu). That would be good enough. Running in non-interactive would allow to prepare a compose file, even without the need to publish an image

Thanks for sharing this with the community!

@igomezl If you are able to prepare a compose file, I would be happy using it :)

[xemle]

I was able to publish docker images via github actions. Now there will be always new docker images on master updates.

[xemle]

@darrepac regarding docker-compose file: I guess you would like to have the home-gallery and the api server within the docker compose file, don't you? Would it be fine for the first run to provide a docker-compose file where the docker images are build locally? Up to now I would avoid publishing a docker image for the api-server with 2GB

What should the docker-compose also cover? Multiple source volumes?

[xemle]

Hi @darrepac

here is a sample docker-compose.yml file:

version: "3.9"
services:
  api:
    build: packages/api-server
    environment:
      - BACKEND=node
    ports:
      - "3000"
  gallery:
    build: .
    # Or use public image
    #image: xemle/home-gallery
    environment:
      - GALLERY_API_SERVER=http://api:3000
    volumes:
      - ./data:/data
      - ${HOME}/Pictures:/data/Pictures
    expose: 
      - "3000"
    entrypoint: ['node', '/app/gallery.js', 'run', 'server']

You need to run

# Build docker images from sources
docker-compose build
# Starts the webserver by default
docker-compose up -d

If you need to update the gallery, run docker-compose exec gallery /app/gallery.js run import --update to process new images

[darrepac]

Sounds interesting. I will give a try in the next days. I am not an expert but why do you prefer that we build the image locally instead of using the online image? online is updated less often?

[darrepac]

verson: "3.9" => version: "3.9" ;)

[xemle]

Thanks. Corrected.

Please do not hesitate to give further feedback.

Regarding the docker image for the api-server: I do not have hard facts not to push this image yet - it is just a gut feeling.

The traffic to HomeGallery is yet also limited. So I try to do the most important things with my limited time. And I think, that the keyboard bindings for the desktop is more important than the api server docker image.

The tricky part with the api server is also that it is part of the mono repo attatched to its global version on the one hand. One the other hand it could be a dedicated service with its own versioning. I could not solve it yet when to build the api server release version and when not. The latest tag is easy, but the version tag question is not solved yet.

You are free to provide some ideas or PR regarding it.

[darrepac]

I was not able to build, see below:

npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated opn@6.0.0: The package has been renamed to `open`
npm WARN deprecated highlight.js@9.18.5: Support has ended for 9.x series. Upgrade to @latest
npm WARN deprecated chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated core-js@2.6.12: core-js@<3.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
npm WARN deprecated @hapi/joi@16.1.8: Switch to 'npm install joi'
npm WARN deprecated @hapi/address@2.1.4: Moved to 'npm install @sideway/address'
npm WARN deprecated @hapi/formula@1.2.0: Moved to 'npm install @sideway/formula'
npm WARN deprecated @hapi/hoek@8.5.1: This version has been deprecated and is no longer supported or maintained
npm WARN deprecated @hapi/pinpoint@1.0.2: Moved to 'npm install @sideway/pinpoint'
npm WARN deprecated @hapi/topo@3.1.6: This version has been deprecated and is no longer supported or maintained
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://npm.community>

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-06-13T07_25_10_590Z-debug.log
The command '/bin/sh -c cp package.json package.json.orig &&   grep -v -e api-server -e styleguide package.json.orig > package.json &&   npm install &&   find node_modules/@ffprobe-installer -name ffprobe -exec chmod ugo+x {} \;' returned a non-zero code: 1
ERROR: Service 'gallery' failed to build

that's why I prefer online images ;)

[xemle]

Thank you for your time and your feedback.

Software development is very complex. To have public docker images is only a little step towards a final product. Docker is meant to be reproducible on other systems, so it is strange why it is not building on your side. It is also frustrating for me if something is not working as expected - especially on customer side as yours where I could not quickly check and fix things. So sorry for the inconvenience. Further, I am curious what are the difference on your machine and on mine and GitHub's (See logs from the action in the build step).

Would your mind to provide some context? Where you try to build the images and the version of docker and docker compose? It would be helpful to harden the system requirements. For the docker part I would like to have it reproducible locally for others as well.

You can provide infos through

uname -r -v
docker version
docker-compose version
git rev-parse HEAD

My output is:

$ uname -r -v
4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19)
$ docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.11.6
 Git commit:        4c52b90
 Built:             Sun, 21 Feb 2021 18:18:35 +0100
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.6
  Git commit:       4c52b90
  Built:            Sun Feb 21 17:18:35 2021
  OS/Arch:          linux/amd64
  Experimental:     false
$ docker-compose version
docker-compose version 1.7.1, build 0a9ab35
docker-py version: 1.8.1
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
$ git rev-parse HEAD
205a53b68f7438d76e8b14dfa5f4de8e36375b2a

[darrepac]

No problem:

$uname -r -v
4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19)
$ docker version
Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:45 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ docker-compose version
docker-compose version 1.28.4, build cabd5cfb
docker-py version: 4.4.3
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
$ git rev-parse HEAD
43026294408e57251393dfd9f6a0023b4eb1a70e

[xemle]

Thank you. Yet I have no clue why your build should not work.

Does docker build -t home-gallery --no-cache . works? I assume that it produces the same error.

[darrepac]

It took hours but

Successfully built 9c003f68e9d3
Successfully tagged home-gallery:latest

[xemle]

Wow! Congratulation!

Do you know the reason for the previous failure?

[darrepac]

Wow! Congratulation!

Do you know the reason for the previous failure?

no idea

[xemle]

Mm. OK.

So you are know happy having a docker compose with a local api-server (while you are recommending public images for home-gallery and api-server)?

[darrepac]

Sorry for not being so reactive these days... After docker-compose launch, I have the following in the log:

Init configuration from /app/gallery.config-example.yml

Loaded gallery configuration from gallery.config.yml

Starting server

Execute: DEBUG=server* /usr/local/bin/node /app/gallery.js server --storage /data/storage --database /data/config/database.db --events /data/config/events.db

Open Home Gallery on http://localhost:3000

Database file /data/config/database.db does not exists. Waiting for the database file...

My docker-compose.yml file is:

version: "3.9"
services:
  api:
    build: packages/api-server
    environment:
      - BACKEND=node
    ports:
      - "3000"
  gallery:
    build: .
    # Or use public image
    #image: xemle/home-gallery
    environment:
      - GALLERY_API_SERVER=http://api:3000
    volumes:
      - ./data:/data
      - /home/pascal/Home-Gallery/test-photos:/data/Pictures
    expose:
      - "8080"
    entrypoint: ['node', '/app/gallery.js', 'run', 'server']

So I am suprised that while I expose the 8080 port, it seems to run at 3000. And I don't understand this: Database file /data/config/database.db does not exists. Waiting for the database file... I need to provide a database file??

[xemle]

HomeGallery requires some folders:

• The configuration folder for the general configuration, the file indices and the database
• The storage directory for the extracted previews and meta data
• One or more media source directories

By default they are defined as following and the structure leans against the common Linux folders

• ConfigDir: ~/.config/home-gallery
• StorageDir: ~/.cache/home-gallery/storage
• One Source ~/Pictures

These default values are chosen for the desktop binaries to get started right away in most cases. And all these values can be configured in the gallery.config.yml. It can also overwritten by environment variables.

For Docker all user data (eg. config and previews) should be located at /data and needs to be mounted from the host to the container. The config and storage directory are defined more explicitly

• ConfigDir: /data/config - indirectly through /data volume
• StorageDir: /data/storage - indirectly through /data volume
• One Source /data/Pictures - mounted as additional volume

Your host directory structure should be

.data/
  config/
  storage/

The docker environment variables from Dockerfile overwrites the default values as default docker locations:

VOLUME [ "/data" ]

WORKDIR /data

ENV HOME=/data
ENV GALLERY_BASE_DIR=/data
ENV GALLERY_CONFIG_DIR=/data/config
ENV GALLERY_CACHE_DIR=/data
ENV GALLERY_CONFIG=/data/config/gallery.config.yml

If you try to use the binary and the docker variant on the same time, all directory needs to match to one definition. Either docker matches the desktop directories or vise versa.

For the docker follows the desktop folders you need to change the docker-compose.yml

services:
  gallery:
    ...
    environment:
      - GALLERY_API_SERVER=http://api:3000
      - GALLERY_BASE_DIR=/data
      - GALLERY_CONFIG_DIR=/data/.config/home-gallery
      - GALLERY_CACHE_DIR=/data/.cache/home-gallery
      - GALLERY_CONFIG=/data/.config/home-gallery/gallery.config.yml
    ...

Another possibility would be to mount ~/.config/home-gallery from your host to /data/config and ~/.cache/home-gallery/storage from your host to /data/storage in the container.

Your "error message" that the database could not be found might be caused by missing configuration for the migration from the binary to the docker variant.

Further the logging of port 3000 comes from your running container. Here the server does not know (or does not need to know) that you like to have port 8080 on your host. Your desired port is usually mapped via docker-compose.yml so that your host offers port 8080 and forwards the requests to default port 3000 in the container. See their port documentation. Than your docker-compose.yml should contain

services:
  gallery:
    ...
    ports:
      - "8080:3000"
    ...

An expose would not make sense here since it is more for service to service communication within your docker compose. See the expose statement for the api server.

Hope that clarifies and will solves your issues.

[darrepac]

I am sorry but I thought it was clear but I still have the same error Database file /data/config/database.db does not exists. Waiting for the database file...

Here is my docker-compose yml file:

version: "3.9"
services:
  api:
    build: packages/api-server
    environment:
      - BACKEND=node
    ports:
      - "3000"
  gallery:
    build: .
    # Or use public image
    #image: xemle/home-gallery
    environment:
      - GALLERY_API_SERVER=http://api:3000
    volumes:
      - /srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data:/data
      - /home/pascal/Home-Gallery/test-photos:/data/Pictures
    ports:
      - "8080:3000"
    entrypoint: ['node', '/app/gallery.js', 'run', 'server']

And the structure seems correct:

$ ls -R /srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data
/srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data:
Pictures  config  gallery.config.yml  storage

/srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data/Pictures:

/srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data/config:

/srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data/storage:

Note that I have just created data/config and data/storage : Pictures folder and gallery.config.yml have been created by the docker deployment

[xemle]

From your provided output your Pictures folder looks empty. Is that true?

If you have no database yet, you need to process/import the files which creates one, yes. In my comment I wrote you can access the CLI eg via docker-compose exec gallery /app/gallery.js -h and you can trigger the import via docker-compose exec gallery /app/gallery.js run import.

Hope that helps.

I apologize if the documentation on a feature that is not yet public does not yet exist. Could you follow in general the steps and comments of the installation tutorial to get an idea what you need to do? Any suggestions or PR for the documentation repository is welcome, too ;-)

[xemle]

Maybe we should clarify in which step you stuck. To see images in the browser you need

1. Install the gallery via docker, docker-compose or prebuilt platform binary
2. Init gallery.config.yml and configure the media sources (via CLI)
3. Start the server (done automatically on docker-compose)
4. Import sources via CLI (in parallel with running server)

While docker and prebuild binary start the CLI right away, docker-compose exec gallery /app/gallery.js starts it in docker-compose environment as mentioned before. The <cli> -h parameter shows supported commands and options. Most relevant actions might be the interactive and the run command. Eg. use ... run -h to see help for the run command.

So the question is: what have you done so far, in which step you discover troubles and what are your expectations. Keep in mind: Any provided relevant context and your desire helps to identify your issue.

[darrepac]

Well, I have built the docker file and wrote the docker-compose yml file as written above. Just that. In your sample from here https://github.com/xemle/home-gallery/issues/3#issuecomment-859076099, you wrote that we can have a dedicated volume for Pictures:

    volumes:
      - ./data:/data
      - ${HOME}/Pictures:/data/Pictures

This is what I have done in my yml file:

    volumes:
      - /srv/dev-disk-by-uuid-3113b027-5b78-45ff-8658-55185af89df2/CONFIG/HomeGallery/data:/data
      - /home/pascal/Home-Gallery/test-photos:/data/Pictures

In /home/pascal/Home-Gallery/test-photos , I have pictures:

~$ ls /home/pascal/Home-Gallery/test-photos
05EFC1C7-DEA6-472F-8C7C-C0422CF2E6B4.jpg               IMG_0164.jpg  IMG_0771.jpg  IMG_9365.jpg
1080p.mov                                              IMG_0169.jpg  IMG_0772.jpg  IMG_9366.jpg
1EC7356C1FAB63091C92C1F22826162F.jpg                   IMG_0170.jpg  IMG_0773.jpg  IMG_9367.jpg
20200106_083429004_iOS.jpg                             IMG_0171.jpg  IMG_0774.jpg  IMG_9368.jpg
20200106_084118808_iOS.jpg                             IMG_0201.jpg  IMG_0775.jpg  IMG_9369.jpg
20200106_090926130_iOS.jpg                             IMG_0202.jpg  IMG_0776.jpg  IMG_9370.jpg
20200111_181538681_iOS.jpg                             IMG_0203.jpg  IMG_0778.jpg  IMG_9371.jpg
20200111_181543979_iOS.jpg                             IMG_0235.jpg  IMG_0779.jpg  IMG_9372.jpg
20200114_180519631_iOS.jpg                             IMG_0240.jpg  IMG_0781.jpg  IMG_9373.jpg
20200116_110849449_iOS.jpg                             IMG_0248.jpg  IMG_0782.jpg  IMG_9374.jpg
20200121_205922634_iOS.jpg                             IMG_0249.jpg  IMG_0783.jpg  IMG_9375.jpg
20200125_115010121_iOS.jpg                             IMG_0250.jpg  IMG_0784.jpg  IMG_9376.jpg
20200125_115106266_iOS.jpg                             IMG_0251.jpg  IMG_0787.jpg  IMG_9389.jpg
20200125_115135192_iOS.jpg                             IMG_0256.jpg  IMG_0788.jpg  IMG_9390.jpg
20200129_194558006_iOS.jpg                             IMG_0347.jpg  IMG_0789.jpg  IMG_9396.jpg
20200131_082022975_iOS.jpg                             IMG_0348.jpg  IMG_0790.jpg  IMG_9397.jpg
20200208_151950773_iOS.jpg                             IMG_0355.jpg  IMG_0791.jpg  IMG_9398.jpg
20200208_203043566_iOS.jpg                             IMG_0356.jpg  IMG_0792.jpg  IMG_9402.jpg
20200220_142009172_iOS.jpg                             IMG_0357.jpg  IMG_0794.jpg  IMG_9403.jpg
20200306_121547774_iOS.jpg                             IMG_0358.jpg  IMG_0795.jpg  IMG_9404.jpg
20200312_144429634_iOS.jpg                             IMG_0359.jpg  IMG_0796.jpg  IMG_9405.jpg
20200316_185506140_iOS.jpg                             IMG_0360.jpg  IMG_0798.jpg  IMG_9406.jpg
20200326_185427046_iOS.jpg                             IMG_0378.jpg  IMG_0799.PNG  IMG_9407.jpg
20200401_183620800_iOS.jpg                             IMG_03

[xemle]

Sounds good so far. So step 1. installation, 2. init config and 3. start server seems to be fine.

The only thing to check: you have a config in /data which wont be read. Is there a config in /data/config as well? If you start the interactive cli action you can check below the system sub menu your config.

How about step 4. importing files?

[darrepac]

ok, I started the CLI and imported the pictures! I was far from thinking that we have to start the CLI to import pictures! Should be automatic... Let see after the import process

[xemle]

Thanks for the feedback. Where would you expect these information?

The reason why the import is triggered manually is historic and will be solved iterative - depends on demand, mood and time. Keep in mind all the work is done in my spare time for free which should/must not impact my general work-life balance.

[darrepac]

Don't take my comment bad: I know that it is an open source projet on spare time. My comments are just here to make things better but I am not waiting things to be done for yesterday, neither I consider my comments as a "must do" ;) I will do more testing, but my main worry is that it seems to need a lightning fast internet connection when I am out from home....

[xemle]

Glad that we are on the same boat regarding service level agreement for this project. Thumbs up.

Regarding the internet connection. What speed is your upload? Mine is about 10 Mbit/s and seems to be OKish for 100K media served by a Raspberry Pi 4. It might take some time until the first page impression (should be faster, but could not tackle the reason by now) and than its fine. Currently I have a SSH tunnel to access my home gallery. At home it is faster through the local network of cause.

What kind of experience do you have?

[darrepac]

I think my upload, which is the limiting factor here I guess, is max 1MBit/s... I am still waiting for the fiber to arrive (coming months I hope)

[xemle]

OK. Hope that's solves your issue.

FYI I do share my public images via the demo page (via an static export) and having the private pics at home. Maybe it could match to you as well

[xemle]

Version 1.1.0 was released with detailed installation instruction for docker and docker-compose.

I am closing this ticket.