search

bitnami / containers

Bitnami container images
https://bitnami.com
Other
3.33k stars 4.8k forks source link

[bitnami/minio] Unable to start minio with TLS and docker-compose #36068

Closed eduard-sukharev closed 1 year ago

eduard-sukharev commented 1 year ago

Name and Version

bitnami/minio:2023.5.27

What architecture are you using?

amd64

What steps will reproduce the bug?

Create docker-compose.yml:

version: '3.9'

networks:
  default:
    name: test_network

volumes:
  s3_data:

services:
  s3:
    image: docker.io/bitnami/minio:2023.5.27
    container_name: test_s3
    environment:
      MINIO_ROOT_USER: 'rootuser'
      MINIO_ROOT_PASSWORD: '12345678'
      MINIO_DEFAULT_BUCKETS: 'mybucket'
      MINIO_SCHEME: https
      BITNAMI_DEBUG: "true"
    volumes:
      - s3_data:/data
      - ./s3:/certs
    hostname: s3.test.local

Generate certificates for s3.test.local hostname (used in docker compose file) and put them into ./s3/CAs directory (see minimal setup in attached ZIP)

Do docker compose up

test.zip

What is the expected behavior?

Minio server is running and is accessible with working TLS

What do you see instead?

Minio container starts, runs server, but then makes several attempts to "Adding local Minio host to 'mc' configuration..." and then dies.

$ docker compose up
[+] Building 0.0s (0/0)                                                                                                     
[+] Running 1/0
 ✔ Container test_s3  Recreated                                                                                        0.0s 
Attaching to test_s3
test_s3  |  21:20:47.70 
test_s3  |  21:20:47.70 Welcome to the Bitnami minio container
test_s3  |  21:20:47.70 Subscribe to project updates by watching https://github.com/bitnami/containers
test_s3  |  21:20:47.71 Submit issues and feature requests at https://github.com/bitnami/containers/issues
test_s3  |  21:20:47.71 
test_s3  |  21:20:47.71 INFO  ==> ** Starting MinIO setup **
test_s3  | minio 21:20:47.72 DEBUG ==> Validating settings in MINIO_* env vars..
test_s3  | minio 21:20:47.74 INFO  ==> Starting MinIO in background...
test_s3  | MinIO Object Storage Server
test_s3  | Copyright: 2015-2023 MinIO, Inc.
test_s3  | License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
test_s3  | Version: DEVELOPMENT.2023-05-27T05-56-19Z (go1.19.9 linux/amd64)
test_s3  | 
test_s3  | Status:         1 Online, 0 Offline. 
test_s3  | S3-API: https://localhost:9000 
test_s3  | Console: http://192.168.96.2:9001 http://127.0.0.1:9001 
test_s3  | 
test_s3  | Documentation: https://min.io/docs/minio/linux/index.html
test_s3  | Warning: The standard parity is set to 0. This can lead to data loss.
test_s3  | minio 21:20:57.82 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:02.89 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:07.96 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:13.03 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:18.10 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:23.18 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:28.25 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:33.32 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:38.39 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:43.48 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:48.55 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | minio 21:21:53.62 INFO  ==> Adding local Minio host to 'mc' configuration...
test_s3  | Failed to add temporary MinIO server
test_s3  | minio 21:21:58.78 INFO  ==> MinIO is already stopped...
test_s3 exited with code 1

Additional information

No response

carrodher commented 1 year ago

It seems a very specific use case difficult to reproduce on our side and tied to your scenario.

For information regarding the application itself, customization of the content within the application, or questions about the use of the technology or infrastructure; we highly recommend checking forums and user guides made available by the project behind the application or the technology.

That said, we will keep this ticket open until the stale bot closes it just in case someone from the community adds some valuable info.

eduard-sukharev commented 1 year ago

Am I reading it right that running minio instance with TLS as a simple service within docker-compose to assist local development is a use-case too specific to be considered important? I thought that was one of the basic use-cases for Docker Containers and directly related to bitnami/containers project, isn't it?

hanzlamateen commented 1 year ago

@carrodher I am also facing this issue. Below is my docker file:

version: '2.3'
services:
  minio:
    image: docker.io/bitnami/minio:latest
    container_name: etherealengine_minio_s3
    ports:
      - '9000:9000'
      - '9001:9001'
    volumes:
      - 'minio_data:/data'
      - ../certs/minio:/certs
    environment:
      MINIO_ROOT_USER: server
      MINIO_ROOT_PASSWORD: password
      BITNAMI_DEBUG: true
      MINIO_SCHEME: https
      MINIO_DEFAULT_BUCKETS: etherealengine-static-resources:public,etherealengine-microk8s-static-resources:public,etherealengine-minikube-static-resources:public
volumes:
  minio_data:
    driver: local

I generated my certs using .\certgen-windows-amd64.exe -host "127.0.0.1,localhost" in powershell as explained in minio docs

As I start, it works fine and I can use console and view buckets. But after a while (5 to 10 seconds) it spams Adding local Minio host to 'mc' configuration... and then fails.

Below are the logs:

2023-06-06 17:26:53  12:26:53.21 
2023-06-06 17:26:53  12:26:53.21 Welcome to the Bitnami minio container
2023-06-06 17:26:53  12:26:53.21 Subscribe to project updates by watching https://github.com/bitnami/containers
2023-06-06 17:26:53  12:26:53.21 Submit issues and feature requests at https://github.com/bitnami/containers/issues
2023-06-06 17:26:53  12:26:53.21 
2023-06-06 17:26:53  12:26:53.21 INFO  ==> ** Starting MinIO setup **
2023-06-06 17:26:53 minio 12:26:53.23 DEBUG ==> Validating settings in MINIO_* env vars..
2023-06-06 17:26:53 minio 12:26:53.25 INFO  ==> Starting MinIO in background...
2023-06-06 17:26:53 MinIO Object Storage Server
2023-06-06 17:26:53 Copyright: 2015-2023 MinIO, Inc.
2023-06-06 17:26:53 License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
2023-06-06 17:26:53 Version: DEVELOPMENT.2023-06-02T23-17-26Z (go1.19.9 linux/amd64)
2023-06-06 17:26:53 
2023-06-06 17:26:53 Status:         1 Online, 0 Offline. 
2023-06-06 17:26:53 S3-API: https://localhost:9000 
2023-06-06 17:26:53 Console: https://172.18.0.6:9001 https://127.0.0.1:9001 
2023-06-06 17:26:53 
2023-06-06 17:26:53 Documentation: https://min.io/docs/minio/linux/index.html
2023-06-06 17:26:53 Warning: The standard parity is set to 0. This can lead to data loss.
2023-06-06 17:27:03 minio 12:27:03.30 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:08 minio 12:27:08.40 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:13 minio 12:27:13.48 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:18 minio 12:27:18.57 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:23 minio 12:27:23.66 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:28 minio 12:27:28.78 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:33 minio 12:27:33.87 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:38 minio 12:27:38.96 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:44 minio 12:27:44.05 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:49 minio 12:27:49.13 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:54 minio 12:27:54.21 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:27:59 minio 12:27:59.30 INFO  ==> Adding local Minio host to 'mc' configuration...
2023-06-06 17:28:04 Failed to add temporary MinIO server
2023-06-06 17:28:04 minio 12:28:04.49 INFO  ==> MinIO is already stopped...

If I comment out MINIO_SCHEME & - ../certs/minio:/certs from my docker compose it works fine.

You can also findout my source code and certificates at https://github.com/EtherealEngine/etherealengine/pull/8054/files

hanzlamateen commented 1 year ago

Also @carrodher it seems like something is wrong around this file: https://github.com/bitnami/containers/blob/1ecf74d27594a8c2bf9273b2340e5436e52ab4d9/bitnami/minio/2023/debian-11/rootfs/opt/bitnami/scripts/minio/setup.sh

Setting MINIO_SKIP_CLIENT: yes in environment of docker-compose fixes the crashing issue though it disabled minio client (mc)

hanzlamateen commented 1 year ago

@carrodher @jotamartos also, with following 3 lines in my docker compose. The MINIO_DEFAULT_BUCKETS thing does not work and buckets are not being created.

- ../certs/minio:/certs
MINIO_SCHEME: https
MINIO_SKIP_CLIENT: yes

Its important to mention here, if run docker compose without using above 3 lines, then the buckets will be created. Now since these buckets are already created and you add back those 3 lines you will still see those buckets as they are already created in storage. Therefore, you need to force clean things using below commands before running docker compose with above 3 lines.

  docker container stop etherealengine_minio_s3
  docker container rm etherealengine_minio_s3
  docker container prune --force
  docker volume prune --force
jotamartos commented 1 year ago

Hi all,

I just sat that there is already a similar ticket in the forum, did you take a look at it?

https://github.com/bitnami/containers/issues/29356

I created a certs folder and generated the certificates 

$ ls -lRa certs/
certs/:
total 2744
drwxrwxr-x 3    1001    1001    4096 Jun  8 07:13 .
drwxrwxr-x 4 bitnami bitnami    4096 Jun  8 07:57 ..
drwxrwxr-x 2    1001    1001    4096 Jun  8 07:13 CAs
-rwxrwxr-x 1    1001    1001 2789376 Jun  8 07:09 certgen-linux-amd64
-rw------- 1    1001    1001     241 Jun  8 07:09 private.key
-rw-rw-r-- 1    1001    1001     749 Jun  8 07:09 public.crt

certs/CAs:
total 12
drwxrwxr-x 2 1001 1001 4096 Jun  8 07:13 .
drwxrwxr-x 3 1001 1001 4096 Jun  8 07:13 ..
-rw-rw-r-- 1 1001 1001  749 Jun  8 07:13 public.crt

Note: Important to set the permissions correctly

Then, I modified the docker-compose file 

    environment:
      - MINIO_SCHEME=https
      - MINIO_DEFAULT_BUCKETS=mybucket
    volumes:
      - 'minio_data:/data'
      - './certs:/certs'

and got no errors when launching the container 

minio_1  | minio 07:48:11.41 INFO  ==> Starting MinIO in background...
minio_1  | minio 07:48:21.46 INFO  ==> Adding local Minio host to 'mc' configuration...
minio_1  | minio 07:48:21.52 INFO  ==> Creating default buckets...
minio_1  | Bucket created successfully `local/mybucket`.
minio_1  | minio 07:48:21.72 INFO  ==> Stopping MinIO...
minio_1  |  07:48:21.85 INFO  ==> ** MinIO setup finished! **

I hope this information helps!

hanzlamateen commented 1 year ago

@jotamartos yes, adding public.crt in /certs/CAs folder fixed the problem. Thanks.

Heracles31 commented 9 months ago

Hey guys,

Here is more for you :

when trying to add itself, Minio does not use its hostname. It tries to add itself as "localhost". As such, if your certificate is signed only for the hostname (ex: minio.test.lan) or the network address (192.168 ; 172.16 ; ...), the failure is the same because the cert ends up not trusted. You get that very same loop trying to add itself, failing without giving the full reason and retrying again.

Strange thing is that the MinIO plugin in my TrueNAS Core worked perfectly with the certificate without the localhost added in the SANs. Only Bitnami's image has this problem.

carrodher commented 9 months ago

Thank you for reaching out and for your interest in our project. We'd be happy to assist you, but to do so effectively, we'll need some more details. Could you please create a separate issue filling the Issue template?

Providing this information will help us better understand your situation and assist you in a more targeted manner.

Heracles31 commented 9 months ago

Hi,

My problem is fixed, no need for more help. Thanks for offering.

I just documented here what I had to do as an extra step for other users to learn about it or for you to make your image more resilient if you wish.

Docker image is bitnami/minio:2023.12.20

Your script tries to add local server as https://localhost As such, the certificate must be valid for the name "localhost" (or 127.0.0.1 may work, did not test it). Mine was not. It was valid for the FQDN over the network or the regular IP address (172.16.x.x) and not 127.0.0.1

When trying to start the container with that certificate, I ended up in the same loop described here. I added the local CA but that did not fixed my problem. I opened your init script and saw that it was using Localhost as hostname, so I understood that this was the reason why the certificate was not accepted as valid. I created a new certificate with the names I need PLUS localhost and voilà.

ckotte commented 2 months ago

I have the same issue with localhost and custom certificates. The container crashes because my certs are signed only for the hostname's FQDN. Can this be fixed? I not want to re-create my certificates just because of the bitnami minio container... Should be possible to specify the hostname with a variable instead of just using localhost