Could we use MINIO as database ？

[duckgun13476]

Describe the bug

When I use Minio's S3 storage, it seems to be unusable, with an error message saying "http: server gave HTTP response to HTTPS client." However, my Minio is already configured with TLS. Is this an operational issue on my part 🤡, or does the otherwise excellent Memos not support using Minio as S3 storage 😭?

Steps to reproduce

memos:

MINIO:

error:

The version of Memos you're using

v0.22.2

Screenshots or additional context

No response

[duckgun13476]

Well, MINIO is available in S3 Browser:

[Zeng1998]

The complete error message?

[duckgun13476]

Error as below:

failed to save resource blob: Failed to upload via s3 client: operation error S3: 
PutObject, https response error StatusCode: 0, RequestID: , HostID: , request send failed, 
Put "https://memos-bucket.picture.alvinthe.tech:9500/assets/1720347865_IMG_20230330_080544.jpg?x-id=PutObject": 
dial tcp: lookup memos-bucket.picture.alvinthe.tech on 192.168.147.1:53: no such host

Prompt message: The access link for MINIO's public bucket is:

https://picture.alvinthe.tech:9500/pictures/2023/12/15/202312151051389.png

The domain name of the server where MINIO is deployed is:

picture.alvinthe.tech

Storage information for memos(error):

Storage information in S3browser(healthy):

Could this be easily solved? Minio's local deployment of S3 is very convenient. It would be even more flexible if it were compatible with minio storage!🤩🏔️

[WyInnovate]

My version is v0.22.2. I can use Minio, but I can't upload files. The files uploaded before can be downloaded correctly.

[duckgun13476]

My version is v0.22.2. I can use Minio, but I can't upload files. The files uploaded before can be downloaded correctly.

Could you tell me how you configured the parameters? Would it be possible for you to send a screenshot?😼

[WyInnovate]

My version is v0.22.2. I can use Minio, but I can't upload files. The files uploaded before can be downloaded correctly.

Could you tell me how you configured the parameters? Would it be possible for you to send a screenshot?😼

There is no problem with your S3. In fact, it is an official problem. The problem is the same as https://github.com/usememos/memos/issues/3601

[Morethanevil]

First MinIO must serve the bucket in vhost style, not path style. MINIO_DOMAIN=mydomain.com, configure your reverse proxy / webserver to this. Your bucket is then available under https://bucketname.yourminio.domain

Avoid special characters for password or encode them as described in the docs

It works, here is my config example

The endpoint is the base domain without the bucket name https://domain.com

I hope this helps

[WyInnovate]

First MinIO must serve the bucket in vhost style, not path style. MINIO_DOMAIN=mydomain.com, configure your reverse proxy / webserver to this. Your bucket is then available under https://bucketname.yourminio.domain

Avoid special characters for password or encode them as described in the docs

It works, here is my config example

The endpoint is the base domain without the bucket name https://domain.com

I hope this helps

Good trouble, and I also use nginx, which is more trouble, I remember before January when the path style is still available. Now my MinIO API: https://share.xxx.xxx:2534, WebUI: https://drive.xxx.xxx:2534, I set my MINIO_DOMAIN = share.xxx.xxx:2534, My Endpoint = https://share.xxx.xxx:2534, nginx is not operated, and then the file still cannot be uploaded

[WyInnovate]

@Gavincent1 参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

[Morethanevil]

For Nginx your file should look like this:

server {
   listen       443;
   listen  [::]:443;
   server_name  .mydomain.com;
.........

Important is the dot in front of the domain ☝🏻 It means including all subdomains, because in vhost style MinIO automaticly creates a subdomain based on the bucket name. In my case memos.mydomain.com

[WyInnovate]

For Nginx your file should look like this:

server {
   listen       443;
   listen  [::]:443;
   server_name  .mydomain.com;
.........

Important is the dot in front of the domain ☝🏻 It means including all subdomains, because in vhost style MinIO automaticly creates a subdomain based on the bucket name. In my case memos.mydomain.com

Set the MINIO _ DOMAIN in the environment variable, right? The mydomain.com then points to the MinIO API? Then I'll try to thank you for the fact that there are few things to be saved and they have been converted into databases, but I'm still willing to toss about.

[WyInnovate]

I succeeded, and the reason for the previous failure was that there was no proper SSL certificate. And then you can try it. @duckgun13476

[Gavincent1]

@Gavincent1

参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

[Issues-translate-bot]

Issue is not in English. It has been translated automatically.

---

@Gavincent1

Refer to Morethanevil and add MINIO_DOMAIN, and then set other things.

I've tried this and it doesn't work, prompting status 404

[Morethanevil]

I give you my full example, but be aware I use a reverse proxy in front of nginx, this is why I don't have SSL configured in nginx itself. Port 9950 is the port which nginx listens to and then it streams the requests to MinIO (port 80 and 443 are used by my reverse proxy and the reverse proxy points to 9950). Port 9980 is the MinIO-API port and 9970 the port of the console. If you run nginx and MinIO in docker, you need to replace localhost with the correct names or IPs from your container. I use nginx and MinIO baremetal.

server {
   listen       9950;
   listen  [::]:9950;
   server_name  .mydomain.com;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      proxy_pass http://localhost:9980; # This uses the upstream directive definition to load balance
   }

   location /minio/ui {
      rewrite ^/minio/ui/(.*) /$1 break;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;

      # This is necessary to pass the correct IP to be hashed
      real_ip_header X-Real-IP;

      proxy_connect_timeout 300;

      # To support websockets in MinIO versions released after January 2023
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      chunked_transfer_encoding off;

      proxy_pass http://localhost:9970; # This uses the upstream directive definition to load balance
   }
}

[WyInnovate]

@Gavincent1 参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

我这边设置好了以后没有问题，我是用的Nginx Proxy Manager然后我的设置如下 MinIO API: https://share.xxx.xxx:2534 WebUI: https://drive.xxx.xxx:2534 环境变量MINIO_DOMAIN = share.xxx.xxx:2534 Endpoint = https://share.xxx.xxx:2534 NPM中新增设置 *.share.xxx.xxx指向http://192.168.50.184:9002 NPM中原有设置 share.xxx.xxx指向http://192.168.50.184:9002

[duckgun13476]

I succeeded, and the reason for the previous failure was that there was no proper SSL certificate. And then you can try it. @duckgun13476

The Minio I deployed which is using its own built-in SSL for deployment, so the structure cannot be modified. Later, I will try to modify it to use Nginx as a proxy and see if it works.👍

[Gavincent1]

@Gavincent1

参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

我这边设置好了以后没有问题，我是用的Nginx Proxy Manager然后我的设置如下

`

MinIO API: https://share.xxx.xxx:2534

WebUI: https://drive.xxx.xxx:2534

环境变量MINIO_DOMAIN = share.xxx.xxx:2534

Endpoint = https://share.xxx.xxx:2534

NPM中新增设置 *.share.xxx.xxx指向http://192.168.50.184:9002

NPM中原有设置 share.xxx.xxx指向http://192.168.50.184:9002

`

非常感谢你的配置示例，我有点明白是怎么回事了，我需要想想我需要怎么配置，因为我是在群晖自带的反向代理中搭建的MinIO。

[duckgun13476]

failed to save resource blob: Failed to upload via s3 client: operation error S3: 
PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: 
, HostID: , request send failed, Put "https://memos-data.picture.alvinthe.tech:2443/assets/1721309928_%E5%B1%8F%E5%B9%95
%E6%88%AA%E5%9B%BE%202023-06-16%20111437.png?x-id=PutObject": 
dial tcp [2409:873b:2535:2811::1030]:2443: connect: cannot assign requested address

😭 I still cannot use the vhost mode and encountered the above error. I suggest adding a way to access with path-style. Absolute paths do not require defining subdomains, and they are very compatible. The code modification should only require adjusting the access structure: 😭

[WyInnovate]

I looked at the code of plugin/storage/s3/s3.go and used AI to explain the code, and found that the AWS SDK defaults to Virtual Hosted Style, but it is possible to add a configuration item to explicitly specify the use of Path Style. However, this requires adding an option in the S3 settings interface, which I don't know much about The modified code for s3.go provided by AI is, but this fixed uses Path Style:

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.BaseEndpoint = aws.String(s3Config. Endpoint)
o.UsePathStyle=true //Setting to true means using Path Style
})

@Zeng1998

[Gavincent1]

@Gavincent1

参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

我这边设置好了以后没有问题，我是用的Nginx Proxy Manager然后我的设置如下 MinIO API: https://share.xxx.xxx:2534 WebUI: https://drive.xxx.xxx:2534 环境变量MINIO_DOMAIN = share.xxx.xxx:2534 Endpoint = https://share.xxx.xxx:2534 NPM中新增设置 *.share.xxx.xxx指向http://192.168.50.184:9002 NPM中原有设置 share.xxx.xxx指向http://192.168.50.184:9002

非常感谢你的配置示例，我有点明白是怎么回事了，我需要想想我需要怎么配置，因为我是在群晖自带的反向代理中搭建的MinIO。

我已经实现了vhost的访问方式，但是还是上传失败了。你们有遇到这个问题吗？

@WyInnovate

[WyInnovate]

@Gavincent1

参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

我这边设置好了以后没有问题，我是用的Nginx Proxy Manager然后我的设置如下 MinIO API: https://share.xxx.xxx:2534 WebUI: https://drive.xxx.xxx:2534 环境变量MINIO_DOMAIN = share.xxx.xxx:2534 Endpoint = https://share.xxx.xxx:2534 NPM中新增设置 *.share.xxx.xxx指向http://192.168.50.184:9002 NPM中原有设置 share.xxx.xxx指向http://192.168.50.184:9002

非常感谢你的配置示例，我有点明白是怎么回事了，我需要想想我需要怎么配置，因为我是在群晖自带的反向代理中搭建的MinIO。

我已经实现了vhost的访问方式，但是还是上传失败了。你们有遇到这个问题吗？

@WyInnovate

看着像是文件上传超过了限制大小，你看看哪里设置的有问题

[Gavincent1]

@Gavincent1

参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

我这边设置好了以后没有问题，我是用的Nginx Proxy Manager然后我的设置如下 MinIO API: https://share.xxx.xxx:2534 WebUI: https://drive.xxx.xxx:2534 环境变量MINIO_DOMAIN = share.xxx.xxx:2534 Endpoint = https://share.xxx.xxx:2534 NPM中新增设置 *.share.xxx.xxx指向http://192.168.50.184:9002 NPM中原有设置 share.xxx.xxx指向http://192.168.50.184:9002

非常感谢你的配置示例，我有点明白是怎么回事了，我需要想想我需要怎么配置，因为我是在群晖自带的反向代理中搭建的MinIO。

我已经实现了vhost的访问方式，但是还是上传失败了。你们有遇到这个问题吗？ @WyInnovate

看着像是文件上传超过了限制大小，你看看哪里设置的有问题

我设置的是上传60MB大小，然后实际只上传了一张几百KB的图片就提示这个报错

[Gavincent1]

@Gavincent1 参考Morethanevil说的加上MINIO_DOMAIN，然后设置好其他的就行了

我已经试过了，并没有起作用，提示状态404 路径访问方式一切正常

我这边设置好了以后没有问题，我是用的Nginx Proxy Manager然后我的设置如下 MinIO API: https://share.xxx.xxx:2534 WebUI: https://drive.xxx.xxx:2534 环境变量MINIO_DOMAIN = share.xxx.xxx:2534 Endpoint = https://share.xxx.xxx:2534 NPM中新增设置 *.share.xxx.xxx指向http://192.168.50.184:9002 NPM中原有设置 share.xxx.xxx指向http://192.168.50.184:9002

我成功在0.22.3版本中使用MinIO存储了，这太棒了，现在就差可以设置URL前缀就可以将正式环境升级到最新本了。 我是在群晖中搭建的memos和MinIO。

[WyInnovate]

啥URL前缀？现在neosmemo/memos:stable就是0.22.3版本吧 你都搞了啥现在可以用了，上次不是上传不了图片吗？解决了？ @Gavincent1

[Issues-translate-bot]

Issue is not in English. It has been translated automatically.

---

What URL prefix? Now neosmemo/memos:stable is version 0.22.3, right? What have you done? Now it can be used. Didn't you fail to upload pictures last time? solved? @Gavincent1

[duckgun13476]

Using MinIO's built-in TLS does not allow for vhost mode; Nginx must be used instead. This is because MinIO's vhost mode requires defining a domain name that includes all subdomains of the containers to function properly. In the case of self-deployed MinIO, certificates that include subdomains might be costy, which goes against the intention of using open-source projects. While using Nginx is a feasible solution, it exceeds the purpose of deploying this project. Therefore, my solution is to mount an NFS NAS storage to the container path and use local storage. If you still want to use local storage, you can start a Docker deployment of MinIO on your local server and access it via the local IP address; accessing locally poses no security issues and allows S3 access. However, this approach seems pointless since using local storage appears faster. Thus, there are three solutions for this part:

1) Use an Nginx proxy to deploy and enable vhost mode access; 2) Deploy a local MinIO image service to utilize local vhost mode; 3) Suggest that the author add functionality for absolute path access in S3 storage, which should be simple—just need to backport it into the container.

[duckgun13476]

3740 I opened a new suggestion issue because it can no longer be defined as a bug; instead, it's a compatibility issue that needs improvement.

[WyInnovate]

If my Memos and MINIO are on the same server, is using local storage a better choice?

[duckgun13476]

If my Memos and MINIO are on the same server, is using local storage a better choice?

Yeah, with a small amount of content, local storage performs better. However, in the case of millions of images, S3 seems faster. But how long will it take us to upload millions of images?🧐

[Gavincent1]

啥URL远端？现在neosmemo/memos:stable就是0.22.3版本吧 你都搞了啥现在可以用了，上次不是上传不了图片吗？解决了吗？ @Gavincent1

上传不了图片是因为使用Mysql版本的有个bug，不过现在解决了，详细 #3718 URL前缀类似于可以给S3文件自定义一个访问域名，这个域名不是MinIO节点域名，例如：给MinIO套上一个CDN域名，在上传的时候使用MinIO节点域名，然后访问的时候是使用CDN域名。

[Issues-translate-bot]

Issue is not in English. It has been translated automatically.

---

What is the remote URL? Now neosmemo/memos:stable is version 0.22.3. What have you done? Now it can be used. Didn’t you fail to upload pictures last time? Solve it? @Gavincent1

The reason why I can't upload pictures is because there is a bug in the Mysql version, but it has been solved now, details #3718 URL prefix is similar to how you can customize an access domain name for S3 files. This domain name is not the MinIO node domain name. For example: add a CDN domain name to MinIO, use the MinIO node domain name when uploading, and then use the CDN domain name when accessing. .

[xiaoweihong]

I looked at the code of plugin/storage/s3/s3.go and used AI to explain the code, and found that the AWS SDK defaults to Virtual Hosted Style, but it is possible to add a configuration item to explicitly specify the use of Path Style. However, this requires adding an option in the S3 settings interface, which I don't know much about The modified code for s3.go provided by AI is, but this fixed uses Path Style:

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.BaseEndpoint = aws.String(s3Config. Endpoint)
o.UsePathStyle=true //Setting to true means using Path Style
})

@Zeng1998

I recompile use the code .it works

[WyInnovate]

I looked at the code of plugin/storage/s3/s3.go and used AI to explain the code, and found that the AWS SDK defaults to Virtual Hosted Style, but it is possible to add a configuration item to explicitly specify the use of Path Style. However, this requires adding an option in the S3 settings interface, which I don't know much about The modified code for s3.go provided by AI is, but this fixed uses Path Style:我查看了的代码并使用AI解释了代码，发现AWS SDK默认为，但可以添加配置项来显式指定的使用。不过，这需要在S3设置界面中添加一个选项，我对此了解不多AI提供的修改代码是，但这个修复使用了Path Style：

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.BaseEndpoint = aws.String(s3Config. Endpoint)
o.UsePathStyle=true //Setting to true means using Path Style
})

@Zeng1998

I recompile use the code .it works我使用代码重新编译。它有效

ok

[xueba0458]

@duckgun13476 You can check out this blog post and follow this tutorial to configure