search

truenas / charts

TrueNAS SCALE Apps Catalogs & Charts
BSD 3-Clause "New" or "Revised" License
301 stars 290 forks source link

MinIO console unable to reach S3 interface #2577

Closed jadedeane closed 3 months ago

jadedeane commented 3 months ago

TrueNAS-SCALE-24.04.1.1, MinIO app 2023-03-13 chart 2.6.0 image minio/minio:RELEASE.2023-07-21T21-12-44Z.

When logging in to console:

Post "https://nas1.domain:9000/": dial tcp 192.168.0.200:9000: i/o timeout

From within the MinIO pod:

root@nas1[~]# k3s kubectl -n ix-minio exec -it minio-7bc8b845f4-bplbj -c minio -- bash
bash-4.4$ curl https://localhost:9000
curl: (51) SSL: no alternative certificate subject name matches target host name 'localhost'
bash-4.4$ curl https://192.168.0.200:9000
curl: (7) Failed to connect to 192.168.0.200 port 9000: Connection timed out

MinIO pod logs:

root@nas1[~]# k3s kubectl -n ix-minio logs -f  minio-7bc8b845f4-bplbj -c minio
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2023-07-21T21-12-44Z (go1.19.11 linux/amd64)

Status:         1 Online, 0 Offline.
S3-API: https://nas1.domain:9000
Console: https://nas1.domain:9002

Documentation: https://min.io/docs/minio/linux/index.html
Warning: The standard parity is set to 0. This can lead to data loss.

 You are running an older version of MinIO released 10 months ago
 Update: Run `mc admin update`

Using curl above to test socket connections (in lieu of natcat), localhost works (cert fail, expected), but IP (and by extension FQDN as well if used) fails.

This was working a few days ago, and problems seem to coincide with an automatic cert update (cert is valid):

❯ echo | openssl s_client -showcerts -servername nas1.domain -connect nas1.domain:9000 2>/dev/null | openssl x509 -inform pem -noout -text
...
        Validity
            Not Before: Jun  3 18:41:47 2024 GMT
            Not After : Sep  1 18:41:46 2024 GMT
        Subject: CN=nas1.domain

tl;dr, MinIO pod console can't reach s3 interface within pod.

stavros-k commented 3 months ago

Hello, from within the container you can't reach the host IP unless you use host network. Your client should be able to reach your host IP, and also be able to resolve the FQDN. Which I assume it should do already considering that it was working before.

Please try "editing" the minio app and saving without any changes. I see tha t the cert is picked up correctly, so this shouldn't bring any new results.

Can you share what curl shows from your client to minio?

jadedeane commented 3 months ago

Hello, from within the container you can't reach the host IP unless you use host network. Your client should be able to reach your host IP, and also be able to resolve the FQDN. Which I assume it should do already considering that it was working before.

~Please try "editing" the minio app and saving without any changes.~ I see tha t the cert is picked up correctly, so this shouldn't bring any new results.

Can you share what curl shows from your client to minio?

Client curl? Sure, cert's valid, so all I'd expect would be complaining about JS:

❯ curl https://nas1.domain:9002
<!doctype html><html lang="en"><head><meta charset="utf-8"/><base href="/"/><meta content="width=device-width,initial-scale=1" name="viewport"/><meta content="#081C42" media="(prefers-color-scheme: light)" name="theme-color"/><meta content="#081C42" media="(prefers-color-scheme: dark)" name="theme-color"/><meta content="MinIO Console" name="description"/><meta name="minio-license" content="agpl" /><link href="./styles/root-styles.css" rel="stylesheet"/><link href="./apple-icon-180x180.png" rel="apple-touch-icon" sizes="180x180"/><link href="./favicon-32x32.png" rel="icon" sizes="32x32" type="image/png"/><link href="./favicon-96x96.png" rel="icon" sizes="96x96" type="image/png"/><link href="./favicon-16x16.png" rel="icon" sizes="16x16" type="image/png"/><link href="./manifest.json" rel="manifest"/><link color="#3a4e54" href="./safari-pinned-tab.svg" rel="mask-icon"/><title>MinIO Console</title><script defer="defer" src="./static/js/main.90ab1516.js"></script><link href="./static/css/main.7f39beab.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"><div id="preload"><img src="./images/background.svg"/> <img src="./images/background-wave-orig2.svg"/></div><div id="loader-block"><img src="./Loader.svg"/></div></div></body></html>

In browser, it was a ~3min spinning circle but finally loaded page. When attempting to login from browser, the following error was presented:

Post "https://nas1.domain:9000/": dial tcp 192.168.0.200:9000: i/o timeout

Screenshot 2024-06-17 at 10 07 43 AM

Was working fine, and this all started right around the time acme letsencrypt cert renewed.

stavros-k commented 3 months ago

Hmm I suppose you have set the domain field right?

jadedeane commented 3 months ago

Hmm I suppose you have set the domain field right?

"Set the domain field" where exactly? Within TrueNAS, yup, box steady state, no changes. All "native" (iSCSI, NFS< SMB) services working as expected. MinIO, which is my only chart application, is the outlier.

The screenshot above reads to me as a simple socket failure.

As I mentioned previously, not remotely sure where this would die in, but things started failing around the time of automatic cert renewal.

stavros-k commented 3 months ago

Hmm I suppose you have set the domain field right?

"Set the domain field" where exactly? Within TrueNAS, yup, box steady state, no changes. All "native" (iSCSI, NFS< SMB) services working as expected. MinIO, which is my only chart application, is the outlier.

The screenshot above reads to me as a simple socket failure.

As I mentioned previously, not remotely sure where this would die in, but things started failing around the time of automatic cert renewal.

Sorry, meant the Domain field in the MinIO Wizard (Apps -> Select installed MinIO -> Edit) This sets which URL will the browser use for redirections and such.

If this is not set, I'm more puzzled as to why it was working before

jadedeane commented 3 months ago

Hmm I suppose you have set the domain field right?

"Set the domain field" where exactly? Within TrueNAS, yup, box steady state, no changes. All "native" (iSCSI, NFS< SMB) services working as expected. MinIO, which is my only chart application, is the outlier. The screenshot above reads to me as a simple socket failure. As I mentioned previously, not remotely sure where this would die in, but things started failing around the time of automatic cert renewal.

Sorry, meant the Domain field in the MinIO Wizard (Apps -> Select installed MinIO -> Edit) This sets which URL will the browser use for redirections and such.

If this is not set, I'm more puzzled as to why it was working before

Ah, yes, Apps -> MinIO -> Edit, FQDN (same as nas itself) is configured.

I only happened to notice Velero backups were failing, tried to log into MinIO, and well, here we are :)

jadedeane commented 3 months ago

Deleted MinIO, fresh install (inclusive of MinIO dataset for storage), same settings.

Same problems.

I'm at a loss here. Has something gone sideways on the path to latest dragonfish?

jadedeane commented 3 months ago

Completely removed apps (i.e., unset pool, removed 'ix-applications' dataset', and selected pool again to re-create).

Installed MinIO again, and all is well.

stavros-k commented 3 months ago

Hmm, glad its working now. I'm not really sure what happened there. Please let me know if you see such issue again.

Thanks

reitermarkus commented 3 months ago

I'm having the same issue following the S3 to MinIO migration guide. I am currently still on Bluefin.

The login page takes ages to load in the first place. After it has loaded and I try logging in, I get the same timeout error. The domain is configured in the app wizard and the login page is using the correct certificate.

reitermarkus commented 3 months ago

@stavros-k, can this be reopened or should I open a new issue?