feat: service restart provides now per node status

harshavardhana commented 9 months ago

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers under the terms of the Apache 2 license. By creating this pull request I represent that I have the right to license the contributions to the project maintainers under the Apache 2 license.

Description

Motivation and Context

How to test this PR?

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Optimization (provides speedup with no functional changes)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] Fixes a regression (If yes, please add commit-id or PR # here)
[ ] Unit tests added/updated
[ ] Internal documentation updated
[ ] Create a documentation update request here

harshavardhana commented 9 months ago

This PR is not ready until https://github.com/minio/madmin-go/pull/263 is merged

klauspost commented 9 months ago

Server PR: https://github.com/minio/minio/pull/18826 for reference.

harshavardhana commented 9 months ago

This PR is ready with all the changes both restart/update so it is good to be reviewed and merged right away.

shtripat commented 9 months ago

In a local single node single disk setup started as minio server ./test the output for mc admin service restart ALIAS shows as below

./mc admin service restart myminio 
Restart command successfully sent to `myminio`. Type Ctrl-C to quit or wait to follow the status of the restart process.
┌────────────────┬────────┐
│ HOST           │ STATUS │
├────────────────┼────────┤
│ 127.0.0.1:9000 │ ✔      │
│                │ ✔      │
└────────────────┴────────┘
...
Restarted `myminio` successfully in 1 seconds

The second tick mark seems unwanted for this scenario.

whereas for a multi node setup looks fine as below

./mc admin service restart myminio
Restart command successfully sent to `myminio`. Type Ctrl-C to quit or wait to follow the status of the restart process.
┌────────────────┬────────┐
│ HOST           │ STATUS │
├────────────────┼────────┤
│ 127.0.0.1:9000 │ ✔      │
│ 127.0.0.1:9001 │ ✔      │
│ 127.0.0.1:9002 │ ✔      │
│ 127.0.0.1:9003 │ ✔      │
└────────────────┴────────┘

...!!
Restarted `myminio` successfully in 3 seconds

Feature wise code looks good to me.

harshavardhana commented 9 months ago

Gonna fix the single node output.

shtripat commented 9 months ago

Gonna fix the single node output.

Also with SNSD mode ./mc admin update myminio "https://dl.min.io/server/minio/release/linux-amd64/minio.RELEASE.2024-01-18T22-51-28Z.sha256sum" failed with error on MinIO side as below

panic: "POST /minio/admin/v3/update?dry-run=false&type=2&updateURL=https%3A%2F%2Fdl.min.io%2Fserver%2Fminio%2Frelease%2Flinux-amd64%2Fminio.RELEASE.2024-01-18T22-51-28Z.sha256sum": runtime error: invalid memory address or nil pointer dereference
goroutine 738 [running]:
runtime/debug.Stack()
    runtime/debug/stack.go:24 +0x5e
github.com/minio/minio/cmd.serverMain.func8.setCriticalErrorHandler.func2.1()
    github.com/minio/minio/cmd/generic-handlers.go:555 +0x97
panic({0x28080e0?, 0x62c2cf0?})
    runtime/panic.go:914 +0x21f
github.com/minio/pkg/v2/workers.(*Workers).Wait(...)
    github.com/minio/pkg/v2@v2.0.8/workers/workers.go:62
github.com/minio/minio/cmd.(*NotificationGroup).Wait(...)
    github.com/minio/minio/cmd/notification.go:101
github.com/minio/minio/cmd.(*NotificationSys).VerifyBinary(0xc001317740, {0x4c51ec0?, 0xc000ecd860}, 0xc08ff0e240, {0xc08ff01e40, 0x20, 0x40}, {0xc08ff195b1, 0x22}, {0xc09c042000, ...})
    github.com/minio/minio/cmd/notification.go:390 +0x38d
github.com/minio/minio/cmd.adminAPIHandlers.ServerUpdateV2Handler({}, {0x4c4c5e0, 0xc000cd2280}, 0xc08ff14300)
    github.com/minio/minio/cmd/admin-handlers.go:148 +0x645
net/http.HandlerFunc.ServeHTTP(0x52f303b?, {0x4c4c5e0?, 0xc000cd2280?}, 0x4c7f108?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.adminMiddleware.func1.httpTraceAll.httpTrace.func1({0x4c4c5e0, 0xc000cd2280}, 0xc000cd2280?)
    github.com/minio/minio/cmd/http-tracer.go:189 +0xef
github.com/minio/minio/cmd.adminMiddleware.func1({0x4c4c5e0, 0xc000cd2280}, 0xc08ff14200)
    github.com/minio/minio/cmd/admin-router.go:132 +0x2df
net/http.HandlerFunc.ServeHTTP(0x62d40a0?, {0x4c4c5e0?, 0xc000cd2280?}, 0x4?)
    net/http/server.go:2136 +0x29
github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0x4c4bc50, 0xc08ff30000}, 0x1b?)
    github.com/klauspost/compress@v1.17.4/gzhttp/compress.go:497 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc000ecd1a0?, {0x4c4bc50?, 0xc08ff30000?}, 0x0?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setBucketForwardingMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
    github.com/minio/minio/cmd/generic-handlers.go:460 +0x2fc
net/http.HandlerFunc.ServeHTTP(0x7f193e5eeee0?, {0x4c4bc50?, 0xc08ff30000?}, 0xc000100000?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setUploadForwardingMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
    github.com/minio/minio/cmd/generic-handlers.go:573 +0x1e5
net/http.HandlerFunc.ServeHTTP(0x2b3ee40?, {0x4c4bc50?, 0xc08ff30000?}, 0x39?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setRequestValidityMiddleware.func1({0x4c4bc50?, 0xc08ff30000}, 0xc08ff14200)
    github.com/minio/minio/cmd/generic-handlers.go:443 +0x168d
net/http.HandlerFunc.ServeHTTP(0xc000ecd500?, {0x4c4bc50?, 0xc08ff30000?}, 0xc000fe4cf0?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setRequestLimitMiddleware.func1({0x4c4bc50?, 0xc08ff30000}, 0xc08ff14200)
    github.com/minio/minio/cmd/generic-handlers.go:134 +0x662
net/http.HandlerFunc.ServeHTTP(0x2f?, {0x4c4bc50?, 0xc08ff30000?}, 0x2253fab?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setCrossDomainPolicyMiddleware.func1({0x4c4bc50?, 0xc08ff30000?}, 0x6?)
    github.com/minio/minio/cmd/crossdomain-xml-handler.go:42 +0xf1
net/http.HandlerFunc.ServeHTTP(0xc08ff14200?, {0x4c4bc50?, 0xc08ff30000?}, 0xc0017e2110?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setBrowserRedirectMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
    github.com/minio/minio/cmd/generic-handlers.go:164 +0x19c
net/http.HandlerFunc.ServeHTTP(0x0?, {0x4c4bc50?, 0xc08ff30000?}, 0x871c46f?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setAuthMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
    github.com/minio/minio/cmd/auth-handler.go:636 +0x6a2
net/http.HandlerFunc.ServeHTTP(0x4c51ec0?, {0x4c4bc50?, 0xc08ff30000?}, 0x4c2a490?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.httpTracerMiddleware.func1({0x4c4c1f0, 0xc08ff1a000}, 0xc08ff14100)
    github.com/minio/minio/cmd/http-tracer.go:89 +0x3b4
net/http.HandlerFunc.ServeHTTP(0x29faa00?, {0x4c4c1f0?, 0xc08ff1a000?}, 0xa?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.addCustomHeadersMiddleware.func1({0x4c4c1f0, 0xc08ff1a000}, 0xc000ecd3e0?)
    github.com/minio/minio/cmd/generic-handlers.go:538 +0x438
net/http.HandlerFunc.ServeHTTP(0xc08ff14000?, {0x4c4c1f0?, 0xc08ff1a000?}, 0x69d73f?)
    net/http/server.go:2136 +0x29
github.com/minio/mux.(*Router).ServeHTTP(0xc000dcaa80, {0x4c4c1f0, 0xc08ff1a000}, 0xc08ff14000)
    github.com/minio/mux@v1.9.0/mux.go:214 +0x1f7
github.com/minio/minio/cmd.corsHandler.(*Cors).Handler.func2({0x4c4c1f0, 0xc08ff1a000}, 0xc08ff14000)
    github.com/rs/cors@v1.10.1/cors.go:281 +0x184
net/http.HandlerFunc.ServeHTTP(0x4186e8?, {0x4c4c1f0?, 0xc08ff1a000?}, 0xf?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.serverMain.func8.setCriticalErrorHandler.func2({0x4c4c1f0?, 0xc08ff1a000?}, 0x63474e0?)
    github.com/minio/minio/cmd/generic-handlers.go:562 +0x78
net/http.HandlerFunc.ServeHTTP(0xc001646ac8?, {0x4c4c1f0?, 0xc08ff1a000?}, 0x727385?)
    net/http/server.go:2136 +0x29
github.com/minio/minio/internal/http.(*Server).Init.func1({0x4c4c1f0?, 0xc08ff1a000?}, 0xc00131e300?)
    github.com/minio/minio/internal/http/server.go:123 +0x19c
net/http.HandlerFunc.ServeHTTP(0x4109a5?, {0x4c4c1f0?, 0xc08ff1a000?}, 0xc08ff1a001?)
    net/http/server.go:2136 +0x29
net/http.serverHandler.ServeHTTP({0x4c44f38?}, {0x4c4c1f0?, 0xc08ff1a000?}, 0x6?)
    net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc08ff0e000, {0x4c51ec0, 0xc000e72570})
    net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 94
    net/http/server.go:3086 +0x5cb

whereas multi node showed message

Server update request sent successfully `myminio`
┌────────────────┬────────────────────────────────────────────────────────────────────┐
│ HOST           │ STATUS                                                             │
├────────────────┼────────────────────────────────────────────────────────────────────┤
│ 127.0.0.1:9002 │ server is already running the latest version: 2023-08-21T19-40-07Z │
│ 127.0.0.1:9003 │ server is already running the latest version: 2023-08-21T19-40-07Z │
│ 127.0.0.1:9000 │ server is already running the latest version: 2023-08-21T19-40-07Z │
│ 127.0.0.1:9001 │ server is already running the latest version: 2023-08-21T19-40-07Z │
└────────────────┴────────────────────────────────────────────────────────────────────┘

harshavardhana commented 9 months ago

Yes server side is fixed @shtripat

commit c88308cf0e6401ca034561357b6e7c932b11a516 (HEAD -> master, origin/master)
Author: Harshavardhana <harsha@minio.io>
Date:   Fri Jan 26 12:07:03 2024 -0800

    avoid 'panic' on mc admin update for single drive setup (#18876)

minio / mc