search

maxisam / mgob

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.
https://maxisam.github.io/mgob/
MIT License
152 stars 22 forks source link

Restore not working #134

Open boomfly opened 3 weeks ago

boomfly commented 3 weeks ago

Backup work, but restore throws panic

   2024/10/27 06:33:00 "POST http://localhost:8090/restore/apartx-kz/apartx-kz-1669572000.gz HTTP/1.1" from [::1]:51854 - 000 0B in 1.153071ms

 panic: runtime error: invalid memory address or nil pointer dereference

 -> github.com/stefanprodan/mgob/pkg/backup.RunRestore
 ->   /go/src/github.com/stefanprodan/mgob/pkg/backup/validate.go:90

    github.com/stefanprodan/mgob/pkg/restore.Run
      /go/src/github.com/stefanprodan/mgob/pkg/restore/restore.go:33
    github.com/stefanprodan/mgob/pkg/api.postRestore
      /go/src/github.com/stefanprodan/mgob/pkg/api/restore.go:33
    net/http.HandlerFunc.ServeHTTP
      /usr/local/go/src/net/http/server.go:2136
    github.com/go-chi/chi/v5.(*Mux).routeHTTP
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/mux.go:444
    net/http.HandlerFunc.ServeHTTP
      /usr/local/go/src/net/http/server.go:2136
    github.com/stefanprodan/mgob/pkg/api.(*HttpServer).Start.func4.configCtx.func1.1
      /go/src/github.com/stefanprodan/mgob/pkg/api/server.go:86
    net/http.HandlerFunc.ServeHTTP
      /usr/local/go/src/net/http/server.go:2136
    github.com/go-chi/chi/v5.(*Mux).ServeHTTP
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/mux.go:73
    github.com/go-chi/chi/v5.(*Mux).Mount.func1
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/mux.go:316
    net/http.HandlerFunc.ServeHTTP
      /usr/local/go/src/net/http/server.go:2136
    github.com/go-chi/chi/v5.(*Mux).routeHTTP
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/mux.go:444
    net/http.HandlerFunc.ServeHTTP
      /usr/local/go/src/net/http/server.go:2136
    github.com/go-chi/chi/v5/middleware.init.0.RequestLogger.func1.1
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/middleware/logger.go:54
    net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2136
    github.com/go-chi/chi/v5/middleware.Recoverer.func1
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/middleware/recoverer.go:45
    net/http.HandlerFunc.ServeHTTP
      /usr/local/go/src/net/http/server.go:2136
    github.com/go-chi/chi/v5.(*Mux).ServeHTTP
      /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.10/mux.go:90
    net/http.serverHandler.ServeHTTP
      /usr/local/go/src/net/http/server.go:2938
    net/http.(*conn).serve
      /usr/local/go/src/net/http/server.go:2009
    created by net/http.(*Server).Serve in goroutine 12
      /usr/local/go/src/net/http/server.go:3086

Manual exec cmd that mgob produce is working

time="2024-10-27T06:28:46Z" level=info msg="Running restore command with : mongorestore --archive=/storage/apartx-kz/apartx-kz-1669572000.gz --gzip ...

Startup logs:

time="2024-10-27T06:18:23Z" level=debug msg="log level set to debug"
time="2024-10-27T06:18:23Z" level=info msg="mgob 2.0.20"
time="2024-10-27T06:18:23Z" level=info msg="starting with config: &{LogLevel:debug JSONLog:false Host: Port:8090 ConfigPath:/config StoragePath:/storage TmpPath:/tmp DataPath:/data Version:2.0.20 UseAwsCli:false HasGpg:false}"
time="2024-10-27T06:18:23Z" level=info msg="mongodump version: 100.8.0 git version: 732ddfaa6b467ffcd5bfa69a455953320eed85f4 Go version: go1.21.1    os: linux    arch: amd64    compiler: gc "
time="2024-10-27T06:18:23Z" level=info msg="NAME:   mc version - manage bucket versioning  USAGE:   mc version COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]  COMMANDS:   enable   enable bucket versioning   suspend  suspend bucket versioning   info     show bucket versioning status    FLAGS:   --config-dir value, -C value  path to configuration folder (default: \"/root/.mc\") [$MC_CONFIG_DIR]   --quiet, -q                   disable progress bar display [$MC_QUIET]   --no-color                    disable color theme [$MC_NO_COLOR]   --json                        enable JSON lines formatted output [$MC_JSON]   --debug                       enable debug output [$MC_DEBUG]   --insecure                    disable SSL certificate verification [$MC_INSECURE]   --limit-upload value          limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]   --limit-download value        limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]   --help, -h                    show help    "
time="2024-10-27T06:18:25Z" level=info msg="aws-cli/1.29.44 Python/3.11.6 Linux/6.8.0-45-generic botocore/1.31.44 "
time="2024-10-27T06:18:25Z" level=warning msg="gpg failed /bin/sh: gpg: not found : exit status 127"
time="2024-10-27T06:18:28Z" level=info msg="Google Cloud SDK 445.0.0 bq 2.0.97 bundled-python3-unix 3.9.16 core 2023.09.01 gcloud-crc32c 1.0.0 gsutil 5.25 "
time="2024-10-27T06:18:32Z" level=info msg="WARNING: You have 2 update(s) available. Consider updating your CLI installation with 'az upgrade' azure-cli                         2.52.0 * "
time="2024-10-27T06:18:32Z" level=info msg="rclone v1.65.0 "
maxisam commented 3 weeks ago

I am not sure why, I think it works sometime but not always. But unfortunately, i don't have bandwidth to address this now.

maxisam commented 3 weeks ago

related to #114

boomfly commented 3 weeks ago

Here's a clearer version in English:

No, it’s not.

I found the issue: the validation section is required for the restore process:

validation:
  database:
    host: 127.0.0.1
    port: 27017
    database: db_name

However, there’s a new problem:

time="2024-10-27T15:20:44Z" level=info msg="Running restore command with : mongorestore --archive=/storage/apartx-kz/apartx-kz-1730015066.gz --gzip --host "***" --port 27017 -u \"admin\" --nsInclude apartx.* --authenticationDatabase admin " plan=apartx-kz
time="2024-10-27T15:20:44Z" level=info msg="Validation: restore backup with : mongorestore --archive=/storage/apartx-kz/apartx-kz-1730015066.gz --gzip --host 127.0.0.1 --port 27017 --nsInclude apartx.* " plan=apartx-kz
2024/10/27 15:20:47 "GET http://10.42.3.126:8090/status HTTP/1.1" from 10.42.3.1:44596 - 200 57B in 47.848µs

The logs indicate that the restore process begins and shows the correct command for it. However, while the validation command runs, it seems the actual restore command is not being executed.

The restore process isn’t starting because the necessary code for it is missing in the source files.