Restore fails when gitaly isn't running

CommanderTso commented 5 years ago

Tying together #1655 and #1744.

I have a backup created in docker-gitlab, Gitlab v11.5.3, which I then restored on a separate VM using the instructions per the readme.

This fails on the "Restoring Repositories" step with:

rake aborted!
GRPC::Unavailable: 14:Connect Failed
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/grpc-1.15.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:136:in `call'
/home/git/gitlab/lib/gitlab/gitaly_client/storage_service.rb:19:in `delete_all_repositories'
/home/git/gitlab/lib/backup/repository.rb:45:in `block in prepare_directories'
/home/git/gitlab/lib/backup/repository.rb:44:in `each'
/home/git/gitlab/lib/backup/repository.rb:44:in `prepare_directories'
/home/git/gitlab/lib/backup/repository.rb:77:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:restore
(See full trace by running task with --trace)

I did some digging, and the issue here is that gitaly isn't running. I was able to get the backup to restore by, while in the container that I was executing the restore with, getting gitaly up and running before that step was executed. After that, everything worked fine.

I'd write a PR for the readme, but I'm not sure exactly how you'd like folks to approach doing a restore; my solution (and the others in the linked tickets) are pretty hacky. If you have some guidance on approach, I'd be happy to take a shot at a fix.

Prash45 commented 5 years ago

Hello everyone,

I am seeing the same issue while restoring the gitlab in a docker container. I have followed the official gitlab documentation for backup and restore https://docs.gitlab.com/ee/raketasks/backup_restore.html . We are using CE - 11.3.5 version. I was able to backup gitlab and copied to new container location. when executed restore command with --trace option, it started unpacking and restoring some of the repositories and couple minutes later it suddely fails in the middle with GRPC OS: error.

I am pasting the output I got below, How can I fix this issue?

Thank you all

`* HQITD51FSWEBD/H5-EMVFunding ... [DONE]
 * Starfire/Starfire ... [DONE]
 * Harvest/SourceSafeHQDATA03 ... Error: 14:OS Error
[Failed] restoring Harvest/SourceSafeHQDATA03 repository
 * HQITD51FSWEBD/H5-CAPS ... rake aborted!
GRPC::Unavailable: 14:OS Error
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:372:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:134:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:35:in `gitaly_client_call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in `add'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:280:in `add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:60:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:81:in `block in restore'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:78:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:restore
`

Prash45 commented 5 years ago

@nanquanyuhao @zer01ike @QuickJack

Can anyone suggest me how to fix this issue?

CommanderTso commented 5 years ago

@Prash45 I was able to work around this. My steps were:

Start up the docker-gitlab container in bash
Connect to the container and make sure all services are off except Gitaly
Still within that same container instance, run the restore rake command

I erred on the side of making sure that GitLab itself wasn't running, since that is recommended for restoring backups in the docker-gitlab repo - so I made sure only Gitaly was running in the container.

Hope this helps.

Prash45 commented 5 years ago

@CommanderTso , I have tried as you did, stopped all services except Gitaly, other services are down.

When started restore rake command, it unpacked tar and restored some of the repositories, later it failed again. now it says, GRPC Unavailable 14: Socket closed .. Gitlab container also existed automatically again.

Can you look the below output. Thank you

 * stargate/stargate ... Error: 14:Socket closed
[Failed] restoring stargate/stargate repository
 * CompServicesGroup/Paycheck_Management ... rake aborted!
GRPC::Unavailable: 14:Socket closed
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:372:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:134:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:35:in `gitaly_client_call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in `add'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:280:in `add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:60:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:81:in `block in restore'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:78:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:restore

CommanderTso commented 5 years ago

@Prash45 I would assume that, by the time you're getting to the restore process, the container was either restarted or killed and recreated, and Gitaly was again off.

I've updated my above steps with a more exact repro of how I did this - take another look at see if that helps?

Prash45 commented 5 years ago

@CommanderTso , I have followed just as below

We have swarm set up , so I have deployed stack from host and it created container on git server.
I have logged into git server and did "docker ps" and it showed gitlab container is healthy
I did docker exec -it "container ID" bash
Now I am inside the container with bash shell.
I have checked the backup tar file at , /var/opt/gitlab/backups..and its there
I have run gitlab-ctl stop on all services except Gitaly
I made sure all the services are down except Gitaly with "gitlab-ctl status". Its good as excepted
Now I ran restore rake command inside the container

Still it did the same thing, Untar and restoring some repo's and failed with below output

Let me know If I did anything wrong in my steps

Restoring repositories ...
 * nguyentul/ereceipt ... Error: 14:Socket closed
[Failed] restoring nguyentul/ereceipt repository
 * kumarip/ROBA ... rake aborted!
GRPC::Unavailable: 14:Socket closed
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:134:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:35:in `gitaly_client_call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in `add'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:280:in `add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:60:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:81:in `block in restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:78:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:restore
(See full trace by running task with --trace)

Prash45 commented 5 years ago

@serenader2014 @bradleydwyer @scheiblr

Can anyone of know how to fix this? Thank you all

solidnerd commented 5 years ago

Hey @Prash45,

which image are you currently using ? Because it seems that you use gitlab/gitlab-ce and not sameersbn/gitlab . These are different ways of using GitLab.

sameersbn/gitlab

Is an installation from source without omnibus. This won't have a gitlab-ctl.

gitlab/gitlab-ce

Uses Omnibus to manage all the process inside.

solidnerd commented 5 years ago

Hey @CommanderTso ,

This issue is problem because gitaly is changed as a core requirement. This means it needs to be run when executing rake tasks relating to git because gitaly does this git commands.

The current way you describe is a workaround with this image. But it would work and I think I would also do it so.

A better approach would be to separate gitaly from this gitlab container and start a separate gitaly container so it's easier to handle from the docker side. But this would be a breaking change. I think we could takle this with 12.X .

Prash45 commented 5 years ago

Hey @Prash45,

which image are you currently using ? Because it seems that you use gitlab/gitlab-ce and not sameersbn/gitlab . These are different ways of using GitLab.

sameersbn/gitlab

Is an installation from source without omnibus. This won't have a gitlab-ctl.

gitlab/gitlab-ce

Uses Omnibus to manage all the process inside.

@solidnerd Hello,

I am using gitlab/gitlab-ce :11.3.5 image ..Do you know how can I fix this issue

CommanderTso commented 5 years ago

Having now backed up and restored a couple times, here are refined instructions on how to run backup and restore. Do not stop your usual GitLab container before doing this.

docker exec -it gitlab bash
supervisorctl stop all active services except gitaly
Still from inside the container, run bundle exec rake gitlab:backup:create or bundle exec rake gitlab:backup:restore, as needed
When done, turn back on all deactivated services with supervisorctl start

iamchanghyunpark commented 5 years ago

@CommanderTso Hi, thank you for the refined instructions!

I was going to ask you how to list the the running services, but running supervisorctl (without the stop argument) sent me to interactive shell and listed the services.

So will this go onto the README? I think this is essential information to backup!

jnbt commented 5 years ago

While @CommanderTso solution is working, it feels more like a workaround. Currently I can only assume that one would need to update the entrypoint for this "specific" use case. @solidnerd did you ever encounter this issue yourself?

solidnerd commented 5 years ago

Hey @jnbt, I did not have directly the problem because I use the cron for backups in the container so this won't happen.

The way like @CommanderTso is a possible way to do your backups and not the most user friendly.

I noted before that a better solving of the problem would to change the container architecture to a separate container https://github.com/sameersbn/docker-gitlab/issues/1836#issuecomment-452982900.

I'm working currently on a migration path that uses the same way but with the old behaviour in #1951. I think it will land in 12.0.0.

I hope this answer your question .

jnbt commented 5 years ago

@solidnerd I think your migration path is a good solution. Thanks!

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.

CommanderTso commented 4 years ago

Looks like this got fixed with the merge of https://github.com/sameersbn/docker-gitlab/pull/1951. Closing. Thanks!

sameersbn / docker-gitlab

Restore fails when gitaly isn't running #1836