sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.87k stars 2.14k forks source link

backup fails 10.6.3 and 10.6.4 #1576

Open gimler opened 6 years ago

gimler commented 6 years ago

i have actual 10.5.6 backup works.

in booth repos bla and docker i have diabled the wiki.

when i try to upgrade to 10.6.3 or 10.6.4 i get the follwing error

Dumping repositories ...
 * gimler/bla ... [DONE]
 * gimler/bla.wiki ...  [DONE]
 * gimler/docker ... [DONE]
 * gimler/docker.wiki ... rake aborted!
Gitlab::Git::CommandError: 14:Connect Failed
/home/git/gitlab/lib/gitlab/git/repository.rb:1370:in `rescue in gitaly_migrate'
/home/git/gitlab/lib/gitlab/git/repository.rb:1363:in `gitaly_migrate'
/home/git/gitlab/lib/gitlab/git/repository.rb:232:in `has_local_branches?'
/home/git/gitlab/app/models/repository.rb:515:in `has_visible_content?'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:17:in `block (2 levels) in cache_method'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `block in fetch'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `block in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in `save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache.rb:22:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:54:in `cache_method_output'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:16:in `block in cache_method'
/home/git/gitlab/app/models/repository.rb:489:in `empty?'
/home/git/gitlab/lib/backup/repository.rb:199:in `empty_repo?'
/home/git/gitlab/lib/backup/repository.rb:51:in `block in dump'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each'
/home/git/gitlab/lib/backup/repository.rb:9:in `dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:82:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
GRPC::Unavailable: 14:Connect Failed
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:372:in `request_response'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/grpc-1.10.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:127:in `call'
/home/git/gitlab/lib/gitlab/gitaly_client/repository_service.rb:74:in `has_local_branches?'
/home/git/gitlab/lib/gitlab/git/repository.rb:234:in `block in has_local_branches?'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:255:in `block (2 levels) in migrate'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:293:in `allow_n_plus_1_calls'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:250:in `block in migrate'
/home/git/gitlab/lib/gitlab/metrics/influx_db.rb:98:in `measure'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:248:in `migrate'
/home/git/gitlab/lib/gitlab/git/repository.rb:1364:in `gitaly_migrate'
/home/git/gitlab/lib/gitlab/git/repository.rb:232:in `has_local_branches?'
/home/git/gitlab/app/models/repository.rb:515:in `has_visible_content?'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:17:in `block (2 levels) in cache_method'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `block in fetch'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `block in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in `save_block_result_to_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache.rb:22:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:54:in `cache_method_output'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:16:in `block in cache_method'
/home/git/gitlab/app/models/repository.rb:489:in `empty?'
/home/git/gitlab/lib/backup/repository.rb:199:in `empty_repo?'
/home/git/gitlab/lib/backup/repository.rb:51:in `block in dump'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each'
/home/git/gitlab/lib/backup/repository.rb:9:in `dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:82:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)
chrislovsund commented 6 years ago

I found this https://gitlab.com/gitlab-org/gitlab-ce/issues/44003 and it works when making sure that gitly (and gitlab) is running during manual restore. I haven't been able to make it work in our deployment pipeline yet.

sunbear0226 commented 6 years ago

As a short-term solution, you can create a backup file by executing the following command.

docker exec -it gitlab_container_name bash
bundle exec rake gitlab:backup:create RAILS_ENV=production
LordFPL commented 6 years ago

Or in one line :

docker exec -it gitlab bash -c 'supervisorctl stop unicorn sshd ; source ${GITLAB_RUNTIME_DIR}/functions ; execute_raketask gitlab:backup:create SKIP=$GITLAB_BACKUP_SKIP ; supervisorctl start sshd unicorn'
paddy-hack commented 6 years ago

For me, this seems to work

docker-compose exec gitlab su -c 'bundle exec rake gitlab:backup:create' git

I haven't figured out yet how to restore from the backup that that generates yet :unamused:

gimler commented 6 years ago

@paddy-hack i can confirm it works as expected.

The documentation should be updated.

Kristaba commented 6 years ago

Same issue here.

@gimler I think the solution from @LordFPL is to be preferred if working, as unicorn and sshd are stopped during the backup process. For what I understand from official Gitlab documentation and the docker image one, backups should not be created on a running instance, or at least not by this way. If there is, even with low probability, any chance of corruptions of the backup'ed data, it should not be recommended at all.

I am not skilled enough to know what is the safest way to do the backups under these conditions. But if someone here is, it would be great if the documentation could be updated. Maybe, at least, a simple warning to say that the current explanations are not correct anymore with a link to this issue?

pilla commented 6 years ago

Did anybody get the restore process to work? edit: nevermind, finally got it working.

paddy-hack commented 6 years ago

Oops! Sorry for not reporting back on that :flushed: Here's what has been working for me, since 10.8.3 at the very least, possibly also 10.7.x.

docker-compose exec gitlab supervisorctl stop unicorn
docker-compose exec gitlab supervisorctl stop sidekiq
docker-compose exec gitlab su -c 'bundle exec rake gitlab:backup:restore' git
docker-compose exec gitlab supervisorctl start sidekiq
docker-compose exec gitlab supervisorctl start unicorn
matejdujic commented 6 years ago

Can confirm that it is issue in 11.1.4 as well and @paddy-hack workaround works here.

partypeopleland commented 5 years ago

Great Job ! Thanks @paddy-hack It's Work!!

Windows 10 / docker for windows 18.06.1-ce / ameersbn/gitlab:11.3.0

nanquanyuhao commented 5 years ago

I have run into the problem...

almereyda commented 5 years ago

I used the last @paddy-hack sequence of commands also for backup:create and it worked.

This may be useful for #1744 and #1539 as well.

paddy-hack commented 5 years ago

I recently tested created and restoring with 11.4.3 using

docker-compose exec gitlab su -c 'bundle exec rake gitlab:backup:create' git

to create the backup and

docker-compose exec gitlab su -c 'bundle exec rake gitlab:backup:restore' git

to restore. No more need to stop and start unicorn and sidekiq.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.