search

dwrensha / gitlab-sandstorm

GitLab as a Sandstorm app
20 stars 8 forks source link

Unable to create new Gitlab grain #31

Closed dominicporter closed 4 years ago

dominicporter commented 4 years ago

Trying to create a new grain I get a 404 page.

Logs show this line which looks like the culprit:

E, [2020-04-01T11:58:09.618001 #21] ERROR -- : Unable to save project: Cannot allocate memory - /gitlab-shell/bin/gitlab-projects

I'm running Sandstorm on a DigitalOcean Droplet following the tuorial on how to transition from hosted server a few months ago.

Some of my previous GL grains do load, some do not. I am fairly certain I have created at least one successful GL grain since migrating, but not certain.

Full log as follows:

Debug log: Untitled GitLab repository
...** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns
+ mkdir -p /var/redis
+ mkdir -p /var/log
+ mkdir -p /var/repositories
+ mkdir -p /var/gitlab-satellites
+ mkdir -p /var/tmp/repositories
+ mkdir -p /var/uploads
+ mkdir -p /var/sqlite3
+ mkdir -p /var/migrations/
+ touch /var/migrations/20160421130527
+ cp initdb.sqlite3 /var/sqlite3/db.sqlite3
+ source continue.sh
+++ head -c 30
+++ base64 /dev/urandom
++ export SECRET_KEY_BASE=xxx
++ SECRET_KEY_BASE=xxx
++ set -x -e
++ head -c 30
++ base64 /dev/urandom
++ export TMPDIR=/var/tmp
++ TMPDIR=/var/tmp
++ rm -rf /var/tmp
++ mkdir -p /var/tmp
++ echo 'started redis-server: ' 0
started redis-server:  0
++ export SSH_CONNECTION=12345
++ SSH_CONNECTION=12345
++ cd gitlab
++ export GEM_HOME=/opt/ruby/gitlab-bundle/ruby/2.1.0
++ GEM_HOME=/opt/ruby/gitlab-bundle/ruby/2.1.0
++ '[' -f /var/migrations/20160421130527 ']'
++ echo 'no migration necessary'
no migration necessary
++ sleep 3
++ RUBYOPT=-r/opt/ruby/gitlab-bundle/bundler/setup
++ RAILS_ENV=production
++ /opt/ruby/gitlab-bundle/ruby/2.1.0/bin/unicorn_rails -p 10001 -E production -c /gitlab/config/unicorn.sandstorm.rb
++ awk '{print "sidekiq: " $0}'
++ RUBYOPT=-r/opt/ruby/gitlab-bundle/bundler/setup
++ RAILS_ENV=production
++ /opt/ruby/gitlab-bundle/ruby/2.1.0/bin/sidekiq -q post_receive -q default -q archive_repo
++ redis-server /etc/redis.conf
[20] 02 Apr 09:19:57.915 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
[20] 02 Apr 09:19:57.917 # Redis can't set maximum open files to 10032 because of OS error: Operation not permitted.
[20] 02 Apr 09:19:57.917 # Current maximum open files is 1024. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 2.8.17 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in stand alone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 20
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

[20] 02 Apr 09:19:57.919 # Server started, Redis version 2.8.17
[20] 02 Apr 09:19:57.919 * The server is now ready to accept connections on port 6379
I, [2020-04-02T09:19:58.441164 #23]  INFO -- : listening on addr=0.0.0.0:10001 fd=9
I, [2020-04-02T09:19:58.441294 #23]  INFO -- : worker=0 spawning...
I, [2020-04-02T09:19:58.442167 #23]  INFO -- : worker=1 spawning...
I, [2020-04-02T09:19:58.446213 #23]  INFO -- : master process ready
I, [2020-04-02T09:19:58.447094 #29]  INFO -- : worker=0 spawned pid=29
I, [2020-04-02T09:19:58.447328 #29]  INFO -- : Refreshing Gem list
I, [2020-04-02T09:19:58.447802 #31]  INFO -- : worker=1 spawned pid=31
I, [2020-04-02T09:19:58.449629 #31]  INFO -- : Refreshing Gem list
++ ../gitlab-workhorse/gitlab-workhorse -listenAddr 127.0.0.1:10000 -authBackend http://127.0.0.1:10001
2020/04/02 09:20:00 Starting gitlab-workhorse (unknown version)
sidekiq: 2020-04-02T09:20:16.432Z 21 TID-gnyth8i7s INFO: Booting Sidekiq 4.0.1 with redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}
sidekiq: 2020-04-02T09:20:16.450Z 21 TID-gnyth8i7s INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
sidekiq: 2020-04-02T09:20:16.470Z 21 TID-gnyth8i7s INFO: Cron Jobs - add job with name: repository_check_worker
sidekiq: 2020-04-02T09:20:16.477Z 21 TID-gnyth8i7s INFO: Cron Jobs - add job with name: admin_email_worker
sidekiq: 2020-04-02T09:20:16.483Z 21 TID-gnyth8i7s INFO: Cron Jobs - add job with name: repository_archive_cache_worker
I, [2020-04-02T09:20:18.845072 #31]  INFO -- : worker=1 ready
I, [2020-04-02T09:20:18.857623 #29]  INFO -- : worker=0 ready
I, [2020-04-02T09:20:18.868621 #31]  INFO -- : Started GET "/" for 127.0.0.1 at 2020-04-02 09:20:18 +0000
I, [2020-04-02T09:20:19.142185 #31]  INFO -- : Processing by RootController#index as HTML
I, [2020-04-02T09:20:19.159177 #31]  INFO -- : Authenticating Sandstorm
I, [2020-04-02T09:20:20.200754 #31]  INFO -- : User was successfully created.
I, [2020-04-02T09:20:20.220812 #31]  INFO -- : Done Authenticating Sandstorm
I, [2020-04-02T09:20:21.182566 #31]  INFO -- : Enqueued ActionMailer::DeliveryJob (Job ID: b55cbaeb-7cdc-41b5-9d95-dff5cc70ee32) to Sidekiq(mailers) with arguments: "Notify", "group_access_granted_email", "deliver_now", 1
E, [2020-04-02T09:20:21.823559 #31] ERROR -- : Unable to save project: Cannot allocate memory - /gitlab-shell/bin/gitlab-projects
I, [2020-04-02T09:20:21.857422 #31]  INFO -- : Redirected to https://ui-69826377c59227977df5e932ea726088.xxx.sandcats.io/gitlab/repo
I, [2020-04-02T09:20:21.861135 #31]  INFO -- : Completed 302 Found in 2715ms (ActiveRecord: 294.1ms)
ui-69826377c59227977df5e932ea726088.xxx.sandcats.io 127.0.0.1:50724 - - [2020-04-02 09:20:01.033962537 +0000 UTC] "GET / HTTP/1.1" 302 138 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 20.846159
I, [2020-04-02T09:20:22.083174 #31]  INFO -- : Started GET "/gitlab/repo" for 127.0.0.1 at 2020-04-02 09:20:22 +0000
I, [2020-04-02T09:20:22.210571 #31]  INFO -- : Processing by ProjectsController#show as HTML
I, [2020-04-02T09:20:22.211878 #31]  INFO -- :   Parameters: {"namespace_id"=>"gitlab", "id"=>"repo"}
I, [2020-04-02T09:20:22.378879 #31]  INFO -- : Filter chain halted as :project rendered or redirected
I, [2020-04-02T09:20:22.380822 #31]  INFO -- : Completed 404 Not Found in 165ms (Views: 49.3ms | ActiveRecord: 15.1ms)
2020/04/02 09:20:22 ErrorPage: serving predefined error page: 404
ui-69826377c59227977df5e932ea726088.xxx.sandcats.io 127.0.0.1:50740 - - [2020-04-02 09:20:22.051301135 +0000 UTC] "GET /gitlab/repo HTTP/1.1" 404 2275 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0.345981
sidekiq: 2020-04-02T09:20:24.233Z 21 TID-gnyth8i7s INFO: Running in ruby 2.1.8p440 (2015-12-16 revision 53160) [x86_64-linux]
sidekiq: 2020-04-02T09:20:24.234Z 21 TID-gnyth8i7s INFO: See LICENSE and the LGPL-3.0 for licensing details.
sidekiq: 2020-04-02T09:20:24.234Z 21 TID-gnyth8i7s INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
sidekiq: 2020-04-02T09:20:24.243Z 21 TID-gnyth8i7s INFO: Starting processing, hit Ctrl-C to stop
ocdtrekkie commented 4 years ago

The obvious question would be if you're hitting your DO droplet's memory limit when loading the grain. Maybe bigger repos are causing you to hit it?

dominicporter commented 4 years ago

So it turns out your rather obvious question was the right one... it looked like I had about half a gig free but when I rean htop in the box whilst creating a new gitlab grain it was spiking right up to 100% memory. I added another gig of swap apace (because I am too stingy to pay more for my droplet) and it worked first time. Might be one to put in docs somewhere (I will if I can find where) - "Best to have more than 1Gb RAM, or at least some swap space".