search

Smashing / smashing

The exceptionally handsome dashboard framework in Ruby and Coffeescript.
https://smashing.github.io/
MIT License
3.22k stars 324 forks source link

decrease memory consumption - Update sprockets... #156

Closed rompic closed 4 years ago

rompic commented 4 years ago

... to 4.0 for ruby >=2.5.0

sprockets 4.0 requires ruby 2.5.0 and tries to autoload sassc.

first of all I'm not a ruby dev, so this might not be the perfect solution and I'm looking forward to your feedback.

I'm running smashing on a Raspberry Pi 3 Model B Plus Rev 1.3 and run out of memory over time. I found some issues in my jobs, but I also want to ask if someone can help with this.

I used the approach described in https://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby

therefore I added rbtrace to my Gemfile and added the following to my config.ru file:

require 'rbtrace'
require 'objspace'

ObjectSpace.trace_object_allocations_start

after that I'm able to create a heap dump with:

bundle exec rbtrace -p <process-id> -e 'Thread.new{GC.start;require "objspace";io=File.open("/tmp/ruby-heap.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'

with the help of the heapy gem I ran some analysis: heapy read /tmp/ruby-heap.dump heapy read /tmp/ruby-heap.dump all

and found that sprockets/mime.rb is on the top of all the outputs. Something similar is discussed here: https://github.com/rails/sprockets/issues/432

before the change:


Analyzing Heap
==============
Generation: nil object count: 137947, mem: 0.0 kb
Generation:  41 object count: 433, mem: 12325.2 kb
Generation:  42 object count: 4383, mem: 4453.9 kb
Generation:  43 object count: 14595, mem: 1293.1 kb
Generation:  44 object count: 25235, mem: 2331.5 kb
Generation:  45 object count: 68681, mem: 6694.5 kb
Generation:  46 object count: 18672, mem: 1875.8 kb
Generation:  47 object count: 3, mem: 0.1 kb
Generation:  48 object count: 3, mem: 0.1 kb
Generation:  49 object count: 1665, mem: 4189.9 kb
Generation:  50 object count: 1123, mem: 361.2 kb
Generation:  51 object count: 973, mem: 20821.6 kb
Generation:  52 object count: 21, mem: 106.8 kb
Generation:  53 object count: 121, mem: 11.5 kb
Generation:  60 object count: 10, mem: 0.7 kb
Generation:  61 object count: 1328, mem: 1170.4 kb
Generation:  62 object count: 41, mem: 3.6 kb

Heap total
==============
Generations (active): 17
Count: 275234
Memory: 55639.9 kb

Analyzing Heap (Generation: all)
-------------------------------

allocated by memory (56975251) (in bytes)
==============================
  20988840  /usr/local/bundle/gems/eventmachine-1.2.7/lib/eventmachine.rb:1067
  14694944  /usr/local/bundle/gems/rufus-scheduler-3.4.2/lib/rufus/scheduler/jobs.rb:280
  10855744  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:121
   2210864  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/directive_processor.rb:119
   2097248  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:112
   1705600  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:118
   1622096  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets.rb:109
   1051542  eval:1
    326024  /usr/local/bundle/gems/coffee-script-2.4.1/lib/coffee_script.rb:42
    260136  /usr/local/bundle/gems/sass-3.4.25/lib/sass/cache_stores/base.rb:51
...

object count (137287)
==============================
  85232  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:121
  42640  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:118
    875  /usr/local/bundle/gems/rack-2.0.9/lib/rack/file.rb:3
    806  /usr/local/lib/ruby/2.5.0/set.rb:349
    267  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/cache/memory_store.rb:53
    260  /usr/local/lib/ruby/2.5.0/json/common.rb:156
    240  /usr/local/bundle/gems/thin-1.7.2/lib/thin/request.rb:84
    200  /usr/local/lib/ruby/2.5.0/set.rb:94
    154  /usr/local/bundle/gems/sawyer-0.8.2/lib/sawyer/serializer.rb:98
    153  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/uri_tar.rb:35
...

High Ref Counts
==============================

  118244  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:118
  117256  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:121
  85280  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/mime.rb:112
   1251  /usr/local/bundle/gems/rack-2.0.9/lib/rack/mime.rb:50
   1155  /usr/local/bundle/gems/tilt-2.0.10/lib/tilt/mapping.rb:243
   1029  /usr/local/lib/ruby/2.5.0/set.rb:136
    920  /usr/local/bundle/gems/rufus-scheduler-3.4.2/lib/rufus/scheduler/jobs.rb:280
    800  /usr/local/bundle/gems/thin-1.7.2/lib/thin/request.rb:59
    760  /usr/local/bundle/gems/eventmachine-1.2.7/lib/eventmachine.rb:1067
    538  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/loader.rb:198
    335  /usr/local/bundle/gems/sprockets-3.7.2/lib/sprockets/loader.rb:205
...

After the change:

Analyzing Heap
==============
Generation: nil object count: 142330, mem: 0.0 kb
Generation:  41 object count: 413, mem: 10275.0 kb
Generation:  42 object count: 630, mem: 4156.4 kb
Generation:  43 object count: 126, mem: 13.3 kb
Generation:  44 object count: 390, mem: 15.8 kb
Generation:  45 object count: 7, mem: 8.6 kb
Generation:  47 object count: 276, mem: 21.5 kb
Generation:  48 object count: 2621, mem: 569.0 kb
Generation:  49 object count: 7498, mem: 27654.7 kb
Generation:  51 object count: 423, mem: 51.7 kb
Generation:  55 object count: 22, mem: 1.2 kb
Generation:  63 object count: 45, mem: 1030.3 kb
Generation:  64 object count: 27, mem: 2.7 kb

Heap total
==============
Generations (active): 13
Count: 154808
Memory: 43800.4 kb

Analyzing Heap (Generation: all)
-------------------------------
allocated by memory (44851581) (in bytes)
==============================
  20988840  /usr/local/bundle/gems/eventmachine-1.2.7/lib/eventmachine.rb:1067
  14694848  /usr/local/bundle/gems/rufus-scheduler-3.4.2/lib/rufus/scheduler/jobs.rb:280
   2951288  /usr/local/bundle/gems/sprockets-4.0.0/lib/sprockets.rb:123
   2252010  /usr/local/bundle/gems/sprockets-4.0.0/lib/sprockets/preprocessors/default_source_map.rb:16
   1050658  eval:1
   1049320  /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/functions_handler.rb:23
    326024  /usr/local/bundle/gems/coffee-script-2.4.1/lib/coffee_script.rb:42
    193635  /usr/local/bundle/gems/sprockets-4.0.0/lib/sprockets/preprocessors/default_source_map.rb:44
     99104  /usr/local/lib/ruby/2.5.0/set.rb:136
     80860  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi_c.so:0
...

object count (12478)
==============================
  1125  /usr/local/bundle/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1333
   738  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi_c.so:0
   644  /usr/local/lib/ruby/2.5.0/set.rb:349
   474  /usr/local/lib/ruby/2.5.0/json/common.rb:156
   389  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:275
   360  /usr/local/lib/ruby/2.5.0/set.rb:136
   339  /usr/local/bundle/gems/sprockets-4.0.0/lib/sprockets/cache.rb:173
   297  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:265
   294  /usr/local/bundle/gems/sprockets-4.0.0/lib/sprockets/loader.rb:69
   216  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/types.rb:187
...

High Ref Counts
==============================

  1262  /usr/local/bundle/gems/rack-2.2.2/lib/rack/mime.rb:52
  1225  /usr/local/bundle/gems/tilt-2.0.10/lib/tilt/mapping.rb:243
  1063  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi_c.so:0
   978  /usr/local/lib/ruby/2.5.0/set.rb:136
   921  /usr/local/bundle/gems/rufus-scheduler-3.4.2/lib/rufus/scheduler/jobs.rb:280
   737  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:265
   735  /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/ffi.rb:34
   687  /usr/local/bundle/gems/eventmachine-1.2.7/lib/eventmachine.rb:1067
   557  /usr/local/lib/ruby/2.5.0/json/common.rb:156
   539  /usr/local/bundle/gems/sprockets-4.0.0/lib/sprockets/loader.rb:237
...
rompic commented 4 years ago

I assume we should get rid of the sass gem altogether and use sassc instead: https://sass-lang.com/blog/ruby-sass-is-unsupported

rompic commented 4 years ago

I think we can just use sassc as a drop in replacement and require it here instead of sass https://github.com/Smashing/smashing/blob/master/lib/dashing/app.rb#L6

and remove sass from the smashing.gemspec.

tests on my site look ok

kinow commented 4 years ago

I assume we should get rid of the sass gem altogether and use sassc instead: https://sass-lang.com/blog/ruby-sass-is-unsupported

+1

rompic commented 4 years ago

https://github.com/rails/sprockets/issues/588#issuecomment-438019730

sprockets 3 doesn't support sassc, so if you want to use sassc-rails you still need sass.

sprockets 4.0 requires ruby 2.5.0 or higher.

so i have required sass / sassc based on the ruby version.

rompic commented 4 years ago

note that sprockets seems to have a problem on some ruby versions: https://github.com/sass/sassc-ruby/issues/133

rompic commented 4 years ago

Two days ago sassc 2.3.0 has been released. looks ok as well.