fastmonkeys / stellar

Fast database snapshot and restore tool for development
MIT License
3.86k stars 119 forks source link

stellar GC: psycopg2.errors.DiskFull error after exiting 0 #82

Open liavkoren opened 4 years ago

liavkoren commented 4 years ago

Hi. I'm snapshoting a large database on my laptop. It looks like a subprocess call or similar stellar is making is falling over when I run gc after taking the snapshot.

This is what I'm seeing:

$ df -g
Filesystem    1G-blocks Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk1s5        465   10       137     7%  487365 4881965515    0%   /
[truncated...]
$ stellar list
snap2: 5 days ago
$ stellar snapshot fresh-staging-snapshot
Snapshotting database suicidegirls
$ echo $?
0
$ stellar list
fresh-staging-snapshot: a minute ago
snap2: 5 days ago
$ df -g
Filesystem    1G-blocks Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk1s5        465   10        45    19%  487365 4881965515    0%   /
[snip]
$ stellar gc  # note: this command exits without any immediate error...
$ df -g  # note here that Available Capacity has gone down, some background process is still working? 
Filesystem    1G-blocks Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk1s5        465   10        34    23%  487365 4881965515    0%   /  
[snip]

[A minute or two goes by...]

$ Traceback (most recent call last):
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    cursor, statement, parameters, context
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.DiskFull: could not write to file "base/8001750/7150731": No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/liavkoren/Envs/suicidegirls/bin/stellar", line 11, in <module>
    load_entry_point('stellar==0.4.5', 'console_scripts', 'stellar')()
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/stellar/command.py", line 279, in main
    stellar()
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/stellar/command.py", line 70, in snapshot
    app.create_snapshot(name, before_copy=before_copy)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/stellar/app.py", line 125, in create_snapshot
    self.start_background_slave_copy(snapshot)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/stellar/app.py", line 191, in start_background_slave_copy
    self.inline_slave_copy(snapshot)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/stellar/app.py", line 198, in inline_slave_copy
    table.get_table_name('slave')
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/stellar/operations.py", line 72, in copy_database
    from_database
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 976, in execute
    return self._execute_text(object_, multiparams, params)
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1151, in _execute_text
    parameters,
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1288, in _execute_context
    e, statement, parameters, cursor, context
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1482, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    cursor, statement, parameters, context
  File "/Users/liavkoren/Envs/suicidegirls/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.errors.DiskFull) could not write to file "base/8001750/7150731": No space left on device

[SQL:
                CREATE DATABASE "stellar_281b2e2b9da9b17b" WITH TEMPLATE "stellar_2160785db75d7d7b";
            ]
(Background on this error at: http://sqlalche.me/e/e3q8)

$ df -g
Filesystem    1G-blocks Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk1s5        465   10        64    14%  487365 4881965515    0%   /
liavkoren commented 4 years ago

I'm pretty sure stellar junk is taking up a massive amount of my disk and being able to run GC to clean it out is important..