search

jazzband / django-dbbackup

Management commands to help backup and restore your project database and media files
BSD 3-Clause "New" or "Revised" License
978 stars 220 forks source link

Run command on celery #251

Closed tporto closed 7 years ago

tporto commented 7 years ago

How do I run the backup on celery? I already have the celery configured

ZuluPro commented 7 years ago

There are many solutions:

RidenShark commented 4 years ago

@ZuluPro, can you a bit more specific?

Django's docs say

from django.core import management
from django.core.management.commands import loaddata

management.call_command('flush', verbosity=0, interactive=False)
management.call_command('loaddata', 'test_data', verbosity=0)
management.call_command(loaddata.Command(), 'test_data', verbosity=0)

So where should I import the dbbackup command from? Sorry, I am a starter. Can you give me an example?

I am trying to do something like management.call_command('dbbackup') with celery beat. Can you help?

ZuluPro commented 4 years ago

Hi @RidenShark ,

I would just write:

management.call_command('dbbackup', interactive=False)

I think it's done in functional and unit tests

Glad to help

RidenShark commented 4 years ago

@ZuluPro, I am not sure if this error is caused by celery or dbbackup. Apologies if this doesn't belong in this thread. I am posting it here as it is still related to the command. Delete it if it is irrelevant.

I am getting django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread error when I try to run the command in a celery beat task.

from __future__ import absolute_import, unicode_literals

from myproject.celery import app
from celery.utils.log import get_task_logger

from celery.schedules import crontab
from django.core import management

@app.task
def db_backup_to_dropbox():
    management.call_command('dbbackup', interactive=False)

app.conf.beat_schedule = {
    'backup-every-second': {
        'task': 'core.tasks.db_backup_to_dropbox',
        'schedule': crontab(minute='*/1'),
        'args': (),
    },
}

Traceback:

[2020-07-25 16:24:00,050: INFO/MainProcess] Received task: core.tasks.db_backup_to_dropbox[a34653fe-a3c6-4259-a98d-7613c5f3446c]
[2020-07-25 16:24:00,054: INFO/MainProcess] Backing Up Database: myproject
[2020-07-25 16:24:00,213: INFO/MainProcess] Writing file to default-myserver-loc1-2020-07-25-162400.dump
[2020-07-25 16:24:00,215: INFO/MainProcess] Request to files/get_metadata
[2020-07-25 16:24:00,287: ERROR/MainProcess] RecursionError: maximum recursion depth exceeded while calling a Python object
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 118, in wrapper
    func(*args, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle
    self._save_new_backup(database)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup
    self.write_to_storage(outputfile, filename)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
    self.storage.write_file(file, path)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "/myproject/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 51, in save
    name = self.get_available_name(name, max_length=max_length)
  File "/myproject/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 82, in get_available_name
    while self.exists(name) or (max_length and len(name) > max_length):
  File "/myproject/venv/lib/python3.8/site-packages/storages/backends/dropbox.py", line 93, in exists
    return bool(self.client.files_get_metadata(self._full_path(name)))
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/base.py", line 1682, in files_get_metadata
    r = self.request(
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/dropbox.py", line 313, in request
    res = self.request_json_string_with_retry(host,
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/dropbox.py", line 463, in request_json_string_with_retry
    return self.request_json_string(host,
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/dropbox.py", line 559, in request_json_string
    r = self._session.post(url,
  File "/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen    httplib_response = self._make_request(
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connection.py", line 343, in connect
    self.ssl_context = create_urllib3_context(
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 277, in create_urllib3_context
    context.options |= options
  File "/usr/lib/python3.8/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.8/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.8/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 460 more times]

[2020-07-25 16:24:00,325: ERROR/MainProcess] Task core.tasks.db_backup_to_dropbox[a34653fe-a3c6-4259-a98d-7613c5f3446c] raised unexpected: DatabaseError("DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140423029811616 and this is thread id 140422957233632.")
Traceback (most recent call last):
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 118, in wrapper
    func(*args, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle
    self._save_new_backup(database)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup
    self.write_to_storage(outputfile, filename)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
    self.storage.write_file(file, path)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "/myproject/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 51, in save
    name = self.get_available_name(name, max_length=max_length)
  File "/myproject/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 82, in get_available_name
    while self.exists(name) or (max_length and len(name) > max_length):
  File "/myproject/venv/lib/python3.8/site-packages/storages/backends/dropbox.py", line 93, in exists
    return bool(self.client.files_get_metadata(self._full_path(name)))
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/base.py", line 1682, in files_get_metadata
    r = self.request(
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/dropbox.py", line 313, in request
    res = self.request_json_string_with_retry(host,
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/dropbox.py", line 463, in request_json_string_with_retry
    return self.request_json_string(host,
  File "/myproject/venv/lib/python3.8/site-packages/dropbox/dropbox.py", line 559, in request_json_string
    r = self._session.post(url,
  File "/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen    httplib_response = self._make_request(
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/connection.py", line 343, in connect
    self.ssl_context = create_urllib3_context(
  File "/myproject/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 277, in create_urllib3_context
    context.options |= options
  File "/usr/lib/python3.8/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.8/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.8/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 460 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myproject/venv/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/myproject/myproject/core/tasks.py", line 11, in db_backup_to_dropbox
    management.call_command('dbbackup', interactive=False)
  File "/myproject/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "/myproject/venv/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/myproject/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 127, in wrapper
    connection.close()
  File "/myproject/venv/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/myproject/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 286, in close
    self.validate_thread_sharing()
  File "/myproject/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 553, in validate_thread_sharing
    raise DatabaseError(
django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140423029811616 and this is thread id 140422957233632.

I am also using django storages with dropbox as you have mentioned in the documentation.