search

nautobot / nautobot

Network Source of Truth & Network Automation Platform
https://docs.nautobot.com
Apache License 2.0
1.03k stars 272 forks source link

Large FileVar in Jobs not possible #1661

Closed jcrystall closed 2 years ago

jcrystall commented 2 years ago

Environment

Steps to Reproduce

  1. Create a Job with a FileVar, e.g.:

    class UploadJob(Job):
source_file = FileVar(description="Test large uploads")

class Meta:
    name = "Upload Job"

def run(self, data, commit):
    return super().run(data, commit)
  2. Run Job via webinterface and upload a large file (e.g. 20mb are sufficient)

Expected Behavior

Job should run without errors. Does run successfully with small files.

Observed Behavior

Job fails with:

Environment:

Request Method: POST
Request URL: http://localhost:8000/extras/jobs/plugins-nautobot_goenet_ssot-jobs-uploadjob/run/

Django Version: 3.2.13
Python Version: 3.10.2
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_jinja',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'nautobot.core.apps.NautobotConstanceConfig',
 'nautobot.core',
 'django.contrib.admin',
 'django_celery_beat',
 'db_file_storage',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'nautobot.third_party.drf_spectacular',
 'drf_spectacular_sidecar',
 'graphene_django',
 'health_check',
 'health_check.storage',
 'django_extensions',
 'nautobot.core.apps.ConstanceDatabaseAppConfig',
 'django_ajax_tables',
 'debug_toolbar',
 'nautobot_goenet_ssot.NautobotGoenetSsotConfig']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.ExternalAuthMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']

Traceback (most recent call last):
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/serialization.py", line 39, in _reraise_errors
    yield
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/core/celery/__init__.py", line 97, in _dumps
    return json.dumps(obj, cls=NautobotKombuJSONEncoder)
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/core/celery/__init__.py", line 75, in default
    return DjangoJSONEncoder.default(self, obj)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/django/core/serializers/json.py", line 105, in default
    return super().default(o)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

During handling of the above exception (Object of type TemporaryUploadedFile is not JSON serializable), another exception occurred:
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/utilities/views.py", line 94, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/extras/views.py", line 1041, in post
    job_result = JobResult.enqueue_job(
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/extras/models/jobs.py", line 536, in enqueue_job
    func.apply_async(args=args, kwargs=kwargs, task_id=str(job_result.job_id), **celery_kwargs)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/celery/app/task.py", line 575, in apply_async
    return app.send_task(
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/celery/app/base.py", line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/celery/app/amqp.py", line 510, in send_task_message
    ret = producer.publish(
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/messaging.py", line 166, in publish
    body, content_type, content_encoding = self._prepare(
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/messaging.py", line 254, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/serialization.py", line 209, in dumps
    with _reraise_errors(EncodeError):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/serialization.py", line 43, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/exceptions.py", line 21, in reraise
    raise value.with_traceback(tb)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/serialization.py", line 39, in _reraise_errors
    yield
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/core/celery/__init__.py", line 97, in _dumps
    return json.dumps(obj, cls=NautobotKombuJSONEncoder)
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/nautobot/core/celery/__init__.py", line 75, in default
    return DjangoJSONEncoder.default(self, obj)
  File "/home/jcrystall/.cache/pypoetry/virtualenvs/nautobot-AA8H3BzY-py3.10/lib/python3.10/site-packages/django/core/serializers/json.py", line 105, in default
    return super().default(o)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

Exception Type: EncodeError at /extras/jobs/plugins-nautobot_goenet_ssot-jobs-uploadjob/run/
Exception Value: Object of type TemporaryUploadedFile is not JSON serializable
smk4664 commented 2 years ago

@jcrystall Try adding the following to your nautobot_config.py:

DATA_UPLOAD_MAX_MEMORY_SIZE = 1024 1024 20 #20M FILE_UPLOAD_MAX_MEMORY_SIZE = DATA_UPLOAD_MAX_MEMORY_SIZE

The default for Django is 2.5mb.

jcrystall commented 2 years ago

I'm aware of that setting, but I thought, maybe this is not the intended behavior, at least with the current error message.

I tried to serialize the file similar to nautobot/extras/jobs.py#L352 which crashed the server instance without any error message (maybe not enough memory available). With a TemporaryUploadedFile the file is already in the database, iiuc. Would it be possible to use this in a FileProxy or is this a bad idea somehow?

smk4664 commented 2 years ago

@jcrystall I was talking with another maintainer, The two settings I sent do resolve the issue, but you have other options.

Instead of using the FIleVar with the FileProxy, you can create a real file model that stores the location to the file, and then use the ObjectVar. OR you can create a view that takes the JSON from your file and stores it in a JSONField then having your job use that JSONField.

jcrystall commented 2 years ago

@smk4664 That might be an option, actually keeping the uploaded files wouldn't be the worst idea in my use case. Thanks!

bryanculver commented 2 years ago

@jcrystall Is it safe to close this?

jcrystall commented 2 years ago

Didn‘t get to trying it yet, but the first solution is definitely going to work.