Question regarding max upload size?

[deanfourie1]

Hi, I am wanting to use this plugin for saving backups to devices.

One of my backups if 1GB in size, is it possible to upload a 1GB file. Currently I receive the error

413 Request Entity Too Large

Thanks

[Kani999]

Hi there,

Thank you for reaching out about this issue. Indeed, the ability to upload large files can be influenced by various factors, including the filesystem storage type and server configurations.

Firstly, regarding the filesystem storage backed, different types may have limitations on file sizes that can be uploaded. This can affect the maximum file size that your system can handle.

Additionally, there's an environment variable called FILE_UPLOAD_MAX_MEMORY_SIZE that can also play a role in determining the maximum size of files that can be uploaded. Adjusting this variable may help accommodate larger file uploads.

Moreover, server configurations, particularly those of your web server (NGINX or Apache), are crucial. For NGINX, you can set the client_max_body_size directive to specify the maximum allowed size of the client request body. In the case of Apache, you can adjust various PHP settings like upload_max_filesize, post_max_size, max_execution_time, and max_input_time to handle larger file uploads.

For NGINX:

server {
    ...
    client_max_body_size 64M;
}

For APACHE:

php_value upload_max_filesize 64M
php_value post_max_size 64M
php_value max_execution_time 300
php_value max_input_time 300

By adjusting these settings to accommodate larger file sizes, you should be able to resolve the "413 Request Entity Too Large" error you're encountering when attempting to upload a 1GB file.

If you have any further questions or need additional assistance, please don't hesitate to ask!

---

This response has been edited by ChatGPT.

[Kani999]

And if you are using netbox-docker there is file /netbox-docker/docker/nginx-unit.json where you can edit the config:

  "settings": {
    "http": {
      "max_body_size": <Your Value>
    }
  },

[deanfourie1]

Thanks,

Not docker, can you please tell me how to add FILE_UPLOAD_MAX_MEMORY_SIZE to the config?

I am adding it to configuration.py as

FILE_UPLOAD_MAX_MEMORY_SIZE = 'xxxxxxx'

Is this a netbox global config?

Thanks

[deanfourie1]

I have tried adding

FILE_UPLOAD_MAX_MEMORY_SIZE=26214400
DATA_UPLOAD_MAX_MEMORY_SIZE=26214400

to configuration.py but netbox fails to start.

[Kani999]

I have tried adding

FILE_UPLOAD_MAX_MEMORY_SIZE=26214400
DATA_UPLOAD_MAX_MEMORY_SIZE=26214400

to configuration.py but netbox fails to start.

What is the error message? Setting up the FILE_UPLOAD_MAX_MEMORY_SIZE in configuration.py is a proper solution.

[deanfourie1]

Ok, solved that, it was formatted incorrectly. Needed spaced between the = and the size.

FILE_UPLOAD_MAX_MEMORY_SIZE = 26214400
DATA_UPLOAD_MAX_MEMORY_SIZE = 26214400

I have also added

client_max_body_size 1000M; to /etc/nginx/nginx.conf

But still, I am getting

413 Request Entity Too Large nginx/1.18.0

Any other ideas?

[deanfourie1]

Also tried with,

client_max_body_size 0;

[deanfourie1]

My NGINX config (just the top half) and the netbox configuration.py (just the bottom)

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##
        client_max_body_size 0;
        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

# database access.) Note that the user as which NetBox runs must have read and write permissions to this path.
SESSION_FILE_PATH = None

# By default, uploaded media is stored on the local filesystem. Using Django-storages is also supported. Provide the
# class path of the storage driver in STORAGE_BACKEND and any configuration options in STORAGE_CONFIG. For example:
# STORAGE_BACKEND = 'storages.backends.s3boto3.S3Boto3Storage'
# STORAGE_CONFIG = {
#     'AWS_ACCESS_KEY_ID': 'Key ID',
#     'AWS_SECRET_ACCESS_KEY': 'Secret',
#     'AWS_STORAGE_BUCKET_NAME': 'netbox',
#     'AWS_S3_REGION_NAME': 'eu-west-1',
# }

# Time zone (default: UTC)
TIME_ZONE = 'UTC'

# Date/time formatting. See the following link for supported formats:
# https://docs.djangoproject.com/en/stable/ref/templates/builtins/#date
DATE_FORMAT = 'N j, Y'
SHORT_DATE_FORMAT = 'Y-m-d'
TIME_FORMAT = 'g:i a'
SHORT_TIME_FORMAT = 'H:i:s'
DATETIME_FORMAT = 'N j, Y g:i a'
SHORT_DATETIME_FORMAT = 'Y-m-d H:i'
FILE_UPLOAD_MAX_MEMORY_SIZE = 26214400
DATA_UPLOAD_MAX_MEMORY_SIZE = 26214400

[deanfourie1]

Resolved by also editing

nano /etc/nginx/sites-available/netbox

and setting client_max_body_size 0;

Thanks!

[Kani999]

@deanfourie1 I'm glad that you figured it out.

[Kani999]

To configure the maximum file upload size in Nginx and Apache, you need to modify specific configuration settings for each web server. Below are the steps and differences in setting up the maximum file upload size for both Nginx and Apache.

Nginx Configuration

1. Open Nginx Configuration File: The primary configuration file is usually located at /etc/nginx/nginx.conf. However, it might also be in a site-specific configuration file within /etc/nginx/sites-available/.

2. Modify the Configuration: Add or update the client_max_body_size directive. This directive sets the maximum allowed size of the client request body, specified in the http, server, or location context.

   http {
      client_max_body_size 100M;  # Set this to the desired value
   }

   Or within a specific server block:

   server {
      client_max_body_size 100M;  # Set this to the desired value
   }

3. Restart Nginx: After making the changes, you need to restart Nginx to apply them.

   sudo systemctl restart nginx

Apache Configuration

1. Open Apache Configuration File: The primary configuration file is typically located at /etc/apache2/apache2.conf or /etc/httpd/conf/httpd.conf. You might also need to edit specific site configuration files within /etc/apache2/sites-available/.

2. Modify the Configuration: Add or update the LimitRequestBody directive. This directive sets the limit on the allowed size of the client request body.

   <Directory "/var/www/html">
      LimitRequestBody 104857600  # Set this to the desired value in bytes (100M in this case)
   </Directory>

   Alternatively, you can add it within a specific <VirtualHost> block:

   <VirtualHost *:80>
      DocumentRoot "/var/www/html"
      ServerName example.com
   
      <Directory "/var/www/html">
          LimitRequestBody 104857600  # Set this to the desired value in bytes (100M in this case)
      </Directory>
   </VirtualHost>

3. Restart Apache: After making the changes, you need to restart Apache to apply them.

   sudo systemctl restart apache2

Differences in Settings

• Directive Names:

  • Nginx uses client_max_body_size to set the maximum upload size.
  • Apache uses LimitRequestBody to set the maximum upload size.

• Configuration Context:

  • In Nginx, client_max_body_size can be set in the http, server, or location context, providing flexibility on where to apply the limit.
  • In Apache, LimitRequestBody is usually set within a <Directory> or <VirtualHost> block.

• Units:

  • Nginx uses standard size units like K, M, and G (e.g., 100M for 100 megabytes).
  • Apache requires the size to be specified in bytes (e.g., 104857600 for 100 megabytes).

By following these instructions, you can configure the maximum file upload size for both Nginx and Apache web servers according to your requirements.

---

When configuring maximum file upload size for a Django application, there are a few considerations and potential limitations to keep in mind:

1. Web Server Configuration

As mentioned, you need to configure the web server (Nginx or Apache) to handle the maximum file upload size. This was detailed in the previous response.

2. Django Application Configuration

In addition to the web server settings, you need to configure Django settings to handle large file uploads. Specifically, you need to adjust the following settings in your settings.py file:

• DATA_UPLOAD_MAX_MEMORY_SIZE: This setting defines the maximum size (in bytes) that a request's body may be before a SuspiciousOperation (RequestDataTooBig) is raised. Set this to None to allow the maximum size defined by the web server.

  DATA_UPLOAD_MAX_MEMORY_SIZE = None  # No limit (follow the web server limit)

• FILE_UPLOAD_MAX_MEMORY_SIZE: This setting defines the maximum size (in bytes) that a file can be before being stored in memory. Files larger than this size will be streamed to the file system.

  FILE_UPLOAD_MAX_MEMORY_SIZE = 104857600  # 100 MB

3. Form and Model Field Validation

Ensure that any forms or model fields that handle file uploads have appropriate validation to enforce the maximum file size. You can add custom validation to your form fields.

For example:

from django import forms

def validate_file_size(file):
    max_size_mb = 100
    if file.size > max_size_mb * 1024 * 1024:
        raise forms.ValidationError(f"Max file size is {max_size_mb}MB")

class UploadFileForm(forms.Form):
    file = forms.FileField(validators=[validate_file_size])

4. File Storage Backend

Ensure that your file storage backend (e.g., local file system, cloud storage) can handle large file uploads efficiently.

Potential Limitations

1. Server Memory: Large file uploads can consume significant server memory, especially if multiple large files are being uploaded concurrently.

2. Timeouts: Long upload times for large files may lead to timeouts. Make sure your web server's timeout settings and Django's timeout settings are appropriately configured.

   For Nginx, you can adjust:

   http {
      send_timeout 300s;
      proxy_read_timeout 300s;
      proxy_connect_timeout 300s;
      client_body_timeout 300s;
   }

   For Apache, you can adjust:

   Timeout 300

3. Network Bandwidth: Ensure your network infrastructure can handle the increased bandwidth required for large file uploads.

4. Storage Quotas: Verify that your storage solution has sufficient space and appropriate quotas to handle large file uploads without running out of space.

5. Security Considerations: Large file uploads may increase the risk of denial-of-service (DoS) attacks. Implement appropriate security measures such as rate limiting, authentication, and monitoring.

By properly configuring both your web server and Django application, you can handle large file uploads efficiently while minimizing potential limitations and risks.

[deanfourie1]

Thanks, I already did this, I edited my

/etc/nginx/sites-available/netbox

server {
    listen [::]:443 ssl ipv6only=off;

    # CHANGE THIS TO YOUR SERVER'S NAME
    server_name netbox.example.com;

    ssl_certificate /etc/ssl/certs/netbox.crt;
    ssl_certificate_key /etc/ssl/private/netbox.key;

    client_max_body_size 0;

    location /static/ {
        alias /opt/netbox/netbox/static/;
    }

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    # Redirect HTTP traffic to HTTPS
    listen [::]:80 ipv6only=off;
    server_name _;
    return 301 https://$host$request_uri;
}

This used to work, but now has stopped working.

[Kani999]

Then I don't really know what else could be the problem. I cannot find any other problem that could lead to a 413 Request Entity Too Large error.

Anything I could find for fixing this error is solved there in the stack overflow thread.

• https://stackoverflow.com/questions/24306335/413-request-entity-too-large-file-upload-issue