Support for large image uploads with nginx upload module

[jpunzel]

We anticipate that after a production deployment of our registry instance, it won't be long before we have to handle very large container image uploads. We'd like to find a way to handle this before we actually open it up to a large audience.

As of right now, with a fairly stock configuration, nginx will buffer the POST body in memory, which will obviously fail on relatively small image uploads. This can be improved by tweaks to the client_body_temp_path and client_max_body_size settings, but no matter what, the upload is being buffered before being proxied to uwsgi, and a bottleneck exists.

Ideally, a large image would be uploaded directly to its final location on the filesystem. After some research, I believe the best method would be to use the nginx upload module: https://www.nginx.com/resources/wiki/modules/upload/

The documentation on how to use this with Django is sparse, but here are some links I've found that might be helpful (only the first one mentions a django solution, but the principle is the same):

https://blog.vrplumber.com/b/2013/06/07/nginx-upload-module/ https://thomas.rabaix.net/blog/2014/05/handling-file-upload-is-not-always-easy https://jiripudil.cz/blog/blazing-fast-file-upload-through-nginx

I'm aware there's also a problem with HTTPS and size limits using Requests (issue #108 ), but we can live with allowing plain HTTP as long as there's a way to handle the use case of very large images.

[vsoch]

Definitely think this is badly needed. Did you see this issue (and specifically the links in it) describing a chunked upload? That was the approach I was going to try first --> https://github.com/singularityhub/sregistry/issues/109

And I can do thix fix after the Globus PR is reviewed and merged!

[vsoch]

This is a repeat of #108, closing here and please reference the other issue.

[vsoch]

hey @jpunzel globus is merged, and I'm working on testing adding SAML right now, this guy is next in the queue!

[vsoch]

hey @jpunzel I'm reopening here because I've investigated the chunked-upload modules, and they are intended for form use (and I've created issues asking about our use case). Thus I'm going to investigate your suggestions as well since the solution I thought would be best/quick/etc. doesn't seem to be the case!

[vsoch]

hey @jpunzel what are your thoughts on handling authentication? If we just allow the user to dump any files directly to the server, checks would have to be done on the metadata after (I'm not sure this is a great idea?) I was looking at something akin to --> http://nginx.org/en/docs/http/ngx_http_auth_request_module.html although I've never used it (and am a bit new in setting up special nginx servers, although I did build the container okay for these upload modules!)

[fearful-symmetry]

Considering how many scaling issues we can run into with super-large images, there's so many gotchas here. One thing I experimented with in Go was having a streaming upload, such that the server hands over a client key, then streams the image to the server in a separate request, and the server can check the metadata in the first n bytes of the stream, and either stop the upload or continue, or whatever. Bytes from the stream are copied directly too the file, so no storing stuff in memory.

[vsoch]

hey @fearful-symmetry ! so the model that you are describing is very reasonable, going like:

client --> nginx --> server (check token) --> upload

the plugins / model that @jpunzel is suggesting actually have the upload occur on the level of nginx (before hitting the server) so any check from the uwsgi (django application) doesn't happen until after the file is uploaded and placed in a temporary location. It looks like this:

client --> nginx (upload ) --> server

The issue with the nginx upload is that I'm terrified by the idea of letting any upload happen via nginx before being checked, which is why some kind of additional nginx module would be needed. This gets complicated because the user credentials / permissions are managed with the uwsgi application (not nginx). I'm sure that this kind of interaction between nginx / django with uwsgi and stream has been done before, I'm just not sure what it looks like. Do you have any sort of ideas or possible places to start? I agree this is very important!

[fearful-symmetry]

Yah, handing the entire upload process over to nginx is definitely scary. In my Go prototype, I wasn't using nginx at all, just using a rest client to talk to the Go server. I didn't get very far, as I realized that creating a shim to go between sregistry and the outside world (which was sorta my idea) would be a lot of work, and require me to re-implement the auth flow that uwsgi has. I got the streaming idea because I've heard of other webapps doing that, in situations where they have large files, or they care about performance.

As far as the nginx thing goes, you can do auth via nginx, and either completely change the API to support that(probably not worth it), or just make some kind of special /large-upload endpoint...thing that uses nginx auth.

I'm still kinda stuck on the idea of some streaming mechanism, as it would prevent nginx (or something else along the line) from doing a malloc(entire_image_file). Which is an issue I ran into a while ago.

[vsoch]

hey @jpunzel @victorsndvg @Trophime I wanted to let you know I started on this today, and made progress! It's a big complicated but I think I can get it to work, TBA hopefully within the next week or two. I think I'll also be able to easily add a web view to upload containers, if that is also of interest!

[victorsndvg]

Great Vanessa!

Let me know when it's ready, I can be your tester ;)

I think a web view is also interesting. I expect users will use sregistry both programatically and with live interaction. It sounds great!!

A simple question (maybe nonsense), this is going to be also implemented for downloads?

[vsoch]

Let's implement this for uploads, and then test if large downloads run into issues, and if so then that's another thing we need to address. One thing at a time @victorsndvg :)

[vsoch]

Another update for today - the web interface upload is finished! You can select the container, and it will upload (chunked) to a collection:

The permissions here is based on authentication in the web interface, and if the user isn't logged in (or doesn't have permission to edit the collection) it shouldn't work. Once the upload is done, the user can see it of course:

Also notice the little "+" icon - that is how you get to the view to upload (the new "add a container here" button).

The main difference thus far is that we can't inspect the container (this is done on the command line) so the interface uploads lose the metadata exposure in the interface:

So this sort of matches a docker push (no Dockerfile).

Next I'll work on the same kind of functions, but from the command line. This will be the harder part to translate the chunked upload (in javascript) to requests multipart POST.

[victorsndvg]

@vsoch , thanks for your work!

unfortunately I cannot check your changes in two weeks ... conferences, demos, etc.

One suggestion, why not to separate container name and tag in two text boxes as with the CLI tool?

[vsoch]

@victorsndvg it will likely take me that long to finish up anyway, enjoy your conferences and demos!

The choice to not separate container and tag is to mirror command line use. If the UI teaches the user to specify with separate name and tag boxes, a command like this is less intuitive:

singularity pull shub://registry.address.org/container:tag

And so I think this is a good design decision to familiarize off the bat.

[vsoch]

hey guys I'm stuck on how to get this working with RESTFUL, so this is it for now. Sorry.

[vsoch]

just kidding, I'm done! The PR has two parts, and both are ready for testing:

• https://github.com/singularityhub/sregistry/pull/134 is the PR here that adds the large upload capability. I got the chunked upload approach working about halfway, but then wasn't happy in the command line implementation and changed back to trying the nginx upload module. This is the final solution, and it is super fast!
• https://github.com/singularityhub/sregistry-cli/pull/127 is the client that now understands the new upload endpoint. You will need to install this branch first before testing the server!

@victorsndvg @jpunzel @fearful-symmetry

[jpunzel]

@vsoch, I'm testing this right now, and this looks promising. I'm getting one error in particular though, after completing an upload via the web interface:

Traceback (most recent call last):
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "./shub/apps/api/actions/upload.py", line 109, in upload_complete
    size = size)
  File "./shub/apps/api/actions/create.py", line 107, in upload_container
    instance = move_upload_to_storage(collection, upload_id)
  File "./shub/apps/api/actions/create.py", line 39, in move_upload_to_storage
    instance = ImageUpload.objects.get(upload_id=upload_id)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
shub.apps.api.models.DoesNotExist: ImageUpload matching query does not exist.

Note that we've manually installed everything instead of using the provided Docker environment, so there might be something I'm missing (although I did run migrations before trying this version).

[vsoch]

Taking a look!

[vsoch]

Okay, so based on the error traceback and seeing calling of the function move_upload_to_storage I'm thinking that we aren't hitting the first section of this if statement:

        if os.path.exists(upload_id):
            storage = os.path.basename(names['storage'])
            new_path = move_nginx_upload_to_storage(collection, upload_id, storage)
            instance = ImageUpload.objects.create(file=new_path)
        else:
            instance = move_upload_to_storage(collection, upload_id)

and this means that the image file upload (upload_id, poorly named here) is not found to exist on the server. I suspect there is a permissions issue or similar, could you look at the nginx logs to see if the /upload endpoint spit out any errors?

And since the other option (to upload based on chunked upload) in the interface is no longer used, I'm thinking of just nixing that function all together.

[vsoch]

It's probably hard to read the updated docs in just the markdown form, but I think the issue is likely your web server (nginx?) doesn't have support for the "nginx uploads" module - basically a plugin that is compiled with nginx.--> https://www.nginx.com/resources/wiki/modules/upload/ It's amazing and fast and magical and you would only be able to use the new upload using it (one of the benefits to using the images provided, since the annoying compulation is handled for you!)

Maybe you could look at the docker file for nginx, and give compiling locally a try? Here is the Dockerfile https://github.com/singularityhub/sregistry/pull/134/files#diff-296e14ae0dc392c7edd9369908467953

[jpunzel]

I do have the upload module compiled in, and I can see the temporary files being placed in the directory I specified with upload_store. It did take some messing with permissions and getting the directory structure in place: I had to manually mkdir {0..9} in this directory to coincide with the hashing level of 1. But the error occurs after the upload completes, so I can tell the upload module is working.

So the temporary file upload location is something like /opt/shub/sregistry/images/_upload/0/0000000010 instead of the /var/www location with your supplied configuration. Perhaps there's a mismatch with the file path that the app expects?

[jpunzel]

Never mind, the problem was simply the upload_store_access directive. I had unset the all:r part, so nginx changed the directory permissions and the uwsgi user wasn't able to chdir to it.

Everything seems to work now!

[fearful-symmetry]

Posted on the wrong issue, oops. Anyways, it looks like we can't get uploads working via the sregistry CLI tool. From the uwsgi log:

[pid: 60566|app: 0|req: 88/88] 140.221.69.17 () {38 vars in 631 bytes} [Tue Jun 19 16:59:54 2018] GET /api/upload/chunked_upload => generated 850 bytes in 4 msecs (HTTP/1.1 500) 3 headers in 128 bytes (1 switches on core 0)
Internal Server Error: /api/upload/chunked_upload
Traceback (most recent call last):
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "./shub/apps/api/actions/push.py", line 49, in collection_auth_check
    body = json.loads(body_unicode)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 

The CLI client version is 0.0.87

[vsoch]

@jpunzel woohoo! Yes I too had trouble with the creation (and permissions) for the directories - I wound up shelling into the container and creating them.

No worries @fearful-symmetry ! The collection_auth_check is basically a first ping to the endpoint with the name of the collection / container, and it ensures that you have permission to interact with it. The fact that None is returned:

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 

suggests that the request body doesn't have the expected content.. Could you:

1. share the command that you are trying to do
2. Get the output of the log from the nginx container too

I think those two things could give us a clearler picture from the issue

[fearful-symmetry]

The command: sregistry push --name registry://pe_test/pi:1.0 /sbuild/craype_pi

The nginx logs don't seem to have much:

140.221.69.17 - - [20/Jun/2018:15:26:10 +0000] "GET /api/upload/chunked_upload HTTP/1.1" 500 850 "-" "python-requests/2.19.1" "-"

Nothing relevant in the nginx error log.

[vsoch]

hmm it might be related to parsing the name, can you try:

export SREGISTRY_CLIENT=registry
sregistry push --name pe_test/pi:1.0 /sbuild/craype_pi

[fearful-symmetry]

Still no dice. However, I did notice this:

 - - [20/Jun/2018:16:08:04 +0000] "POST /api/upload/chunked_upload HTTP/1.1" 301 185 "-" "python-requests/2.19.1" "10.233.15.71"
 - - [20/Jun/2018:16:08:04 +0000] "GET /api/upload/chunked_upload HTTP/1.1" 500 850 "-" "python-requests/2.19.1" "-"

Looks like there's both a GET and a POST?

[fearful-symmetry]

Okay so I found one...issue which is that the client seems to be reporting the 500 wrong: I had my .sregistry file incorrectly pointing to the http:// address. According to mitimdump this returned a 301:

<html>
<head>
  <title>301 Moved Permanently</title>
</head>
<body bgcolor="white">
  <center>
    <h1>301 Moved Permanently</h1>
  </center>
  <hr>
  <center>nginx/1.14.0</center>
</body>
</html>

However, the client seems to report that as a 500:

 sregistry push --name pe_test/pi:1.0 ./craype_pi 
WARNING Singularity is not installed, function might be limited.
WARNING Singularity is not installed, function might be limited.
[client|registry] [database|sqlite:////home/fearful/.singularity/sregistry.db]

[1. Collection return status 500 Internal Server Error]

Now that we've fixed that, we get a different error in the uwsgi log:

Internal Server Error: /api/upload/chunked_upload
Traceback (most recent call last):
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "./shub/apps/api/actions/push.py", line 69, in collection_auth_check
    raise PermissionDenied(detail="Unauthorized")
rest_framework.exceptions.PermissionDenied: Unauthorized

[fearful-symmetry]

So, I think what's happening here is that the call to requests.post in the push CLI handler is seeing the 301, then trying to do a GET to the Location field in the response. This is probably where the 500 is coming from.

As for the Unauthorized error we're getting now, I'm looking into it.

[fearful-symmetry]

Alright, that first Unauthorized error was due to collection permissions, now we're getting another Unauthorized error.

DEBUG 20180620T16Z is expired, must be 20180620T17Z.
Internal Server Error: /api/uploads/complete/
Traceback (most recent call last):
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/shub/sregistry/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "./shub/apps/api/actions/upload.py", line 91, in upload_complete
    raise PermissionDenied(detail="Unauthorized")
rest_framework.exceptions.PermissionDenied: Unauthorized

Presumably this is a different auth issue, but it's a tad hard to debug.

[fearful-symmetry]

Problem solved! Images have been uploaded via the webup and CLI client.

I'd like to open a different issue regarding the permissions, it seems to me that the collection PermissionDenied errors should return a 403 or something, not a 500

[vsoch]

Woohoo! Can you tell why the PerimssionDenied was being triggered, and then the better response to return? the PermissionDenied itself should be a 401 I think, so the 500 means server error (possibly something else!)

[fearful-symmetry]

The first one was due to not having write permissions to the container. Not sure caused the second one, i just copied my token out of the web UI and it was fixed, so presumably a key issue.

[vsoch]

ah ok, that sounds like the case.

I think the permissions are likely due to the directory creation needing to happen after build, what I can do is try to put this command in the run_uwsgi.sh script so it happens after that.

[vsoch]

okay, just added the creation of directories there! We can see if that makes a difference. Are there any other issues?

[vsoch]

thanks @fearful-symmetry and @jpunzel for your speedy testing! It sounds like this first version is satisfactory on your end, so let's wait for @victorsndvg to give it a test, and when he is also happy I will merge.

[vsoch]

@victorsndvg are you able to review this soon? We have other important features in the queue (for the client too), and we are nearing 3 weeks! (well, 17 days :) )

[victorsndvg]

@vsoch , today my GitHub says 16days from my last comment 😆 Sorry for the delay

I'm flying back home. Next week I will test, but if you think it is working and robust enough, please, proceed as you want (To merge or not to merge).

Anyway, I will test it and provide feedback.

This comments also is applicable to https://github.com/singularityhub/sregistry/pull/134#

[vsoch]

We found you!! 😊

And of course will wait for your review. Happy Friday!

[victorsndvg]

Hi @vsoch ,

I was able to do some testing. It seems that it's working great! :clap:

The only issue I found till the moment is with frozen images. If I try to overwrite a frozen image a get this error in the client side:

sregistry push --name test/test --tag test example-latest.simg 
[client|registry] [database|sqlite:////home/vsande/.singularity/sregistry.db]

[1. Collection return status 200 OK]
Expecting value: line 2 column 1 (char 1)MB - 00:00:00

An this error in the server side:

db_1      | 2018-07-04 07:55:05.261 UTC [148] ERROR:  duplicate key value violates unique constraint "main_container_name_tag_collection_id_19a25eb9_uniq"
db_1      | 2018-07-04 07:55:05.261 UTC [148] DETAIL:  Key (name, tag, collection_id)=(test, test, 1) already exists.
db_1      | 2018-07-04 07:55:05.261 UTC [148] STATEMENT:  INSERT INTO "main_container" ("add_date", "collection_id", "image_id", "metadata", "metrics", "name", "tag", "secret", "version", "frozen") VALUES ('2018-07-04T07:55:05.258686+00:00'::timestamptz, 1, 6, '{"size_mb": "92626975"}', '{}', 'test', 'test', '42c50895-4192-4490-98b6-e628d9edae3a', 'c6c536e5a32959fbfdbd49fb15e11ddf', false) RETURNING "main_container"."id"
uwsgi_1   | Internal Server Error: /api/uploads/complete/
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
uwsgi_1   |     return self.cursor.execute(sql, params)
uwsgi_1   | psycopg2.IntegrityError: duplicate key value violates unique constraint "main_container_name_tag_collection_id_19a25eb9_uniq"
uwsgi_1   | DETAIL:  Key (name, tag, collection_id)=(test, test, 1) already exists.
uwsgi_1   | 
uwsgi_1   | 
uwsgi_1   | The above exception was the direct cause of the following exception:
uwsgi_1   | 
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
uwsgi_1   |     response = get_response(request)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
uwsgi_1   |     response = self._get_response(request)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
uwsgi_1   |     response = self.process_exception_by_middleware(e, request)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
nginx_1   | 193.144.44.171 - - [04/Jul/2018:07:55:05 +0000] "POST /upload HTTP/1.1" 500 850 "-" "python-requests/2.18.4" "-"
nginx_1   | 2018/07/04 07:55:05 [error] 5#5: *87 failed to remove destination file "/var/www/images/_upload/6/0000000006" after http status 500 (2: No such file or directory) while closing request, client: 193.144.44.171, server: 0.0.0.0:80
uwsgi_1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
uwsgi_1   |     return view_func(*args, **kwargs)
uwsgi_1   |   File "./shub/apps/api/actions/upload.py", line 109, in upload_complete
uwsgi_1   |     size = size)
uwsgi_1   |   File "./shub/apps/api/actions/create.py", line 136, in upload_container
uwsgi_1   |     version=names['version'])
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
uwsgi_1   |     return getattr(self.get_queryset(), name)(*args, **kwargs)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 394, in create
uwsgi_1   |     obj.save(force_insert=True, using=self.db)
uwsgi_1   |   File "./shub/apps/main/models.py", line 343, in save
uwsgi_1   |     super(Container, self).save(*args, **kwargs)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 806, in save
uwsgi_1   |     force_update=force_update, update_fields=update_fields)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 836, in save_base
uwsgi_1   |     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 922, in _save_table
uwsgi_1   |     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 961, in _do_insert
uwsgi_1   |     using=using, raw=raw)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
uwsgi_1   |     return getattr(self.get_queryset(), name)(*args, **kwargs)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1063, in _insert
uwsgi_1   |     return query.get_compiler(using=using).execute_sql(return_id)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1099, in execute_sql
uwsgi_1   |     cursor.execute(sql, params)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
uwsgi_1   |     return self.cursor.execute(sql, params)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
uwsgi_1   |     six.reraise(dj_exc_type, dj_exc_value, traceback)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
uwsgi_1   |     raise value.with_traceback(tb)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
uwsgi_1   |     return self.cursor.execute(sql, params)
uwsgi_1   | django.db.utils.IntegrityError: duplicate key value violates unique constraint "main_container_name_tag_collection_id_19a25eb9_uniq"
uwsgi_1   | DETAIL:  Key (name, tag, collection_id)=(test, test, 1) already exists.