Add API endpoint to create a collection

[pini-gh]

API path: /v1/collections Method: POST Parameters:

• 'entity': owner's entity id (= user.id)
• 'name': nmae for the new collection
• 'private' (optional): boolean; default to get_privacy_default()

Partial implementation for singularityhub/sregistry#344

[vsoch]

I just pushed some suggestions:

• we do a get_or_create to either get the collection (and return an error if it exists) or create it if it does not.
• I added error messages to the responses, as it looks like the sylabs json parser expects an error to be formatted as such (and likely will print it to the calling user) https://github.com/sylabs/json-resp/blob/master/json_response.go#L16

[pini-gh]

Could you please add to the documentation how to go about this? I think you would want to write up the example at the end of this Singularity section https://github.com/singularityhub/sregistry/blob/master/docs/_docs/client.md#singularity and you can also add the equivalent operations for curl there.

Well, I don't see any documentation there about the REST API. Then it seems weird to write about this endpoint only. BTW thanks to the docstrings, it is automagically documented via the /api/ path.

[vsoch]

@pini-gh that is the client documentation that shows the user how to interact with different endpoints (push, pull, etc.) We would want a section for creating a collection, and since you are using Curl it makes sense to have Singularity and curl as well.

[vsoch]

Yes that looks great!

So basically, if the Sylabs library client has a function to create a collection, we'd want to understand the context this is used from Singularity, so we can test that out with Singularity Registry server. Since the Sylabs API will identify the user with a token, and also since there is the entity id in the request, I figure there is a mechanism that first identifies the user, and then prepares the request with the user ID. In a nutshell, we want to be able to test sregistry with the singularity client, and the first thing is to figure out how and when that call is made, and see if the current implementation can support it.

I can look at this if it's unclear, but I won't be able to for a little bit. Your docs so far (with curl) look great, btw! :)

[pini-gh]

In a nutshell, we want to be able to test sregistry with the singularity client, and the first thing is to figure out how and when that call is made, and see if the current implementation can support it.

As far as I know the singularity client has no feature to create collections. I still don't understand what you want to test on that matter.

How would it be a blocker regarding this PR?

[vsoch]

I can show you. As I suspected, the client gets / creates the entity here:

https://github.com/sylabs/scs-library-client/blob/82d22e604229c73a27d339106ebcb2c6e137e03f/client/push.go#L132

Given that a collection doesn't exist, this data is then passed to the function here:

https://github.com/sylabs/scs-library-client/blob/82d22e604229c73a27d339106ebcb2c6e137e03f/client/push.go#L146

So to test this with Singularity, after following the instructions to compile it to allow http, you should be able to create a container and collection that does not exist via the standard push workflow. That is how you test it.

[pini-gh]

OK, I understand now.

Just tested a singularity push to a non existent collection, and it works now! Looking at the uwsgi container logs I see a successful POST to the new endpoint:

[pid: 21|app: 0|req: 2/3] 172.18.0.4 () {44 vars in 614 bytes} [Sat Feb 20 20:10:07 2021] GET /v1/entities/pini => generated 317 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 1)
GET GetNamedCollectionView
Not Found: /v1/collections/pini/macollection
[pid: 21|app: 0|req: 3/4] 172.18.0.4 () {44 vars in 646 bytes} [Sat Feb 20 20:10:08 2021] GET /v1/collections/pini/macollection => generated 0 bytes in 63 msecs (HTTP/1.1 404) 4 headers in 115 bytes (1 switches on core 2)
POST CollectionsView
[pid: 21|app: 0|req: 4/5] 172.18.0.4 () {46 vars in 640 bytes} [Sat Feb 20 20:10:08 2021] POST /v1/collections => generated 350 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 148 bytes (1 switches on core 3)
GET GetNamedContainerView
[pid: 21|app: 0|req: 5/6] 172.18.0.4 () {44 vars in 676 bytes} [Sat Feb 20 20:10:08 2021] GET /v1/containers/pini/macollection/scibian10-poste => generated 542 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0)
GET PushNamedContainerView
[pid: 13|app: 0|req: 1/7] 172.18.0.4 () {44 vars in 833 bytes} [Sat Feb 20 20:10:08 2021] GET /v1/images/pini/macollection/scibian10-poste:sha256.3ad816fbe662e34f799e4b17cc10a5160293e88961f158d99388a8eccbf8539a?arch=amd64 => generated 560 bytes in 3525 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0)
[pid: 21|app: 0|req: 6/8] 172.18.0.4 () {44 vars in 596 bytes} [Sat Feb 20 20:10:12 2021] GET /version => generated 58 bytes in 11 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 1)
POST RequestPushImageFileView
[pid: 17|app: 0|req: 2/9] 172.18.0.4 () {46 vars in 638 bytes} [Sat Feb 20 20:10:12 2021] POST /v2/imagefile/3 => generated 416 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 137 bytes (1 switches on core 2)

[vsoch]

Yes that’s perfect! So all you need to do is add a line in the make collection section to indicate that the create collection endpoint is used during a push in the case that the collection does not exist. I should then be able to make some time tomorrow to test this all out, approve and merge!

[pini-gh]

Adding this at the end of the "Push logic" section's bulleted list:

• If you push an image to a non existing collection, the collection will be created first, then the image will be added (version 1.1.32).

Adding this at the end of the "Create a collection" section:

In case of a singularity push to a non existing collection, the client triggers the collection creation first, using this endpoint, then pushes the image.

[vsoch]

ok! Just added a little more verbosity to the docs and I tested the curl, works like a charm! Take a look over my changes, if you approve, I'll merge first thing tomorrow.

[pini-gh]

Seems all good. Thanks!