Best effort to fix issues with library push and pull

[vsoch]

This is a best effort to fix the current issues described in #255. I'm at a stopping point because there have been changes to singularity and the library client, undocumented, so I'm afraid that we will start to see different behaviors depending on the version. This is the best effort that I can do given such a moving target.

[Aneoshun]

Alright, so I have pulled this branch, copied my config.py and secrets.py files, recompiled the SRegistry docker image (to with LDAP enabled and the other plugins commented out from the Dockerfile the docker version on Synology does not support --build-arg).

The double push issue is fixed. I can still see my images on the filesystem and on the web interface. Great.

Now, the pull issue. I have a different issue:

FATAL:   While pulling library image: could not pull image: file hash(sha256.0e0f94ed05b5a8828ea96240c13d285682e084877026c734a7e3786b58525aec) and expected hash(0e0f94ed05b5a8828ea96240c13d285682e084877026c734a7e3786b58525aec) does not match

Is this a difference between singularity 3.4.1 and 3.5?

[vsoch]

oup no I think that's a little bug - let me see if I can fix it up quickly.

[vsoch]

okay just pushed a change, and you'll need to delete your current images and then restart the uwsgi container and start the push and then pull from fresh. The issue is that the url matching pattern captures the sha256:

    url(r'^v1/images/(?P<username>.+?)/(?P<collection>.+?)/(?P<name>.+?):sha256[.](?P<version>.+?)$', views.PushImageView.as_view(), name="library_push_image"),

And we need to add it back in when we get or create a container, Singularity expects the prefix of sha256.

[Aneoshun]

Thanks! It's compiling, I will let you know.

[Aneoshun]

It works like a charm on my side. I can push and pull (multiple times) images. Thanks a lot that's really great!

However, I don't reach your issue with GET, is it because I am not using singularity 3.5?

[vsoch]

What do you mean don't reach my issue?

[vsoch]

The issue comes up when you don't specify a tag I think - it should be able to default to latest, but the Singularity client doesn't trigger the last call so the image remains as a DUMMY- and it's largely hidden from the user. But if you specify a tag, always, you should be okay :)

[Aneoshun]

In your last message here https://github.com/singularityhub/sregistry/issues/255#issuecomment-564307894 , and I thought that it systematically breaks here, but I read it too quickly sorry.

[vsoch]

Oh sorry yes I can better explain - I'm not sure this will work across versions, because I think there is different behavior by the client. I wish the Singularity API design was different (it's weird to do the tag after the image is uploaded) but the best effort here ensures it's working at least for some versions.

[vsoch]

Let me know if this looks good to you - if it's an improvement on the current we should merge. I noticed a few other issues with the modeling of the uuid field for upload_id, along with adding code formatting with black, and I'll work on those next.

[Aneoshun]

I see. Is there a simple way to see that HTTP/HTTPS requests made between the client and sylabs? Given that the requests are encrypted it might be a pain to use a sniffer, but maybe there is way to recompile singularity to display the received and sent requests?

[Aneoshun]

Hoo yes, for me that's definitely an improvement!

[vsoch]

The way I did it was just carefully walking through their code (not just sylabs/singularity but also sylabs/scs-library-client) and then adding print statements to output files to see what was sent vs. received. Go is fairly good to do this with because you have to define the data structures for the requests / responses, but that doesn't make it any less arduous!

Let me know if/when okay to merge.

[Aneoshun]

As I said above, I am totally OK to merge.

[vsoch]

okay doke, thanks for the review!