google_storage: cannot search uploaded images

[tschoonj]

Describe the bug

Hi @vsoch,

While awaiting the deployment of our own singularity registry server (K8S is apparently not easy to get up and running...), I am moving my images to Google Storage. I created a bucket and got the credentials file and everything else AFAIK.

Now uploading an image worked fine:

$ sregistry push --name test/rhel6 /scratch/singularity/images/rhel6-dev.simg 
[client|google-storage] [database|sqlite:////dls/science/users/awf63395/singularity/sregistry.db]
[bucket][singularity-savu]
https://www.googleapis.com/download/storage/v1/b/singularity-savu/o/test%2Frhel6:latest@2fea79f7e6fb94cfd48bc486b1fa634a91a4c1fa81b2bccf68bda4f5a9ac3738.sif?generation=1564672288547665&alt=media

I can confirm via the Google Storage Console and Cyberduck that the image has indeed been uploaded successfully.

Now the problem is that sregistry search fails to find the image:

$ sregistry search
[client|google-storage] [database|sqlite:////dls/science/users/awf63395/singularity/sregistry.db]
[bucket][singularity-savu]
No containers found, based on metadata type:container
[gs://singularity-savu] Containers

I hacked the code and saw that the image has the following metadata:

{'collection': 'test', 'original': 'test/rhel6', 'image': 'rhel6', 'url': 'test/rhel6', 'tag': 'latest', 'version': '2fea79f7e6fb94cfd48bc486b1fa634a91a4c1fa81b2bccf68bda4f5a9ac3738', 'storage': 'test/rhel6:latest@2fea79f7e6fb94cfd48bc486b1fa634a91a4c1fa81b2bccf68bda4f5a9ac3738.sif', 'uri': 'test/rhel6:latest@2fea79f7e6fb94cfd48bc486b1fa634a91a4c1fa81b2bccf68bda4f5a9ac3738'}

which doesn't include the type metadata key that is used to determine whether or not it should be listed:

https://github.com/singularityhub/sregistry-cli/blob/413b9a4e6aedfdb9335aa4f922299dce8f12ab8e/sregistry/main/google_storage/query.py#L42-L46

I also copied some images from my S3 bucket via Cyberduck and these have no metadata associated with them at all, so they are also not listed :cry:

To Reproduce

Push an image to a Google Storage Bucket and try to search for it afterwards

Expected behavior

The image is found.

Version of Singularity and Singularity Registry Client

sregistry: 0.2.29

[vsoch]

This should be easy to fix ! Do you agree that we should add it here? https://github.com/singularityhub/sregistry-cli/blob/master/sregistry/main/google_storage/push.py#L43

[tschoonj]

Yes certainly.

But what do you suggest I do with the images that I would like to transfer from an AWS S3 bucket that have no metadata at all?

[vsoch]

The metadata isn't internal to the images, it's generated when you push! So the bug is just not adding it. I'm working on a fix now for you to test!

[tschoonj]

I am not using the sregistry client to migrate them from AWS S3 to Google Storage: just using Cyberduck to copy them. I could also have used the Google Storage Transfer tool, but in both no metadata at all is indeed transferred.

[vsoch]

You will need to patch it then, it should be fairly reasonable to do - I haven't tested this (but you can walk carefully though it, making sure you are using the google_storage client)

$ sregistry shell

# First just try printing them
for image in client._bucket.list_blobs():
    print(image)

# Now try changing one image from above (manually) and then do this loop
for blob in client._bucket.list_blobs():    
    if blob.exists():
        if blob.metadata is not None:
            blob.metadata['type'] = 'container'
            blob._properties['metadata']['type'] = 'container'
            blob.patch()     

And can you do a test to push with sregistry to confirm that the type:container doesn't show up? If so I'll open this PR I prepared.

[vsoch]

Oh so you are saying there is no metadata at all? You'll need to generate it again then, and update as I showed above. Technically, no metadata is required other than the type: container, but it is nice to have.

[tschoonj]

I will add the metadata using your suggested script. Many thanks!