lost the images on sregistry

[tejasdixit836]

Hi ,

I seem to have lost all the containers i used to maintain through sregitsry. my admin and super user privileges are also gone(I added them back later) all i had done was changed my AD Account password and updated the same in secrets file, since i am using my user ID to bind it with registry. I did not find anything in backup folder. could you please help?

[manbat]

With the default setup, the images are bound to the host, in the same folder as sregistry is located they should be under images. I'm not clear if you are talking about Singularity Registry Server (this repository) or Singularity Registry Client (since you mentioned credentials). Changing credentials shouldn't have any influence on a Singularity Registry Server - minimally it would just be an incorrect credential and you wouldn't be able to pull or push, given using the sregistry client (which isn't required anymore).

Could you give me details about what commands you ran to reset things? There is no logical way to simply change a text file on the host and have a container be destroyed - it wouldn't just happen on it's own. It sounds like you accidentally re-created your database container, which would result in what you see - the containers not being present in the database, and the user permissions reset.

When you are developing, it's typically okay to use a postgres container for the database, but the risk here is that if you recreate the container (for any reason) you will lose the database entries. Is this what happened? It sounds like your backup wasn't running, which is a bug that I've also seen in production (for some hosts the command needs to be issued from outside of the container, the cron job doesn't seem to run inside of it). If you create the container again, I'd strongly recommend using a real database (and not a container). Minimally you should have the container binaries in the images folder, so that's a place to start.

[tejasdixit836]

i have changed the password of my Linux id , so i have updated the same in secrets file and restarted the docker compose

and i see that the contents are wiped out. I see a few containers in Images folder but not the recent ones(probably the ones before backup stopped working?) is there a way i could restore these images remaining? also could you tell me how do i run the back up manually outside the container?

is there a documentation for configuring database for sregistry?

Thank you!!

[vsoch]

i have changed the password of my Linux id , so i have updated the same in secrets file and restarted the docker compose

I don't have experience with that, but just make sure that you didn't also change your user id (or similar) in the case that the container still exists but isn't any longer associated with your username. Did you have other containers that aren't existing anymore? You have to be careful with docker and the user name space, I haven't had this happen to me, but I've heard things like that happening.

and i see that the contents are wiped out. I see a few containers in Images folder but not the recent ones(probably the ones before backup stopped working?)

The containers in your images folder aren't relevant to the backup - the folder is always bound so whatever is there reflects the last state of the registry. And if the backup didn't ever work, it would be incorrect to say it stopped working.

is there a way i could restore these images remaining?

If you don't have images in your bound folder to the host I'm not sure where else they would be.

also could you tell me how do i run the back up manually outside the container?

Definitely! I just added these details to the docs: https://singularityhub.github.io/sregistry/docs/setup/backup. As I mentioned, you want to check first running the script manually from inside the container, then test the docker exec command from the outside, and then (after adding the cron job) check a day later (or the unit of time you've chosen) to ensure the timestamp of the files is updated, and the contents are there.

is there a documentation for configuring database for sregistry?

The section where I mention it is here https://singularityhub.github.io/sregistry/docs/install/settings#database and I don't go into details because largely you just change the host name, password, etc. to your real database. Most of those fields are fairly straight forward I think, but likely if you have trouble you can Google search around for specific examples (e.g., using a Droplet, etc.)

Thank you!!

[tejasdixit836]

Thanks a lot for the documents!

so an update. one of the admin maintaining the sregistry ran docker-compose down and that was the reason that the images were not available.

i have put the cron job in place to do the backup as you mentioned

is it possible to sync up the contents of the images folder to sregistry after containers are restarted? so the next time if the containers are restarted for any reason i could sync up the images

I will also work on getting a new database.

[vsoch]

so an update. one of the admin maintaining the sregistry ran docker-compose down and that was the reason that the images were not available.

That would do it. Sorry about the misstep!

i have put the cron job in place to do the backup as you mentioned

Great!

is it possible to sync up the contents of the images folder to sregistry after containers are restarted? so the next time if the containers are restarted for any reason i could sync up the images

It wouldn't really make sense for Singularity Hub to hold it's own containers - they are docker. Instead, see the cron commands that I added to the shared docs - the ones that "commit" the containers would save the containers at some interval of your choosing.

I will also work on getting a new database.

Great! For your admin, if you need to restart containers, it's best to just restart the ones that need to be restarted:

docker-compose restart uwsgi

For example, when I'm updating certificates and need nginx on the host, I typically just stop nginx, and then start nginx on the host, and then it can easy be brought up:

docker-compose stop nginx 
sudo service nginx start
# update certificates
sudo service nginx stop
docker-compose start nginx

And then when I update code for the server, I just restart uwsgi, the worker, and nginx, sometimes just one at a time too (or all together if appropriate).

docker-compose restart uwsgi worker nginx

What you always want to do before restarting containers is to pull / update code, and before you restart, shell inside interactively, and try doing python manage.py shell and even manually doing migrations. It's hugely rare given that everything is tested, but you just need to be careful about these things. if you hit an error interactively, you can debug and fix it and ensure the container will run before you restart again. On the other hand, if you restart without testing this, a container might not start (and you'll be in a similar situation to losing it entirely).

Good luck!