BurmillaOS boot fails to start console container

[pwFoo]

BurmillaOS Version: (ros os version) v1.9.5-rc1

Where are you running BurmillaOS? (docker-machine, AWS, GCE, baremetal, etc.) vmware vm

Which processor architecture you are using? x86_64

Do you use some extra hardware? (GPU, etc)?

Which console you use (default, ubuntu, centos, etc..) default

Do you use some service(s) which are not enabled by default?

Have you installed some extra tools to console?

Do you use some other customizations?

During reboot it stops with failed console start. Tried to change console before reboot, but only "default enabled" was available. Now it looks like TWO consoles try to start?!

Any hint how to restore a working console to get my personal (production) server up again?

[olljanat]

How the issue started? Did you changed something or is it new installation?

As it is running on VMware I would guess that it run out of memory. I think something like 3 or 4 GB is minimum needed with those recent VMware tools.

[pwFoo]

It' a very old Installation and I cleaned up old kernels / initrds some time ago. RAM 8GB or more should bei fine.

There was a problem with console. Missing libs, vi stopped working. I checked engine and console. Console list only shows "Default" available and active. Updated engine (should only be user-docker?!) to latest and rebootet. I hoped console container would bei recreated... Instead console stopped working.

Read also something like "all volumes" volume is missing...

Is it possible to reset console or force recreate system-docker containers?

Don't understand why overlayfs mount some directories twice?!

[olljanat]

Honestly, it sounds that it is time you to create new server and migrate to that one.

Servers (as well as containers) are supposed to be cattle, not pets:

[pwFoo]

Maybe... But services are down and No chance to migrate in the next days. It's a system service an should bei possible to force recreate or delete broken container services.

Recovery console doesn't help because system-docker isn't the persistent one.

Would need a quick fix to get current Installation up again without loose user-docker data.

[pwFoo]

Played around with reinstall (failed from recovery console because of non configured ipv6, image pull failed...) and restored system by delete the .sysimages-preload.done file. Next boot os imported the images and recreated volumes. System is up and running again...

[olljanat]

Thanks for sharing. I was not aware of that feature.

[pwFoo]

Would be a nice option to reset system-docker images / volumes?

Exact file name is /var/lib/rancher/.sysimages_v1_9_5-rc1_loaded.done. Just delete it and reboot. Also possible to mount partition in recovery console to do that.