search

minishift / minishift

Run OpenShift 3.x locally | This project does not see active developement and maintenance.
Apache License 2.0
2.48k stars 475 forks source link

Large files will cause the CentOS image to lock-up due to exhausting the live-rw filesystem #355

Closed cmoulliard closed 7 years ago

cmoulliard commented 7 years ago

Minishift : 1.0.0.Beta3 VirtualBox : VirtualBox = 5.1.14r112924

Scenatio used : https://github.com/redhat-microservices/lab_swarm-openshift/blob/master/scripts/install.sh#L6

Error reported during S2I build process

Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: received unexpected HTTP status: 500 Internal Server Error

We can't ssh to the VM

minishift ssh
E0201 12:06:49.907735   64851 ssh.go:39] Cannot establish SSH connection to the VM:  exit status 255

or get docker-env info

minishift docker-env
E0201 14:30:37.226030   69358 env.go:248] Error setting environment variables: Something went wrong running an SSH command!
command : ip addr show
err     : exit status 255
output  :
hferentschik commented 7 years ago

@cmoulliard, so I clone the repo and then run ./scripts/install.sh cluster to re-produce?

cmoulliard commented 7 years ago

@hferentschik yes and you will see the error reported during s2i build

hferentschik commented 7 years ago

@cmoulliard and this did not happen with an earlier version?

cmoulliard commented 7 years ago

Situation is even worst using Centos 1.0.0.Beta1 @hferentschik 

Downloading ISO 'https://github.com/minishift/minishift-centos-iso/releases/download/v1.0.0-beta.1/minishift-centos.iso'
 316.00 MB / 316.00 MB [==========================================================================================================================================================================================================================] 100.00% 0s
E0201 14:01:08.853522   68313 start.go:135] Error starting the VM: Error creating the VM. Error creating machine: Error running provisioning: Something went wrong running an SSH command!
command : sudo systemctl -f start docker
err     : exit status 1
output  : Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

. Retrying.
cmoulliard commented 7 years ago

A message which is perhaps not related but that I see as event when S2I build process has started is 

Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "addNDotsOption: ResolvConfPath \"/mnt/sda1/var/lib/docker/containers/d99cae62b9140f091065f455b875fd591c44d67e8b9f6d6dd0f31397dba73504/resolv.conf\" does not exist"
gbraad commented 7 years ago

Will continue investigation, but have been unable to reproduce so far. Will update issue with findings...

Note: instructions moved to https://github.com/redhat-microservices/lab_swarm-openshift/blob/master/scripts/create_vm.sh

gbraad commented 7 years ago

@cmoulliard At the moment my build fails because of not having maven installed. But the current situation looks as follows: https://gist.github.com/gbraad/bffc0035f8ce9af30011a74db448563c Build runs, image pushed, run fails due to missing dependencies (express). Note: I deployed the current master of minishift with 4G of memory and virtualbox as provider.

gbraad commented 7 years ago

Have rerun the installation at least three times now. It seems my install continues until the missing mvn command, which I deem at this point not relevant to the issue.

At this point I am unable to reproduce the issue...

gbraad commented 7 years ago

@cmoulliard can you provide more information about your environment? what is the host platform? which verson of virtualbox? anything else that might run on the host, such as kvm instances?

cmoulliard commented 7 years ago

Here are the steps to follow to reproduce the error

ISO_URL=https://github.com/minishift/minishift-centos-iso/releases/download/v1.0.0-rc.1/minishift-centos7.iso
minishift start --memory=4000 --vm-driver=virtualbox --iso-url=$ISO_URL --docker-env=[storage-driver=devicemapper]

HOST_IP=$(minishift ip)

oc login https://$HOST_IP:8443 -u system:admin
oc adm policy add-cluster-role-to-user cluster-admin admin

git clone https://github.com/redhat-microservices/lab_swarm-openshift.git
cd lab_swarm-openshift

HOST_IP=$(minishift ip)

oc login https://$HOST_IP:8443 -u admin -p admin
oc new-project snowcamp
cd cdfront
oc new-build --binary --name=cdfront -l app=cdfront
oc start-build cdfront --from-dir=. --follow
oc new-app cdfront -l app=cdfront

Useful info

oc version
oc v1.4.1+3f9807a
kubernetes v1.4.0+776c994
features: Basic-Auth

Host : MacBook Pro -> OS X El Capitan

vboxmanage --version
5.1.14r112924
dabou-macosx:~/RedHat/GP/projects/redhat-microservices/lab_swarm_openshift$ vboxmanage list vms
"minishift" {de87d055-6c79-4971-bdf1-7d1b6c33ca87}
cmoulliard commented 7 years ago

minishift log 

I0202 08:47:24.849493   11528 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") to pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939"). Volume is already mounted to pod, but remount was requested.
I0202 08:47:24.849544   11528 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") to pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939"). Volume is already mounted to pod, but remount was requested.
I0202 08:47:24.902139   11528 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
I0202 08:47:24.902443   11528 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
E0202 08:47:24.992827   11528 thin_pool_watcher.go:72] encountered error refreshing thin pool watcher: error performing thin_ls on metadata device /dev/loop6: Error running command `thin_ls --no-headers -m -o DEV,EXCLUSIVE_BYTES /dev/loop6`: exit status 127
output:

E0202 08:47:40.114245   11528 thin_pool_watcher.go:72] encountered error refreshing thin pool watcher: error performing thin_ls on metadata device /dev/loop6: Error running command `thin_ls --no-headers -m -o DEV,EXCLUSIVE_BYTES /dev/loop6`: exit status 127
output:

I0202 08:47:43.853691   11528 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/0863c2f3-e921-11e6-8e75-080027bbe939-registry-token-ekcnk" (spec.Name: "registry-token-ekcnk") to pod "0863c2f3-e921-11e6-8e75-080027bbe939" (UID: "0863c2f3-e921-11e6-8e75-080027bbe939"). Volume is already mounted to pod, but remount was requested.
I0202 08:47:43.882498   11528 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0863c2f3-e921-11e6-8e75-080027bbe939-registry-token-ekcnk" (spec.Name: "registry-token-ekcnk") pod "0863c2f3-e921-11e6-8e75-080027bbe939" (UID: "0863c2f3-e921-11e6-8e75-080027bbe939").
E0202 08:47:50.758788   11528 kubelet.go:1813] Unable to mount volumes for pod "mysql-1-deploy_snowcamp(c10ab43a-e923-11e6-8e75-080027bbe939)": timeout expired waiting for volumes to attach/mount for pod "mysql-1-deploy"/"snowcamp". list of unattached/unmounted volumes=[deployer-token-7copi]; skipping pod
E0202 08:47:50.759679   11528 pod_workers.go:184] Error syncing pod c10ab43a-e923-11e6-8e75-080027bbe939, skipping: timeout expired waiting for volumes to attach/mount for pod "mysql-1-deploy"/"snowcamp". list of unattached/unmounted volumes=[deployer-token-7copi]
E0202 08:47:55.188609   11528 thin_pool_watcher.go:72] encountered error refreshing thin pool watcher: error performing thin_ls on metadata device /dev/loop6: Error running command `thin_ls --no-headers -m -o DEV,EXCLUSIVE_BYTES /dev/loop6`: exit status 127
output:

E0202 08:48:10.260136   11528 thin_pool_watcher.go:72] encountered error refreshing thin pool watcher: error performing thin_ls on metadata device /dev/loop6: Error running command `thin_ls --no-headers -m -o DEV,EXCLUSIVE_BYTES /dev/loop6`: exit status 127
output:

I0202 08:48:24.804808   11528 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/c3032570-e923-11e6-8e75-080027bbe939-default-token-ikznw" (spec.Name: "default-token-ikznw") to pod "c3032570-e923-11e6-8e75-080027bbe939" (UID: "c3032570-e923-11e6-8e75-080027bbe939"). Volume is already mounted to pod, but remount was requested.
I0202 08:48:24.833178   11528 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/c3032570-e923-11e6-8e75-080027bbe939-default-token-ikznw" (spec.Name: "default-token-ikznw") pod "c3032570-e923-11e6-8e75-080027bbe939" (UID: "c3032570-e923-11e6-8e75-080027bbe939").
E0202 08:48:25.340713   11528 thin_pool_watcher.go:72] encountered error refreshing thin pool watcher: error performing thin_ls on metadata device /dev/loop6: Error running command `thin_ls --no-headers -m -o DEV,EXCLUSIVE_BYTES /dev/loop6`: exit status 127
output:

I0202 08:48:36.855453   11528 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/0921bd80-e921-11e6-8e75-080027bbe939-server-certificate" (spec.Name: "server-certificate") to pod "0921bd80-e921-11e6-8e75-080027bbe939" (UID: "0921bd80-e921-11e6-8e75-080027bbe939"). Volume is already mounted to pod, but remount was requested.
I0202 08:48:36.856187   11528 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/0921bd80-e921-11e6-8e75-080027bbe939-router-token-mjaeq" (spec.Name: "router-token-mjaeq") to pod "0921bd80-e921-11e6-8e75-080027bbe939" (UID: "0921bd80-e921-11e6-8e75-080027bbe939"). Volume is already mounted to pod, but remount was requested.
I0202 08:48:36.880038   11528 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0921bd80-e921-11e6-8e75-080027bbe939-server-certificate" (spec.Name: "server-certificate") pod "0921bd80-e921-11e6-8e75-080027bbe939" (UID: "0921bd80-e921-11e6-8e75-080027bbe939").
I0202 08:48:36.880059   11528 operation_executor.go:803] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0921bd80-e921-11e6-8e75-080027bbe939-router-token-mjaeq" (spec.Name: "router-token-mjaeq") pod "0921bd80-e921-11e6-8e75-080027bbe939" (UID: "0921bd80-e921-11e6-8e75-080027bbe939").
I0202 08:48:39.236625   11528 factory.go:211] Replication controller "snowcamp/" has been deleted
I0202 08:48:39.400681   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
I0202 08:48:39.401456   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
E0202 08:48:39.474072   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:39.958078342 +0000 UTC (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (volume.spec.Name: "builder-dockercfg-nzvhl-push") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: remove /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/builder-dockercfg-nzvhl-push: read-only file system
E0202 08:48:39.474488   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:39.974401522 +0000 UTC (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (volume.spec.Name: "builder-token-1dt66") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: remove /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/builder-token-1dt66: read-only file system
I0202 08:48:40.007096   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
I0202 08:48:40.007203   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
E0202 08:48:40.029172   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:41.029143436 +0000 UTC (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (volume.spec.Name: "builder-token-1dt66") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-token-1dt66.deleting~854563378: read-only file system
E0202 08:48:40.029338   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:41.029319432 +0000 UTC (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (volume.spec.Name: "builder-dockercfg-nzvhl-push") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-dockercfg-nzvhl-push.deleting~865875945: read-only file system
E0202 08:48:40.407650   11528 thin_pool_watcher.go:72] encountered error refreshing thin pool watcher: error performing thin_ls on metadata device /dev/loop6: Error running command `thin_ls --no-headers -m -o DEV,EXCLUSIVE_BYTES /dev/loop6`: exit status 127
output:

I0202 08:48:41.116660   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
I0202 08:48:41.117224   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
E0202 08:48:41.145392   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:43.145367973 +0000 UTC (durationBeforeRetry 2s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (volume.spec.Name: "builder-dockercfg-nzvhl-push") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-dockercfg-nzvhl-push.deleting~405802036: read-only file system
E0202 08:48:41.145496   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:43.145479918 +0000 UTC (durationBeforeRetry 2s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (volume.spec.Name: "builder-token-1dt66") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-token-1dt66.deleting~810236419: read-only file system
I0202 08:48:43.147759   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
I0202 08:48:43.155989   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
E0202 08:48:43.170304   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:47.170276519 +0000 UTC (durationBeforeRetry 4s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (volume.spec.Name: "builder-token-1dt66") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-token-1dt66.deleting~777624454: read-only file system
E0202 08:48:43.171255   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:47.171218602 +0000 UTC (durationBeforeRetry 4s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (volume.spec.Name: "builder-dockercfg-nzvhl-push") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-dockercfg-nzvhl-push.deleting~064203053: read-only file system
I0202 08:48:47.201632   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (spec.Name: "builder-dockercfg-nzvhl-push") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
I0202 08:48:47.202052   11528 reconciler.go:184] UnmountVolume operation started for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (spec.Name: "builder-token-1dt66") from pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939").
E0202 08:48:47.228695   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:55.228612054 +0000 UTC (durationBeforeRetry 8s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-dockercfg-nzvhl-push" (volume.spec.Name: "builder-dockercfg-nzvhl-push") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-dockercfg-nzvhl-push.deleting~598473896: read-only file system
E0202 08:48:47.230368   11528 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66\" (\"d6c1127e-e923-11e6-8e75-080027bbe939\")" failed. No retries permitted until 2017-02-02 08:48:55.230340816 +0000 UTC (durationBeforeRetry 8s). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/d6c1127e-e923-11e6-8e75-080027bbe939-builder-token-1dt66" (volume.spec.Name: "builder-token-1dt66") pod "d6c1127e-e923-11e6-8e75-080027bbe939" (UID: "d6c1127e-e923-11e6-8e75-080027bbe939") with: mkdir /var/lib/origin/openshift.local.volumes/pods/d6c1127e-e923-11e6-8e75-080027bbe939/volumes/kubernetes.io~secret/wrapped_builder-token-1dt66.deleting~061009383: read-only file system
gbraad commented 7 years ago

Able to reproduce with the following instructions: https://gist.github.com/gbraad/bffc0035f8ce9af30011a74db448563c#7th-try

praveenkumar commented 7 years ago

@gbraad @cmoulliard https://gist.github.com/gbraad/bffc0035f8ce9af30011a74db448563c#gistcomment-1986577 first try works without any issue I only used cpus 4 instead of 2 as default will try with default also.

Used steps https://github.com/minishift/minishift/issues/355#issuecomment-276889216

praveenkumar commented 7 years ago

Check the cpu usage pattern when pushing image back to registry.

img_20170202_145558 img_20170202_145601 img_20170202_145604

gbraad commented 7 years ago

@praveenkumar @cmoulliard also reran the test with 4 vcpu and now passes.

gbraad commented 7 years ago

@praveenkumar what could cause the load to be higher (more severe) for the CentOS image when compared to the boot2docker image?

hferentschik commented 7 years ago

:+1:

praveenkumar commented 7 years ago

@praveenkumar what could cause the load to be higher (more severe) for the CentOS image when compared to the boot2docker issue?

dockerd process as you can see from screenshot, will compare with b2d also but I think that is only culprit.

gbraad commented 7 years ago

@praveenkumar well, the dockerd process likely runs the registry container/process. but with the boot2docker image this wasn't an issue... does feel like something else causes the load to increase, maybe? WDYT?

gbraad commented 7 years ago

Whatever happened seriously crippled the environment. Re-running the build causes an Unexpected EOF and it shows the following issues on the console. 2017-02-02_19-37-55 after this the system is rendered useless... I will rerun the setup...

gbraad commented 7 years ago

This is what happens at the moment of the 500 Internal server error.

2017-02-02_20-08-19 2017-02-02_20-09-11

Note: in the first image you see "te exception", but ATM I have been unable to get the whole message.

gbraad commented 7 years ago

It feels a lot like an issue with the devicemapper loopback storage driver. 

device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Buffer I/O error on device dm-0
cmoulliard commented 7 years ago

This is really strange as I have a VM box created using vagrant with Centos 7 where docker is started with device storage = devicemapper without problem (https://github.com/obsidian-toaster/platform/blob/master/tooling/openshift/provision-openshift.sh#L69). This vagrant VM deploys OpenShift 1.4.1

gbraad commented 7 years ago

@cmoulliard device-mapper can be a hit-and-miss situation... atomic deploys with thinpool

I tried with overlay, but SELinux does not play well with this... 2017-02-03_16-02-45

gbraad commented 7 years ago

overlayfs, with selinux in permissive mode fails with the same 500 as reported 2017-02-03_16-26-44

gbraad commented 7 years ago

Same issue with kvm as vm-driver.

Pushed 10/11 layers, 98% complete
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: received unexpected HTTP status: 500 Internal Server Error
error: The build snowcamp/cdfront-1 status is "Failed"
gbraad commented 7 years ago

Similar issue with OpenShift set to version 1.4.0 (virtualbox)

Pushed 9/11 layers, 97% complete
unexpected EOF
gbraad commented 7 years ago

Same with OpenShift set to version 1.3.1 (virtualbox)

Pushed 9/11 layers, 94% complete
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: received unexpected HTTP status: 500 Internal Server Error
gbraad commented 7 years ago

Running the same configuration, OC 1.4.1 with --cpus 4, but using the kvm vm provider will have the same error:

Pushed 8/11 layers, 76% complete
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: received unexpected HTTP status: 500 Internal Server Error
error: The build snowcamp/cdfront-1 status is "Failed"
gbraad commented 7 years ago

Not reproducible on a CentOS install with KVM as:

virt-install \
 -n centos7 \
 --description "CentOS7 for OpenShift" \
 --os-type=Linux \
 --os-variant=rhel7 \
 --hvm \
 --ram=4096 \
 --vcpus=2 \
 --disk path=.machines/centos.img,bus=virtio,size=20 \
 --cdrom /media/storage/images/CentOS-7-x86_64-DVD-1611.iso \
 --network bridge:virbr0 \
 --accelerate \
 --vnc \
 --noautoconsole

Note: Install with default settings (let anaconda handle the partition layout, network enabled [hostname: openshift], etc)

After login:

yum update -y
yum install -y docker wget
# assign openshift to 127.0.0.1 in /etc/hosts 
# set insecure registry in /etc/sysconfig/docker
systemctl disable firewalld
systemctl stop firewalld
systemctl enable docker
systemctl start docker
wget https://github.com/openshift/origin/releases/download/v1.4.1/openshift-origin-client-tools-v1.4.1-3f9807a-linux-64bit.tar.gz
# unpack and set to path
oc cluster up
oc login https://192.168.122.192:8443 -u admin -p admin

On a remote (host) machine do

git clone https://github.com/redhat-microservices/lab_swarm-openshift.git
git lab_swarm-openshift
git checkout 18f8521
# modify scripts/install.sh and hardcode or modify oc login to point to HOST_IP=192.168.122.192
scripts/install.sh

Install succeeds... comparing with https://github.com/minishift/minishift-centos-iso/blob/master/scripts/handle-user-data what else could be an issue.

cmoulliard commented 7 years ago

So, what are the (pre)conclusions

gbraad commented 7 years ago

Ongoing, @cmoulliard At the moment we are trying to create a situation in which stock centos7 matches our live image. As mentioned above, a normal image does not fail... so after this, I am trying to introduce the layout with ln -sf as used in our prepare scripts. For now, the situation is not tackled. I did encounter issues when handling the folder /mnt/sda1/var/lib/docker/devicemapper/devicemapper/ on a minishift install...

gbraad commented 7 years ago

Formatting /dev/sda1 with mkfs.xfs results in failing even earlier (while extracting the Origin v1.4.1 container after pulling): 2017-02-06_22-11-20

gbraad commented 7 years ago

Recreated a stock CentOS7 with the same layout as the minishift installation, with the same symbolic link structure, pointing to a partition (mounted as /mnt/vdb1). Build succeeds: 2017-02-06_23-38-59

Note: unable to reproduce the issue outside of our minishift image... :-s

LalatenduMohanty commented 7 years ago

I could reproduce it the issue on OS X with Virtualbox. Saw the exact same error as @cmoulliard and @gbraad mentioned. After the error most of commands in the VM returning "Segmentation fault"

$ git clone https://github.com/redhat-microservices/lab_swarm-openshift.git
$ cd lab_swarm-openshift
$ git checkout 18f8521

$ export PATH=$PATH:~/.minishift/cache/oc/v1.4.1

$ echo $ISO_URL
file:/Users/lmohanty/Downloads/minishift-centos7.iso

$ minishift start --memory=4000 --vm-driver=virtualbox --iso-url=$ISO_URL

$ ./scripts/install.sh 
=========================================
Log on to OpenShift
=========================================
Login successful.

You don't have any projects. You can try to create a new project, by running

    oc new-project <projectname>

=========================================
Create SnowCamp namespace/project
=========================================
Now using project "snowcamp" on server "https://192.168.99.100:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
=========================================
Install MysQL server using OpenShift template
=========================================
--> Deploying template "openshift/mysql-ephemeral" to project snowcamp

     MySQL (Ephemeral)
     ---------
     MySQL database service, without persistent storage. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/mysql-container/blob/master/5.6/README.md.

WARNING: Any data stored will be lost upon pod destruction. Only use this template for testing

     The following service(s) have been created in your project: mysql.

       Username: mysql
       Password: mysql
  Database Name: catalogdb
 Connection URL: mysql://mysql:3306/

For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/mysql-container/blob/master/5.6/README.md.

     * With parameters:
        * Memory Limit=512Mi
        * Namespace=openshift
        * Database Service Name=mysql
        * MySQL Connection Username=mysql
        * MySQL Connection Password=mysql
        * MySQL Database Name=catalogdb
        * Version of MySQL Image=5.6

--> Creating resources ...
    service "mysql" created
    deploymentconfig "mysql" created
--> Success
    Run 'oc status' to view your app.
=========================================
Use S2I build to deploy the node image using our Front
=========================================
    * A Docker build using binary input will be created
      * The resulting image will be pushed to image stream "cdfront:latest"
      * A binary build was created, use 'start-build --from-dir' to trigger a new build

--> Creating resources with label app=cdfront ...
    imagestream "cdfront" created
    buildconfig "cdfront" created
--> Success
Uploading directory "." as binary input for the build ...
build "cdfront-1" started
Receiving source from STDIN as archive ...

Pulling image ryanj/centos7-s2i-nodejs:5.12.0 ...
Pulled 0/9 layers, 0% complete
Pulled 0/9 layers, 1% complete
Pulled 0/9 layers, 2% complete
Pulled 0/9 layers, 3% complete
Pulled 0/9 layers, 4% complete
Pulled 0/9 layers, 5% complete
Pulled 0/9 layers, 6% complete
Pulled 0/9 layers, 7% complete
Pulled 0/9 layers, 8% complete
Pulled 1/9 layers, 20% complete
Pulled 2/9 layers, 31% complete
Pulled 3/9 layers, 41% complete
Pulled 3/9 layers, 46% complete
Pulled 3/9 layers, 49% complete
Pulled 4/9 layers, 49% complete
Pulled 5/9 layers, 60% complete
Pulled 6/9 layers, 71% complete
Pulled 7/9 layers, 84% complete
Pulled 7/9 layers, 87% complete
Pulled 8/9 layers, 93% complete
Pulled 8/9 layers, 95% complete
Pulled 8/9 layers, 97% complete
Pulled 8/9 layers, 99% complete
Pulled 9/9 layers, 100% complete
Extracting
Step 1 : FROM ryanj/centos7-s2i-nodejs:5.12.0
 ---> d0ca9f4b4998
Step 2 : ADD . /opt/app-root/src/
 ---> dbcc5245d92e
Removing intermediate container a2f00e351871
Step 3 : EXPOSE 8080
 ---> Running in 1c72d765ed64
 ---> a841395ed80b
Removing intermediate container 1c72d765ed64
Step 4 : USER root
 ---> Running in 69af383f5ebd
 ---> 5f0d5b21247a
Removing intermediate container 69af383f5ebd
Step 5 : RUN chmod -R 777 /opt/app-root/src/
 ---> Running in 71af58a7728b
 ---> 0ce46277bda3
Removing intermediate container 71af58a7728b
Step 6 : CMD /bin/bash -c 'npm start'
 ---> Running in eec07c66bc93
 ---> 0e399da644a9
Removing intermediate container eec07c66bc93
Step 7 : ENV "OPENSHIFT_BUILD_NAME" "cdfront-1" "OPENSHIFT_BUILD_NAMESPACE" "snowcamp"
 ---> Running in 493e46b63bbd
 ---> d9a913fc7b44
Removing intermediate container 493e46b63bbd
Successfully built d9a913fc7b44

Pushing image 172.30.171.194:5000/snowcamp/cdfront:latest ...
Pushed 0/11 layers, 0% complete
Pushed 1/11 layers, 16% complete
Pushed 2/11 layers, 22% complete
Pushed 3/11 layers, 30% complete
Pushed 4/11 layers, 39% complete
Pushed 5/11 layers, 48% complete
Pushed 6/11 layers, 56% complete
Pushed 7/11 layers, 70% complete
Pushed 8/11 layers, 78% complete
Pushed 9/11 layers, 93% complete
Pushed 10/11 layers, 96% complete
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: received unexpected HTTP status: 500 Internal Server Error
error: The build snowcamp/cdfront-1 status is "Failed"

$ df -h             
Segmentation fault
gbraad commented 7 years ago

@praveenkumar @LalatenduMohanty this gist contains the docker info output: https://gist.github.com/gbraad/824779957101e8cfe6f5da41639964db

jasonbrooks commented 7 years ago

I encountered this w/ kvm, too. I've been poking at the iso trying to figure out how to enable direct lvm, as loopback lvm is pretty problematic. I proceeded by modifying the handle-user-data script: https://github.com/minishift/minishift-centos-iso/compare/master...jasonbrooks:pr-enable-direct-lvm

I changed the docker storage settings in the service file in centos-7.template, but the VM isn't picking them up. It seems that the storage driver is hard-coded in minishift? https://github.com/minishift/minishift/blob/master/pkg/minishift/provisioner/minishift_provisioner.go

gbraad commented 7 years ago

It seems the storage driver is hard-coded in minishift?

@jasonbrooks In https://github.com/minishift/minishift/blob/master/pkg/minishift/provisioner/minishift_provisioner.go#L108 the storage driver is set to "devicemapper". @praveenkumar should be able to know where to make the changes needed. At least I will run the test again with the changes made manually and see if it still produces the same errors.

Note: one of our concerns is the fact that we are not able to reproduce it on a CentOS install (instead of using our livecd). Similar setups did not cause the issue to occur.

jasonbrooks commented 7 years ago

"devicemapper" alone is not enough -- that's what gets you loopback storage. Something like --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/mini_vg-docker--pool --storage-opt dm.use_deferred_removal=true is needed. I messed around a bunch w/ docker-env to no avail...

I tried to add in the storage opts here: https://github.com/jasonbrooks/minishift/commit/4e6ae52e29128cbe2c51e279e101dd0b844c1fef, but they still weren't showing up for me.

It seems like it should be possible to get direct lvm working, and since that's what centos and rhel use, that'd be best to use here. If not, overlay does work with fedora and selinux right now, that might be worth considering...

gbraad commented 7 years ago

That is funny. That was actually the file I just wanted to point out to you.

Overlay has been a consideration for us too, and we wanted to run a test with this. As in our case we likely need to modify the policy a little. Good to hear that SELinux works right now. However, are there any other drawbacks for this option?

jasonbrooks commented 7 years ago

Overlay w/ selinux requires a newer kernel, so that's why it works with fedora but not centos right now.

gbraad commented 7 years ago

@jasonbrooks in utils.go, L70 is where those changes would be needed as the provison_detector.go handles the situation of RHEL/CentOS (for adding the registry.redhat.com, etc). Did you use RHEL for the image?

gbraad commented 7 years ago

@jasonbrooks check! That is why https://github.com/minishift/minishift/issues/355#issuecomment-277188163 occured.

Got home and image has been generated with the changes, so going to continue this. Thanks so far!

gbraad commented 7 years ago

Ran the installation again with the following changes:

[docker@minishift ~]$ docker info
Containers: 5
 Running: 5
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 1.12.5
Storage Driver: devicemapper
 Pool Name: mini_vg-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 875.6 MB
 Data Space Total: 6.208 GB
 Data Space Available: 5.332 GB
 Metadata Space Used: 307.2 kB
 Metadata Space Total: 20.97 MB
 Metadata Space Available: 20.66 MB
 Thin Pool Minimum Free Space: 620.8 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-514.6.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 1.954 GiB
Name: minishift
ID: DMBQ:PTJU:LV64:4YTO:DQRK:QVL2:AJFJ:4KID:QJZU:DYWN:4DTK:M2UC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=kvm
Insecure Registries:
 172.30.0.0/16
 127.0.0.0/8
Registries: docker.io (secure)

1st try

However, this resulted in the pull failing:

=========================================
Use S2I build to deploy the node image using our Front
=========================================
    * A Docker build using binary input will be created
      * The resulting image will be pushed to image stream "cdfront:latest"
      * A binary build was created, use 'start-build --from-dir' to trigger a new build

--> Creating resources with label app=cdfront ...
    imagestream "cdfront" created
    buildconfig "cdfront" created
--> Success
Uploading directory "." as binary input for the build ...
build "cdfront-1" started
Receiving source from STDIN as archive ...

Pulling image ryanj/centos7-s2i-nodejs:5.12.0 ...
Pulled 0/9 layers, 0% complete
Pulled 0/9 layers, 6% complete
Pulled 1/9 layers, 17% complete
Pulled 2/9 layers, 29% complete
Pulled 3/9 layers, 40% complete
error streaming logs (unexpected EOF), waiting for build to complete
The connection to the server 192.168.42.41:8443 was refused - did you specify the right host or port?

2nd try

2017-02-08_11-36-55

The result is the same as commands like ps are not found and docker ps result in Input/output error after this.

gbraad commented 7 years ago

Increased vcpus to 4 and memory to 8G:

3rd try

Pulling image ryanj/centos7-s2i-nodejs:5.12.0 ...
Pulled 0/9 layers, 0% complete
Pulled 1/9 layers, 15% complete
Pulled 2/9 layers, 27% complete
Pulled 3/9 layers, 39% complete
Pulled 4/9 layers, 47% complete
Pulled 5/9 layers, 59% complete
Pulled 6/9 layers, 71% complete
Pulled 7/9 layers, 83% complete
Pulled 8/9 layers, 90% complete
error streaming logs (unexpected EOF), waiting for build to complete
Domain minishift destroyeder 192.168.42.240:8443 was refused - did you specify the right host or port?

4th try

Pulling image ryanj/centos7-s2i-nodejs:5.12.0 ...
Pulled 0/9 layers, 0% complete
Pulled 1/9 layers, 12% complete
Pulled 2/9 layers, 23% complete
Pulled 3/9 layers, 36% complete
error streaming logs (unexpected EOF), waiting for build to complete
The connection to the server 192.168.42.6:8443 was refused - did you specify the right host or port?

This crashes even earlier than in earlier findings (repeatable, but inconsistent in the point where it happens) :-s.

gbraad commented 7 years ago

2017-02-08_12-00-44

Was able to get a little bit of garbled text from the console. I was/am able to do a ps and I noticed the same EXT4 reports as before. The only ext4 left is /dev/mapper/live-rw/ mounted to /. Might be something happening here... as I have noticed / increasing during the pull and push actions of the registry.

In the case when the pull fails, I can log into the minishift instance,

$ minishift ssh
Last login: Tue Feb  7 22:51:59 2017 from 192.168.42.1
-bash: /home/docker/.bash_profile: Input/output error
-bash-4.2$

Notice the error, as this lives on /. Also /dev/mapper/live-rw is now read-only:

/dev/mapper/live-rw on / type ext4 (ro,noatime,seclabel,data=ordered)

Before the error occurred this was:

/dev/mapper/live-rw on / type ext4 (rw,noatime,seclabel,data=ordered)
-bash-4.2$ cd /home
-bash-4.2$ ls -al
ls: reading directory .: Input/output error

however, this I/o error does not happen in /... are we writing to the user or root directory ? The only difference now with the stock image is that /, as this is also a devicemapper... during the pull, build and push in earlier tests, I have noticed the root partition growing... in intervals of 1M to 10M.

jasonbrooks commented 7 years ago

I think that non-docker storage half of the persistent disk, /dev/mini_vg/data, isn't getting mounted like it should be, so all the stuff in /var/lib/boot2docker and /var/lib/minishift is being stored in the nonpersistent root.

LalatenduMohanty commented 7 years ago

Just for the record, here are the errors from journalctl log.

Feb 07 03:00:07 minishift kernel: XFS (dm-6): Unmounting Filesystem
Feb 07 03:00:09 minishift kernel: XFS (dm-17): Unmounting Filesystem
Feb 07 03:00:09 minishift kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2883584 starting block 318142)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318142
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2887680 starting block 318143)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318143
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2891776 starting block 318144)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318144
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2895872 starting block 318145)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318145
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2899968 starting block 318146)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318146
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2904064 starting block 318147)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318147
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2908160 starting block 318148)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318148
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2912256 starting block 318149)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318149
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2916352 starting block 318150)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318150
Feb 07 03:00:09 minishift kernel: EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 131332 (offset 58720256 size 2920448 starting block 318151)
Feb 07 03:00:09 minishift kernel: Buffer I/O error on device dm-0, logical block 318151
Feb 07 03:00:09 minishift kernel: Aborting journal on device dm-0-8.
Feb 07 03:00:09 minishift kernel: Buffer I/O error on dev dm-0, logical block 10004, lost sync page write
Feb 07 03:00:09 minishift kernel: JBD2: Error -5 detected when updating journal superblock for dm-0-8.
Feb 07 03:00:09 minishift kernel: Buffer I/O error on dev dm-0, logical block 0, lost sync page write
Feb 07 03:00:09 minishift kernel: EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
Feb 07 03:00:09 minishift kernel: EXT4-fs (dm-0): Remounting filesystem read-only
Feb 07 03:00:09 minishift kernel: EXT4-fs (dm-0): previous I/O error to superblock detected
Feb 07 03:00:09 minishift kernel: Buffer I/O error on dev dm-0, logical block 0, lost sync page write

Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 2135, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 2156, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 94528, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524288, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524289, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524304, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524324, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524335, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524337, lost async page write
Feb 07 03:00:39 minishift kernel: Buffer I/O error on dev dm-0, logical block 524338, lost async page write
Feb 07 03:00:39 minishift kernel: EXT4-fs error (device dm-0): ext4_find_entry:1312: inode #131112: comm dockerregistry: reading directory lblock 0
Feb 07 03:00:39 minishift kernel: EXT4-fs (dm-0): previous I/O error to superblock detected
Feb 07 03:00:39 minishift docker[9759]: time="2017-02-07T03:00:39.674836024-05:00" level=error msg="Upload failed, retrying: received unexpected HTTP status: 500 Internal Server Error"
Feb 07 03:00:39 minishift kernel: EXT4-fs error (device dm-0): ext4_find_entry:1312: inode #131112: comm dockerregistry: reading directory lblock 0
Feb 07 03:00:39 minishift kernel: EXT4-fs (dm-0): previous I/O error to superblock detected
Feb 07 03:00:39 minishift docker[9759]: time="2017-02-07T03:00:39.681152049-05:00" level=error msg="Upload failed, retrying: received unexpected HTTP status: 500 Internal Server Error"
Feb 07 03:00:45 minishift kernel: EXT4-fs warning: 2590 callbacks suppressed
Feb 07 03:00:45 minishift kernel: EXT4-fs warning (device dm-0): __ext4_read_dirblock:902: error reading directory block (ino 131112, block 0)
Feb 07 03:00:45 minishift kernel: EXT4-fs warning (device dm-0): __ext4_read_dirblock:902: error reading directory block (ino 131351, block 0)
Feb 07 03:00:45 minishift kernel: EXT4-fs warning (device dm-0): __ext4_read_dirblock:902: error reading directory block (ino 131347, block 0)
Feb 07 03:00:47 minishift docker[9759]: time="2017-02-07T03:00:47.083944492-05:00" level=info msg="{Action=exec, LoginUID=4294967295, PID=11594}"
Feb 07 03:00:47 minishift docker[9759]: time="2017-02-07T03:00:47.085595947-05:00" level=info msg="{Action=start, LoginUID=4294967295, PID=11594}"
Feb 07 03:00:53 minishift dhclient[819]: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 4 (xid=0x4e0a5421)
Feb 07 03:00:53 minishift dhclient[819]: DHCPREQUEST on eth1 to 255.255.255.255 port 67 (xid=0x4e0a5421)
Feb 07 03:00:53 minishift dhclient[819]: DHCPOFFER from 192.168.99.6
Feb 07 03:00:53 minishift dhclient[819]: DHCPACK from 192.168.99.6 (xid=0x4e0a5421)
Feb 07 03:00:53 minishift dhclient[819]: DHCPDECLINE on eth1 to 255.255.255.255 port 67 (xid=0x4e0a5421)
Feb 07 03:00:57 minishift docker[9759]: time="2017-02-07T03:00:57.083601366-05:00" level=info msg="{Action=exec, LoginUID=4294967295, PID=11594}"
Feb 07 03:00:57 minishift docker[9759]: time="2017-02-07T03:00:57.084647215-05:00" level=info msg="{Action=start, LoginUID=4294967295, PID=11594}"
Feb 07 03:00:59 minishift kernel: EXT4-fs error (device dm-0): ext4_find_entry:1312: inode #131112: comm dockerregistry: reading directory lblock 0
Feb 07 03:00:59 minishift kernel: EXT4-fs (dm-0): previous I/O error to superblock detected
Feb 07 03:00:59 minishift kernel: buffer_io_error: 12 callbacks suppressed
Feb 07 03:00:59 minishift kernel: Buffer I/O error on dev dm-0, logical block 0, lost sync page write
Feb 07 03:00:59 minishift docker[9759]: time="2017-02-07T03:00:59.700790915-05:00" level=error msg="Upload failed: received unexpected HTTP status: 500 Internal Server Error"
Feb 07 03:00:59 minishift kernel: EXT4-fs error (device dm-0): ext4_find_entry:1312: inode #131112: comm dockerregistry: reading directory lblock 0
Feb 07 03:00:59 minishift kernel: EXT4-fs (dm-0): previous I/O error to superblock detected
Feb 07 03:00:59 minishift docker[9759]: time="2017-02-07T03:00:59.704584926-05:00" level=error msg="Upload failed: received unexpected HTTP status: 500 Internal Server Error"
Feb 07 03:00:59 minishift docker[9759]: time="2017-02-07T03:00:59.704691231-05:00" level=error msg="Attempting next endpoint for push after error: received unexpected HTTP status: 500 Internal Server Error"
Feb 07 03:00:59 minishift kernel: Buffer I/O error on dev dm-0, logical block 0, lost sync page write
LalatenduMohanty commented 7 years ago

This seems to be a similar issue with LiveCD https://bugzilla.redhat.com/show_bug.cgi?id=490245

gbraad commented 7 years ago

only difference now with the stock image is that /, as this is also a devicemapper

We might have run into https://bugzilla.redhat.com/show_bug.cgi?id=490245 Will retest with placing /var/ on the datadisk.