Include xfsprogs in hyperkube image

[Thermi]

Hello,

It seems like some executable is missing on coreOS version 1032.0.0.

Trying to run a replicationController with pods inside, that require an EBS volume mounted just gives those errors:

                                 status code: 400, request id: 
2016-05-07 02:03:03 +0200 CEST   2016-05-07 02:04:30 +0200 CEST   7         mongodb-1-23ipk   Pod                               Warning   FailedMount        {kubelet ip-172-16-4-25.eu-west-1.compute.internal}   Unable to mount volumes for pod "mongodb-1-23ipk_default(11a231b9-13e7-11e6-8500-02a1a1cc6ab1)": executable file not found in $PATH
2016-05-07 02:03:03 +0200 CEST   2016-05-07 02:04:30 +0200 CEST   7         mongodb-1-23ipk   Pod                               Warning   FailedSync         {kubelet ip-172-16-4-25.eu-west-1.compute.internal}   Error syncing pod, skipping: executable file not found in $PATH
2016-05-07 02:08:14 +0200 CEST   2016-05-07 02:08:14 +0200 CEST   1         mongodb-1         ReplicationController             Normal    SuccessfulCreate   {replication-controller }                             Created pod: mongodb-1-rgi7c
2016-05-07 02:08:14 +0200 CEST   2016-05-07 02:08:14 +0200 CEST   1         mongodb-1-rgi7c   Pod                               Normal    Scheduled          {default-scheduler }                                  Successfully assigned mongodb-1-rgi7c to ip-172-16-4-26.eu-west-1.compute.internal
2016-05-07 02:08:19 +0200 CEST   2016-05-07 02:08:19 +0200 CEST   1         mongodb-1-rgi7c   Pod                               Warning   FailedSync         {kubelet ip-172-16-4-26.eu-west-1.compute.internal}   Error syncing pod, skipping: executable file not found in $PATH
2016-05-07 02:08:19 +0200 CEST   2016-05-07 02:08:19 +0200 CEST   1         mongodb-1-rgi7c   Pod                               Warning   FailedMount        {kubelet ip-172-16-4-26.eu-west-1.compute.internal}   Unable to mount volumes for pod "mongodb-1-rgi7c_default(cb34ddc0-13e7-11e6-8500-02a1a1cc6ab1)": executable file not found in $PATH
FIRSTSEEN                        LASTSEEN                         COUNT     NAME              KIND      SUBOBJECT   TYPE      REASON        SOURCE                                                MESSAGE
2016-05-07 02:08:19 +0200 CEST   2016-05-07 02:08:32 +0200 CEST   2         mongodb-1-rgi7c   Pod                   Warning   FailedMount   {kubelet ip-172-16-4-26.eu-west-1.compute.internal}   Unable to mount volumes for pod "mongodb-1-rgi7c_default(cb34ddc0-13e7-11e6-8500-02a1a1cc6ab1)": executable file not found in $PATH
2016-05-07 02:08:19 +0200 CEST   2016-05-07 02:08:32 +0200 CEST   2         mongodb-1-rgi7c   Pod                 Warning   FailedSync   {kubelet ip-172-16-4-26.eu-west-1.compute.internal}   Error syncing pod, skipping: executable file not found in $PATH

That is the replicationController yaml file:

apiVersion: v1
kind: ReplicationController
metadata:
  name: mongodb-1
spec:
  replicas: 1
  # selector identifies the set of Pods that this
  # replication controller is responsible for managing
  selector:
    app: mongo
  # podTemplate defines the 'cookie cutter' used for creating
  # new pods when necessary
  template:
    metadata:
      labels:
        # Important: these labels need to match the selector above
        # The api server enforces this constraint.
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo
        ports:
        - containerPort: 27017
          name: mongo-port
      volumes:
      - name: mongo-persistent-db
        awsElasticBlockStore:
          volumeID: vol-b9e70c7c
          fsType: xfs

How can this be fixed? What executable does it want? Shouldn't a bind mount be enough?

[AlmogBaku]

Ok, I solved it.

1) changed the file system type to ext4 2) mounted it to the image

apiVersion: v1
kind: ReplicationController
metadata:
  name: mongodb-test
spec:
  replicas: 1
  # selector identifies the set of Pods that this
  # replication controller is responsible for managing
  selector:
    app: mongo
  # podTemplate defines the 'cookie cutter' used for creating
  # new pods when necessary
  template:
    metadata:
      labels:
        # Important: these labels need to match the selector above
        # The api server enforces this constraint.
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo
        ports:
        - containerPort: 27017
          name: mongo-port
        volumeMounts:
            - name: mongo-persistent-db
              mountPath: /data/db
      volumes:
      - name: mongo-persistent-db
        awsElasticBlockStore:
          volumeID: <volume_id> # or aws://<region_id>/<volume_id>
          fsType: ext4

[colhom]

I don't believe there are plans to include xfsprogs in our hyperkube image, which is why I believe you're seeing the file not in $PATH message.

\cc @aaronlevy

[Thermi]

The recommended FS to run mongodb off is XFS. Mongodb is a very wide spread database, so please include xfsprogs into the hyperkube image.

[aaronlevy]

You could try bind mounting in /usr/sbin/mkfs.xfs (I'm assuming that's the binary it's expecting). But it has a few shared library dependencies I'm not sure are present in the hyperkube image.

I've changed title / labels to track this as a feature request.

[somejfn]

I would be +1 on adding XFS as well for it's dynamic inodes allocation. Use case: Prometheus instance on an iSCSI PV onprem. With ext4 you run out of inodes real quick and oversizing the volume to get more inodes is not a great workaround.

[Thermi]

Aren't xfsprogs in hyperkube now? I remember seeing it when I was still running on AWS.

[somejfn]

On my 1.7 hyperkube: mkfs.bfs mkfs.cramfs mkfs.ext2 mkfs.ext3 mkfs.ext4 mkfs.ext4dev mkfs.minix

[jbguerraz]

On hyperkube 1.10.1 (quay.io/coreos/hyperkube:v1.10.1_coreos.0) : mkfs.bfs mkfs.cramfs mkfs.ext2 mkfs.ext3 mkfs.ext4 mkfs.minix mkfs.xfs Looks like it's ok now; going to give it a try!