search

kubernetes / kubernetes

Production-Grade Container Scheduling and Management
https://kubernetes.io
Apache License 2.0
109.68k stars 39.27k forks source link

Ceph volumes ignore kube configuration if ceph-common exists on host machine #117653

Closed yaegel closed 5 months ago

yaegel commented 1 year ago

What happened?

I have a Kubernetes v1.18.17 cluster which mounts a ceph volume for most running pods. The volume definition in the yaml includes a user, a secret, and a monitor ip.

This works fine when pods are scheduled for nodes that don't have the ceph-common binaries installed on the host machine. However, when the pods get scheduled on nodes that have the ceph-common binaries installed K8s seems to ignore the volume configuration details in the yaml and instead default to looking for configuration in /etc/ceph/...

In my case that folder is empty because I just need to binaries to mount the cephfs for research storage, but not to act as monitors or other ceph control plane daemons.

What did you expect to happen?

If a volume is defined in a yaml created in K8s, I would expect the values provided in the yaml to override anything on the bare node.

How can we reproduce it (as minimally and precisely as possible)?

Try to run a pod which includes a ceph volume mount on a node which also has ceph-common installed, but no /etc/ceph/ceph.conf file.

Anything else we need to know?

No response

Kubernetes version

```console $ kubectl version Kubernetes v1.18.17 ```

Cloud provider

N/A

OS version

```console # On Linux: $ cat /etc/os-release VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal $ uname -a Linux csubertemp12 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
yaegel commented 1 year ago

/sig Storage

xing-yang commented 1 year ago

/assign @humblec

humblec commented 1 year ago

@yaegel can you provide the POD yaml you have submitted for the plugin mount ? Have you specificied secretFile configuration in the yaml? if yes, where is it pointing to ?

As a side note, please note that , the Kube version you are running in your setup is really old ie v1.18 and unsupported..

yaegel commented 1 year ago

I'm aware that 1.18 is well out of support, it's hard to keep up with the pace the k8s team keeps!

Here's the volume snippet from my yaml:

volumes:
      - name: cephfs
        cephfs:
          monitors:
          - "ipaddress:6789"
          user: admin
          secretRef:
            name: new-ceph-secret
          path: "/erams/catena-gis"
          readOnly: false

I don't specify secretFile since I don't want to use it, I want to use the secret defined in the yaml.

Thanks for your attention and help!

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 5 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/kubernetes/issues/117653#issuecomment-2016783440): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.