search

kubernetes / enhancements

Enhancements tracking repo for Kubernetes
Apache License 2.0
3.44k stars 1.48k forks source link

CustomResourceDefinitions #95

Closed deads2k closed 5 years ago

deads2k commented 8 years ago

Enhancement Description

Scope of work planned for v1.15

Scope of work planned for v1.11

Scope of work planned for v1.10

Scope of work planned for v1.9

Scope of work planned for v1.8

Scope of work planned for v1.7

Other plans not in scope for this release

Plans with unclear status

Still investigating or TBD. Please comment/edit with any updates.

deads2k commented 8 years ago

@lavalamp I've created this to try to have a place where we can at least consolidate our thoughts and track progress on third party resources. I've tried to create a list of known shortcomings to be resolved before promotion to stable.

I don't have an owner in mind, but recognition of the problem seems like step 1.

adohe-zz commented 8 years ago

@deads2k I am learning third party resource recently, also wish to help with something.

deads2k commented 8 years ago

@deads2k I am learning third party resource recently, also wish to help with something.

I've re-ordered the list in terms of what I see as tactical priority. People are trying to use this now and these problems will burn them badly.

If you're comfortable taking the "multiple resources" item, that would be a great start. You could create a separate issue and we can talk about implementation in there.

adohe-zz commented 8 years ago

@deads2k I spent some time trying to reproduce the first issue:

Multiple Resources, single version, different add times - Adding resource A, waiting for it to appear, then adding resource B fails. Resource B is never added.

but with unluck. Below is my reproduce steps:

  1. create a custom thirdparty resource&wait it to appear
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/lbclaim.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancerclaim.k8s.io
description: "Allow user to claim a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/lbclaim.yaml
thirdpartyresource "loadbalancerclaim.k8s.io" created
[root@localhost kubernetes]# curl  http://localhost:8080/apis/extensions/v1beta1/thirdpartyresources/
{
  "kind": "ThirdPartyResourceList",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/",
    "resourceVersion": "170"
  },
  "items": [
    {
      "metadata": {
        "name": "loadbalancerclaim.k8s.io",
        "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/loadbalancerclaim.k8s.io",
        "uid": "dcb88b3a-9857-11e6-a19b-08002767e1f5",
        "resourceVersion": "146",
        "creationTimestamp": "2016-10-22T13:03:01Z"
      },
      "description": "Allow user to claim a loadbalancer instance",
      "versions": [
        {
          "name": "v1"
        }
      ]
    }
  ]
}
  1. after a moment(more than 10s), create another custom thirdparty resource
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/loadbalancer.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancer.k8s.io
description: "Allow user to curd a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/loadbalancer.yaml
thirdpartyresource "loadbalancer.k8s.io" created
  1. verify both resources exist
[root@localhost kubernetes]# curl http://localhost:8080/apis/k8s.io/v1/
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "k8s.io/v1",
  "resources": [
    {
      "name": "loadbalancerclaims",
      "namespaced": true,
      "kind": "Loadbalancerclaim"
    },
    {
      "name": "loadbalancers",
      "namespaced": true,
      "kind": "Loadbalancer"
    }
  ]
}
[root@localhost kubernetes]# kc get loadbalancers
No resources found.
[root@localhost kubernetes]# kc get loadbalancerclaims
No resources found.

seems we already support multiple resources, single version.

adohe-zz commented 8 years ago

And I take a deep look at TPR related code. The thirdparty_controller will do periodically sync(every 10 seconds), it will install every new TPR, and also do some deletion job. The ThirdPartyResourceServer contains all installed TPR mappings. As we can see from SyncOneResource and InstallThirdPartyResource, even this this group exists, it will still update the group with the new API.

adohe-zz commented 8 years ago

Also I found that I am able to delete a TPR schema def even there are TPR instances in the system. I think this should not be allowed.

deads2k commented 8 years ago

@deads2k I spent some time trying to reproduce the first issue:

Try to enable this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 . If it works, we're good. If it fails, something is wrong.

adohe-zz commented 8 years ago

@deads2k Hi David, please take a look at the message I sent on Slack. Besides, I add a fix to the failed integration test, the third party resource controller will remove the corresponding routes handler when a TPR get deleted, this will help with the integration test, but I am not sure whether this will bring in any other problems.

brendandburns commented 8 years ago

For problem #1, it was fixed here:

https://github.com/kubernetes/kubernetes/pull/28414

adohe-zz commented 8 years ago

@brendandburns actually not, you can run the comment out integration test, and it will fail.

adohe-zz commented 8 years ago

@brendandburns More correctly, we did support multiple resources, single version, but the deletion logical has some problem.

brendandburns commented 8 years ago

@AdoHe did you file an issue? I can take a look.

adohe-zz commented 8 years ago

@brendandburns you can see here:

https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137

enable this test, and you will see it will fail. I have tried to fix this on my local, and I will open a PR later today.

adohe-zz commented 8 years ago

@brendandburns I am afraid I don't file an issue.

nikhiljindal commented 8 years ago

Also ref https://github.com/kubernetes/kubernetes/issues/32306 (TPR should be deleted when namespace is deleted)

grodrigues3 commented 8 years ago

@deads2k can you update the checklist ?

deads2k commented 8 years ago

@deads2k can you update the checklist ?

All issues still outstanding. This is actually a feature to track the problems in the (already) beta thirdparyresources implementation from 1.3. We needed a place to keep track of our problems, but had to devote energy to other efforts in 1.5.

adohe-zz commented 8 years ago

@deads2k I am already working on Multiple Resources, single version and Multiple versions, I think a lot of code need to be update.

idvoretskyi commented 7 years ago

@deads2k does still feature still target 1.5?

adohe-zz commented 7 years ago

@idvoretskyi I am afraid not :(

sdminonne commented 7 years ago

@deads2k: ThirdPartyResources should be added to federated APIs.

rmohr commented 7 years ago

@deads2k: Currently field selectors are not working when querying for ThirdPartyObjects, is that something for your list?

adohe-zz commented 7 years ago

@deads2k @rmohr kubectl still has many outstanding capabilities against TPR, list above should be updated to track these.

deads2k commented 7 years ago

@deads2k: Currently field selectors are not working when querying for ThirdPartyObjects, is that something for your list?

That's a more general problem of inconsistent field selector support across all API types.

foxish commented 7 years ago

I'm starting to look at this as well. ThirdPartyResources are very important to supporting "external" controllers like spark, and before we can add things like sub-resources, we should be fixing this.

lavalamp commented 7 years ago

Field selectors only work on hand-curated fields in the regular API objects. I would not expect them to work for any fields in TPRs--apiserver isn't built to do arbitrary queries. If you need that behavior TPR will not work for you.

foxish commented 7 years ago

Is the next step here to move the TPRs into an addon API server? It seems like there are some outstanding PRs out to fix some of the issues in the list here which may be blocked on this item.

/cc @liggitt @deads2k @AdoHe

sttts commented 7 years ago

To get the complexity down of TPRs in the apiserver code and to make the TPR logic much more explicit, I would definitely vote for a standalone tpr-apiserver. But IMO this does not really block any of the fixes.

deads2k commented 7 years ago

I'm adding some items about handling API semantics (get, list, watch, update, patch) when dealing with multiple non-convertible Kinds. I think that probably needs a design document, since the semantics are unlikely to match normal API semantics.

brendandburns commented 7 years ago

I'll take (yet another) run at fixing some of these issues...

liggitt commented 7 years ago

https://github.com/kubernetes/kubernetes/pull/40260 and https://github.com/kubernetes/kubernetes/pull/40096 will get us in decent shape on the kubectl side

The most severe server-side issue at the moment is the garbage collector losing its mind over ownerRefs that point to TPRs.

Once we get that resolved, we should decide what the API semantics around multiple versions of a given TPR should be, and make sure the TPR type has the data we need. That's likely to affect the server-side storage impl, so I'd rather nail the design down before we do too much server-side work.

brendandburns commented 7 years ago

@liggitt I'll take a look at reviewing those. thx

jheiss commented 7 years ago

Does anyone have a pointer to how to refer to TPRs in RBAC rules? I have a TPR named like foo-bar.something.example.com. As a cluster admin I can get a list of foobars in a given namespace with kubectl get foobars.

When a regular user tries the same thing they get Error from server (Forbidden): the server does not allow access to the requested resource (get foobars.something.example.com).

I've tried every variation of foobar, foo-bar, etc. that I can think of in an RBAC rule with no luck so far.

deads2k commented 7 years ago

In the rule, you're looking for resource=foobars apigroup=something.example.com verb=get,list,watch

jheiss commented 7 years ago

@deads2k That did the trick. Thanks!

adohe-zz commented 7 years ago

@liggitt 

The most severe server-side issue at the moment is the garbage collector losing its mind over ownerRefs that point to TPRs.

anything related with the TPR cleanup issue?

liggitt commented 7 years ago

No, it was an issue with the garbage collector not knowing how to look up ownerRefs to anything other than compiled in types. The reverse issue exists as well, with the garbage collector not paying attention to finalizers on anything other than compiled-in types.

Both of those garbage collector issues are distinct from the need to clean up ThirdPartyResourceData objects reliably when the ThirdPartyResource object is removed.

adohe-zz commented 7 years ago

@liggitt Thanks for patient explanation, so what's the plan of TPR in 1.6?

lavalamp commented 7 years ago

The GC now only logs 1k times per second instead of 50k times per second, so it no longer wins the race with the log rotator. But a real fix will be coming soon, hopefully.

On Sat, Feb 4, 2017 at 11:54 PM, TonyAdo notifications@github.com wrote:

@liggitt https://github.com/liggitt Thanks for patient explanation, so what's the plan of TPR in 1.6?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/95#issuecomment-277503470, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnglmGf00K6W7SsJ1aSqWOI_M-A7Hf2ks5rZYBPgaJpZM4KLBmK .

bgrant0607 commented 7 years ago

Some of the open issues relating to TPR. Not exhaustive.

Group/version problems: https://github.com/kubernetes/kubernetes/pull/24299, https://github.com/kubernetes/kubernetes/pull/36977 Watch: https://github.com/kubernetes/kubernetes/issues/25340 Self link: https://github.com/kubernetes/kubernetes/issues/37622 Namespace deletion: https://github.com/kubernetes/kubernetes/issues/37554 GC: https://github.com/kubernetes/kubernetes/issues/39816 Finalizers: https://github.com/kubernetes/kubernetes/issues/40715 Cleanup of TPR data: https://github.com/kubernetes/kubernetes/issues/35949 Stronger validation of metadata: https://github.com/kubernetes/kubernetes/issues/22768#issuecomment-215940468 Lack of unit tests: https://github.com/kubernetes/kubernetes/pull/40956 Cleanup: https://github.com/kubernetes/kubernetes/issues/36998

Features that users think are bugs because they work for other resources: Async behavior: https://github.com/kubernetes/kubernetes/issues/29002 Integers: https://github.com/kubernetes/kubernetes/issues/30213 YAML: https://github.com/kubernetes/kubernetes/issues/37455 Decent kubectl output: https://github.com/kubernetes/kubernetes/issues/31343 Simplify resource naming: https://github.com/kubernetes/kubernetes/issues/29415 Apply: https://github.com/kubernetes/kubernetes/issues/29542, https://github.com/kubernetes/kubernetes/issues/39906 Edit: https://github.com/kubernetes/kubernetes/issues/35993

soltysh commented 7 years ago

/cc

maciaszczykm commented 7 years ago

Subscribing as we are trying to handle TPRs in Dashboard.

Tracking issues are https://github.com/kubernetes/dashboard/issues/1671 and https://github.com/kubernetes/dashboard/issues/1504.

@kubernetes/dashboard-maintainers

resouer commented 7 years ago

What's the status/plan for non-namespaced TPR? I did not find discussions about it, maybe missed something?

grpndrs commented 7 years ago

@sttts To start, I am intrigued by the development at Kubernetes. And I want to contribute to it, but Go is a new language for me. What you guys recommend me doing so that I can get this project for GSoC 2017?

To add something about me, I am fairly good at C++ and Java and I hold Bachelors in Computer Science. I have also started reading the documentation and took Udacity course involving Kubernetes.

sttts commented 7 years ago

@grpndrs we have a list of labeled issues which are a good starting point to get into the code: https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Afor-new-contributors. Feel free to contact me in slack and we can go through a few of them.

erictune commented 7 years ago

@enisoc

jamiehannaford commented 7 years ago

Is Multiple Resources, single version, different add times still an issue? I can both create and delete multiple TPRs without a problem.

Also, can we number the checkboxes in Outstanding Capabilities so it's easier to refer to? @deads2k I think you can do it like so:

1. - [ ] ...
2. - [ ] ...
frankgreco commented 7 years ago

Does anyone know how the validation component of this is coming along? I work with TPRs a lot and this feature would be priceless and save A LOT of custom code. I'd love to contribute to this feature but would like to know if anyone subscribed to this issue knows it's status

deads2k commented 7 years ago

Does anyone know how the validation component of this is coming along?

I don't expect it to happen for 1.7. At the moment, we're discussing some structural growing pains here https://github.com/kubernetes/community/pull/524 to provide a more stable base to grow upon.

deads2k commented 7 years ago

I don't expect it to happen for 1.7. At the moment, we're discussing some structural growing pains here kubernetes/community#524 to provide a more stable base to grow upon.

We plan to move forward with https://github.com/kubernetes/community/blob/master/contributors/design-proposals/thirdpartyresources.md in the 1.7 timeframe. I'll make updates here and in the sig-apimachinery calls as we move along.