Convert JSON or Dict to client objects

remram44 commented 5 years ago

See also https://github.com/kubernetes-client/python/issues/340#issuecomment-526249075 (was closed for inactivity, NOT resolved) See also #63

Motivation It is sometimes useful to be able to load Python objects from JSON, for example to read it from a config file. It should be possible to get Python objects from JSON, to pass around the code in a type-safe manner, for modification and ultimately sending through the client.

Feature request There should be functions to convert JSON to objects and objects to JSON. This code already exists (it's required to talk to the API server), it should be exposed.

Workaround? Right now, you can put this raw JSON into Python objects' fields, and the API client will apparently accept this, however if you use such mixed objects your code will probably break since their attribute names are different (API uses PascalCase, Python objects use snake_case). Even if taking care to handle that, you will reach the point where you have to handle things coming from the API client (Python objects using snake_case) differently from what you load from config files (JSON using PascalCase). Really messy.

Right now I am left considering dropping this lib altogether and issuing requests directly, since the point of the wrapper is to expose safe typed Python classes instead of the raw JSON, but this doesn't work if you use any kind of config file.

Current status It is somewhat possible to convert from a Python object to JSON using client.api_client.sanitize_for_serialization(), however converting from JSON to Python is completely impossible; the deserialize() method that exists expects a requests.Response, not JSON. Calling client.api_client._ApiClient__deserialize() works, but that is a private method, and you need to pass it the kind read from the JSON yourself.

Apologies for duplicating #340 but I cannot re-open it myself.

yliaog commented 5 years ago

/assign @roycaihw

chekolyn commented 5 years ago

I was just trying to do this. This would be a useful function.

yliaog commented 5 years ago

@chekolyn could you take this issue, and work on a pr?

chekolyn commented 5 years ago

@yliaog sure, I'll give it a shot. i think the deserializer() might have been auto generated by swagger codegen. So where would this be appropriate? utils?

micw523 commented 5 years ago

Yes that looks like generated code. I think utils should be fine, but somewhere in the base repo may work as well since we don’t necessarily require e2e tests on this one (do we?)

chekolyn commented 5 years ago

I've been doing some tests and using a similar approach to utils.create_from_yaml to infer the object response_type to use in the already generated codegen deserialize function.

Using this approach is possible to deserialize single objects and a list of multiple items to a list of individual namespaced objects; however this not covered all the different deserialize options we have. For example we have:

list_xxx_for_all_namespaces
list_namespaced_xxx

My question is is there a need to deserialize to these type of objects? Or deserializing to single objects covers the use case? One complication of this is when the is a List with multiple kind of objects (something like json output ofkubectl get all -o json)

@remram44 will the proposed solution work for you? Anyone else has any opinions or suggestions ?

remram44 commented 5 years ago

I'm not sure what you mean, what kind of lists wouldn't be deserializable?

chekolyn commented 5 years ago

Is really more about the object we output and not the input. And almost every object has a list object as well.

For example:

pods = core_v1.list_pod_for_all_namespaces()
type(pods)
kubernetes.client.models.v1_pod_list.V1PodList

type(pods.items[0])
kubernetes.client.models.v1_pod.V1Pod

So in the proposed solution we would output a native python list/array that has individually deserialize objects.

Something like this:

objs = deserialize.load_from_json('/Users/user/all.ns.data.json')

for obj in objs:
  print(type(obj))
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_daemon_set.V1DaemonSet'>
<class 'kubernetes.client.models.v1_deployment.V1Deployment'>
<class 'kubernetes.client.models.v1_deployment.V1Deployment'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>

type(objs)
list

The alternative is to separate the list items by kind and return another native python list but instead of individual objects return k8s list objects that would be accesible via the each obj.items

kubernetes.client.models.v1_pod_list.V1PodList
kubernetes.client.models.v1_service_list.V1ServiceList
etc

The reason I mentioned this is that they way we would iterate over the items would be different. I'm inclined to take the first approach since it's less complicated, but was wondering if anyone needs to deserialize to these other k8s List objects.

davlum commented 5 years ago

I see there is a PR already open for this functionality. I'm not sure if the discussion is better had here or on the PR directly. There has been talk over on the apache-airflow repo of how useful the private deserialize methods would be exposed as public. It seems to me that the way the private method approaches this is the simplest, leaving it to the caller to also pass in the kind. Are there not ambiguities when trying to infer the kind? For example if I have a json { "name": "source" } this could be inferred as a V1SecretEnvSource or a V1ConfigMapEnvSource

chekolyn commented 5 years ago

@davlum if I understand correctly, you would like to pass the object type directly; perhaps we can pass it along and not infer it, basically skip this line: https://github.com/chekolyn/python/blob/json_deserialize/kubernetes/utils/deserialize.py#L252 and pass the value directly.

Additionally, we look for more than just the kind to infer an object. We look for the api version and api group.

Do you have any json/yaml examples I can test with?

davlum commented 5 years ago

Yeah, I'm essentially looking for the functionality of __deserialize_model(self, data, klass): but made public. Because you have to pass the model and the models include the API version number, the API version is explicit as well.

__deserialize_model({ "name": "source" }, V1ConfigMapEnvSource)

chekolyn commented 5 years ago

@davlum i think a klaas/response_type str parameter in here https://github.com/chekolyn/python/blob/json_deserialize/kubernetes/utils/deserialize.py#L238 would do pass the klass to the __deserialize_model via this line: https://github.com/kubernetes-client/python/blob/ca4f31198e9748f1075af0e02abb8f7e1e174159/kubernetes/client/api_client.py#L289

I will amend/add this tonight. This is still work in progress and I thank you for the suggestion.

remram44 commented 5 years ago

Passing in the kind is acceptable for my use case, I always know what kind of object I am trying to deserialize.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

remram44 commented 4 years ago

/remove-lifecycle stale

davlum commented 4 years ago

Any mouvement on this?

marcellodesales commented 4 years ago

Alright, I saw that https://github.com/kubernetes-client/python/issues/574#issuecomment-405400414 works... It's awkward, but works...

marcellodesales commented 4 years ago

@davlum try the method at https://github.com/kubernetes-client/python/issues/574#issuecomment-405400414 works... I needed to clone objects and change the names and properties. So I had it transformed into a dict, worked on that lavel, and then converted to yaml... That's how far I got...

https://stackoverflow.com/questions/59977058/clone-kuberenetes-objects-programmatically-python/59977059#59977059

davlum commented 4 years ago

That's not quite what I'm looking for. I would like to go from JSON/YAML to client objects defined within this library. If I'm not mistaken your example returns JSON from the client, modifies it and writes back YAML, never making the conversion from YAML -> client object.

roycaihw commented 4 years ago

the deserialize() method that exists expects a requests.Response, not JSON

Reading deserialize, it mostly needs the data from a RESTResponse. The method could be reused for JSON by mocking a RESTResponse object, like what @chekolyn did in https://github.com/kubernetes-client/python/pull/989/files#diff-0204f9aa3f2d80c3fe32d62f37882dfaR256-R263

piroszhog commented 4 years ago

That's how I have to do it now:

class FakeKubeResponse:
    def __init__(self, obj):
        import json
        self.data = json.dumps(obj)

fake_kube_response = FakeKubeResponse(initial_object_json)
v1pod = api_client.deserialize(fake_kube_response, 'V1Pod')

Will be glad to remove this hack after this issue close.

roycaihw commented 4 years ago

@piroszhog I think that is the right workaround at the moment, and this issue should be closed once #989 is finished.

postmaxin commented 4 years ago

That's how I have to do it now:

class FakeKubeResponse:
    def __init__(self, obj):
        import json
        self.data = json.dumps(obj)

fake_kube_response = FakeKubeResponse(initial_object_json)
v1pod = api_client.deserialize(fake_kube_response, 'V1Pod')

Will be glad to remove this hack after this issue close.

To make this hack work for me, I had to add explicit datetime deserialization. I'm not sure but I think this may be because I'm also doing funky stuff like deepcopies in my code;

[...]
            def _time_dumps(val):
                if isinstance(val, datetime.datetime):
                    return iso8601time(val)
                else:
                    raise TypeError(
                        "Don't know how to dump {}".format(type(val)))
            self.data = json.dumps(body, default=_time_dumps)
[...]

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

remram44 commented 4 years ago

/remove-lifecycle stale

cmoulliard commented 4 years ago

Will this issue be implemented to easily convert Json to K8s objects ?

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

klarose commented 4 years ago

Any more interest in this?

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

remram44 commented 4 years ago

I don't have enough interest to fight the bot ad-vitam. Feel free to close it like you did #340 and pretend it's not a problem. It's not like you own anyone anything, even civility.

Bye!

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-client/python/issues/977#issuecomment-748314197): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

aantn commented 2 years ago

If anyone is still struggling with this, you can do it with Hikaru. I'm not affiliated with the project in any way, but I've used it in my own open source projects to convert YAML/Json to Python objects and back.

pigletfly commented 1 year ago

the origin PR https://github.com/kubernetes-client/python/pull/989 is closed, so can we reopen this issue?

pigletfly commented 1 year ago

/reopen

k8s-ci-robot commented 1 year ago

@pigletfly: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-client/python/issues/977#issuecomment-1516092720): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

pigletfly commented 1 year ago

@roycaihw

Don-Burns commented 1 year ago

👍 if this could be reopened. Would be very useful

remram44 commented 1 year ago

Open another duplicate, it's what I did. If they'd rather create all this noise and duplication by using stupid bots, we must play along.

yliaog commented 1 year ago

/reopen

k8s-ci-robot commented 1 year ago

@yliaog: Reopened this issue.

In response to [this](https://github.com/kubernetes-client/python/issues/977#issuecomment-1706910802): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

msw-kialo commented 1 year ago

/remove-lifecycle rotten

k8s-triage-robot commented 9 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

postmaxin commented 8 months ago

/remove-lifecycle rotten

shouhanzen commented 8 months ago

Hi, I'm interested in contributing to this issue. How can I get started?

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

remram44 commented 4 months ago

The project might be rotten but the issue is fresh

/remove-lifecycle stale

remram44 commented 4 months ago

Sorry if I didn't use the right incantation to scare away the bots

/remove-lifecycle rotten

kubernetes-client / python

Convert JSON or Dict to client objects #977