search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.8k stars 2.42k forks source link

RFE: Add tracking/monitoring/events of what will/is currently performed by `kube play` command #15902

Open benoitf opened 2 years ago

benoitf commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Today, we have the nice podman kube play command and REST API that is accepting a yaml file.

The issue that UI client are facing, is the lack of 'progress/report' of what the command will do or is doing. For example you may wait a couple of minutes before having the command succeeding and it's only returning a status at the end but we don't have progress.

Describe the results you expected:

Several ways are possible.

From UI POV, it will be great if we know in advance the number of items that will be performed, so we can provide a better progress system.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

(paste your output here)

Output of podman info:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

praveenkumar commented 2 years ago

/label kube

openshift-ci[bot] commented 2 years ago

@praveenkumar: The label(s) /label kube cannot be applied. These labels are supported: platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, downstream-change-needed, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/valid-bug, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to [this](https://github.com/containers/podman/issues/15902#issuecomment-1274610877): >/label kube Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 2 years ago

@umohnani8 PTAL

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@umohnani8 ANy chance to look at this?

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

vyasgun commented 1 year ago

/assign

vyasgun commented 1 year ago

To address this issue, the progress can be recorded as a sequence of podman events till the resource creation is complete. The relevant events can be extracted using the filter flag (By using filters like pod=pod-name). The events can be sent to podman client by using a similar design as in podman build which is using a stream-based approach to send back progress report to the client. It should however be noted that the pod=pod-name filter will not record any events related to the containers inside it and those events can be helpful to track the details of pod creation. Introducing another filter which prints the events of constituent containers as well can be helpful for this.

vyasgun commented 1 year ago

Thanks for the code review of #18686 @vrothberg. As you suggested, let’s step back and ensure the design is right. Do you have some suggestions? I’ll reuse code from the existing PR if possible, which fetches play command events (using filter resource-type=name ) and sends them along to the writer.

vrothberg commented 1 year ago

Do you have some suggestions?

No, I am waiting on a clearer description of what the expected output is. The issue description is focusing more on potential solutions than the motivating problem.

During some of the conversations I was under the impression that the underlying issue is that the Linux-native Podman will emit certain outputs that the remote does not. For instance, when pulling an image, the remote client will print nothing. That is understandably a bad UX and something to improve on.

We should also try to anticipate future needs to make sure that whatever we build will be extensible.

If the desire is to just display (roughly) what the Linux-native Podman shows, than a simple stream will do the trick. If there is a desire to have typed information, it will be challenging as the various stacks involved have no idea in which context they are being called. For instance, when pulling an image, there is no context to know that it's being pulled by kube play. Same applies to any other object.

benoitf commented 1 year ago

Hello @vrothberg

First, a good UI/UX is when we’re able to display progress. But unlike a CLI where we can for example display like tons of log events, we try to display less information and more progress-bar like widgets.

Motivating problem: when we call the REST API for kube play command, there is nothing until it has completed

If we take as example a dummy yaml like

apiVersion: v1
kind: Pod
metadata:
  name: rss-site
  labels:
    app: web
spec:
  containers:
    - name: front-end
      image: nginx
      ports:
        - containerPort: 8080
    - name: rss-reader
      image: nickchase/rss-php-nginx:v1
      ports:
        - containerPort: 8880

and play that yaml

 curl  -X POST -H 'Content-type: text/x-yaml' --data-binary @/tmp/example.yaml  --unix-socket /Users/benoitf/.local/share/containers/podman/machine/qemu/podman.sock http://d/v4.0.0/libpod/play/kube

there is nothing during 16s

and then the result

{"Pods":[{"ID":"589b45124d9e61d12bf06a039cf84d10f78e25aec74717a8d8374754cf9f6d59","Containers":["92c5a83c0928ce7b9fccbec5df1562d3199c2c4de7d09472b6aebcecf725b8ba","f897597854b57a4cdaf8ee405a55e183bb2086ab09a8fd43c30c80db3912d3b5"],"InitContainers":null,"Logs":null,"ContainerErrors":null}],"Volumes":null,"StopReport":null,"RmReport":null,"VolumeRmReport":null,"SecretRmReport":null,"Secrets":null,"ServiceContainerID":""}

so from UI perspective, waiting without displaying anything during more than a couple seconds, user may think something is broken.

and if we look at global events:

curl -s --unix-socket /Users/benoitf/.local/share/containers/podman/machine/qemu/podman.sock http://d/v4.0.0/libpod/events 

{"status":"create","id":"0f70e3e5332c7f1d698616271e54d805edbbd46884f8d55671badd6d9f2fc46c","from":"localhost/podman-pause:4.5.1-1685123899","Type":"container","Action":"create","Actor":{"ID":"0f70e3e5332c7f1d698616271e54d805edbbd46884f8d55671badd6d9f2fc46c","Attributes":{"containerExitCode":"0","image":"localhost/podman-pause:4.5.1-1685123899","io.buildah.version":"1.30.0","name":"9d495363e88e-infra","podId":"9d495363e88e8b9f9c73e65d05df733bfe1e68a2431b5c50d37f7762f8afd4d7"}},"scope":"local","time":1688719418,"timeNano":1688719418333183288}
{"status":"create","id":"9d495363e88e8b9f9c73e65d05df733bfe1e68a2431b5c50d37f7762f8afd4d7","Type":"pod","Action":"create","Actor":{"ID":"9d495363e88e8b9f9c73e65d05df733bfe1e68a2431b5c50d37f7762f8afd4d7","Attributes":{"containerExitCode":"0","image":"","name":"rss-site","podId":""}},"scope":"local","time":1688719418,"timeNano":1688719418333908911}
{"status":"create","id":"35d1ede413d26a5d48ea42705848240bf26a75ab4e2612b974532f3a998fef27","from":"docker.io/library/nginx:latest","Type":"container","Action":"create","Actor":{"ID":"35d1ede413d26a5d48ea42705848240bf26a75ab4e2612b974532f3a998fef27","Attributes":{"app":"web","containerExitCode":"0","image":"docker.io/library/nginx:latest","maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e","name":"rss-site-front-end","podId":"9d495363e88e8b9f9c73e65d05df733bfe1e68a2431b5c50d37f7762f8afd4d7"}},"scope":"local","time":1688719419,"timeNano":1688719419557692287}
{"status":"pull","id":"2002d33a54f72d1333751d4d1b4793a60a635eac6e94a98daf0acea501580c4f","Type":"image","Action":"pull","Actor":{"ID":"2002d33a54f72d1333751d4d1b4793a60a635eac6e94a98daf0acea501580c4f","Attributes":{"containerExitCode":"0","image":"","name":"nginx","podId":""}},"scope":"local","time":1688719419,"timeNano":1688719419528734754}
{"status":"pull","id":"148f1dbceb292b38b40ae6cb7f12f096acf95d85bb3ead40e07d6b1621ad529e","Type":"image","Action":"pull","Actor":{"ID":"148f1dbceb292b38b40ae6cb7f12f096acf95d85bb3ead40e07d6b1621ad529e","Attributes":{"containerExitCode":"0","image":"","name":"nickchase/rss-php-nginx:v1","podId":""}},"scope":"local","time":1688719420,"timeNano":1688719420680347505}
{"status":"create","id":"a4c196358dd22dee9550946c9bf9253d4ecc83b868d5e7e143bedcd1937e2d68","from":"docker.io/nickchase/rss-php-nginx:v1","Type":"container","Action":"create","Actor":{"ID":"a4c196358dd22dee9550946c9bf9253d4ecc83b868d5e7e143bedcd1937e2d68","Attributes":{"app":"web","containerExitCode":"0","image":"docker.io/nickchase/rss-php-nginx:v1","io.webdevops.layout":"8","io.webdevops.version":"0.57.1","name":"rss-site-rss-reader","podId":"9d495363e88e8b9f9c73e65d05df733bfe1e68a2431b5c50d37f7762f8afd4d7","vendor":"WebDevOps.io"}},"scope":"local","time":1688719420,"timeNano":1688719420700278679}

There are events, but nothing is linking these events to the previous action. For example, at the same time user could be pulling or using podman commands.

Request number 1:

Have something when calling play kube

But for example it should not mimic what is done on the images/pull

curl  -X POST --unix-socket /Users/benoitf/.local/share/containers/podman/machine/qemu/podman.sock http://d/v4.0.0/libpod/images/pull\?reference\=docker.io/httpd
{"stream":"Trying to pull docker.io/library/httpd:latest...\n"}
{"stream":"Getting image source signatures\n"}
{"stream":"Copying blob sha256:251404d7ab9a5e4fd895b14e4d4f57ac1003f4ed6ba4be978de9a54f45074af9\n"}
{"stream":"Copying blob sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7\n"}
{"stream":"Copying blob sha256:6fa4aedca1cbcf9b275d3930f78a5c8ad7ba9aa67268bb47ab205fa564540671\n"}
{"stream":"Copying blob sha256:d191c9009da014534313b22798a7e0ae9dfbf858f2c66a5fbe8f1f672a84655d\n"}
{"stream":"Copying blob sha256:aca4832580a22c505c84613d2ca279ff744dbec75afcd1f55bce2fe2b8a5724e\n"}
{"stream":"Copying config sha256:34373ecba74c279bb751376d1018816275fce2a73a5776745d921fc260b79070\n"}
{"stream":"Writing manifest to image destination\n"}
{"stream":"Storing signatures\n"}
{"images":["34373ecba74c279bb751376d1018816275fce2a73a5776745d921fc260b79070"],"id":"34373ecba74c279bb751376d1018816275fce2a73a5776745d921fc260b79070"}

As you see, it's just streaming text so we don't know when it starts or ends

Docker API is slightly better but not perfect.

Here is how podman is interepreting:

curl  -X POST --unix-socket /Users/benoitf/.local/share/containers/podman/machine/qemu/podman.sock http://dummy/images/create\?fromImage\=httpd\&tag\=latest                                                         ✔
{"status":"Already exists","progressDetail":{},"id":"3ae0c06b4d3a"}
{"status":"Pulling fs layer","progressDetail":{},"id":"251404d7ab9a"}
{"status":"Download complete","progressDetail":{},"id":"251404d7ab9a"}
{"status":"Pulling fs layer","progressDetail":{},"id":"6fa4aedca1cb"}
{"status":"Download complete","progressDetail":{},"id":"6fa4aedca1cb"}
{"status":"Pulling fs layer","progressDetail":{},"id":"aca4832580a2"}
{"status":"Pulling fs layer","progressDetail":{},"id":"d191c9009da0"}
{"status":"Download complete","progressDetail":{},"id":"aca4832580a2"}
{"status":"Download complete","progressDetail":{},"id":"d191c9009da0"}
{"status":"Pulling fs layer","progressDetail":{},"id":"34373ecba74c"}
{"status":"Download complete","progressDetail":{},"id":"34373ecba74c"}
{"status":"Download complete","progressDetail":{},"id":"34373ecba74c"}

and how docker engine is doing:

{"status":"Pulling from library/httpd","id":"latest"}
{"status":"Pulling fs layer","progressDetail":{},"id":"3ae0c06b4d3a"}
{"status":"Pulling fs layer","progressDetail":{},"id":"6fa4aedca1cb"}
{"status":"Pulling fs layer","progressDetail":{},"id":"d191c9009da0"}
{"status":"Pulling fs layer","progressDetail":{},"id":"aca4832580a2"}
{"status":"Pulling fs layer","progressDetail":{},"id":"251404d7ab9a"}
{"status":"Waiting","progressDetail":{},"id":"aca4832580a2"}
{"status":"Waiting","progressDetail":{},"id":"251404d7ab9a"}
{"status":"Downloading","progressDetail":{"current":40982,"total":4013521},"progress":"[\u003e                                                  ]  40.98kB/4.014MB","id":"d191c9009da0"}
{"status":"Downloading","progressDetail":{"current":176,"total":176},"progress":"[==================================================\u003e]     176B/176B","id":"6fa4aedca1cb"}
{"status":"Verifying Checksum","progressDetail":{},"id":"6fa4aedca1cb"}
{"status":"Download complete","progressDetail":{},"id":"6fa4aedca1cb"}
{"status":"Downloading","progressDetail":{"current":294912,"total":29152458},"progress":"[\u003e                                                  ]  294.9kB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Downloading","progressDetail":{"current":3385692,"total":4013521},"progress":"[==========================================\u003e        ]  3.386MB/4.014MB","id":"d191c9009da0"}
{"status":"Verifying Checksum","progressDetail":{},"id":"d191c9009da0"}
{"status":"Download complete","progressDetail":{},"id":"d191c9009da0"}
{"status":"Downloading","progressDetail":{"current":5626202,"total":29152458},"progress":"[=========\u003e                                         ]  5.626MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Downloading","progressDetail":{"current":16042330,"total":29152458},"progress":"[===========================\u003e                       ]  16.04MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Downloading","progressDetail":{"current":27322714,"total":29152458},"progress":"[==============================================\u003e    ]  27.32MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Verifying Checksum","progressDetail":{},"id":"3ae0c06b4d3a"}
{"status":"Download complete","progressDetail":{},"id":"3ae0c06b4d3a"}
{"status":"Extracting","progressDetail":{"current":294912,"total":29152458},"progress":"[\u003e                                                  ]  294.9kB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Downloading","progressDetail":{"current":310576,"total":30328396},"progress":"[\u003e                                                  ]  310.6kB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":5603328,"total":29152458},"progress":"[=========\u003e                                         ]  5.603MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Downloading","progressDetail":{"current":295,"total":295},"progress":"[==================================================\u003e]     295B/295B","id":"251404d7ab9a"}
{"status":"Download complete","progressDetail":{},"id":"251404d7ab9a"}
{"status":"Downloading","progressDetail":{"current":8702299,"total":30328396},"progress":"[==============\u003e                                    ]  8.702MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":9732096,"total":29152458},"progress":"[================\u003e                                  ]  9.732MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Downloading","progressDetail":{"current":23349595,"total":30328396},"progress":"[======================================\u003e            ]  23.35MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":12386304,"total":29152458},"progress":"[=====================\u003e                             ]  12.39MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Verifying Checksum","progressDetail":{},"id":"aca4832580a2"}
{"status":"Download complete","progressDetail":{},"id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":22118400,"total":29152458},"progress":"[=====================================\u003e             ]  22.12MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Extracting","progressDetail":{"current":24182784,"total":29152458},"progress":"[=========================================\u003e         ]  24.18MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Extracting","progressDetail":{"current":27721728,"total":29152458},"progress":"[===============================================\u003e   ]  27.72MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Extracting","progressDetail":{"current":28311552,"total":29152458},"progress":"[================================================\u003e  ]  28.31MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Extracting","progressDetail":{"current":29152458,"total":29152458},"progress":"[==================================================\u003e]  29.15MB/29.15MB","id":"3ae0c06b4d3a"}
{"status":"Pull complete","progressDetail":{},"id":"3ae0c06b4d3a"}
{"status":"Extracting","progressDetail":{"current":176,"total":176},"progress":"[==================================================\u003e]     176B/176B","id":"6fa4aedca1cb"}
{"status":"Extracting","progressDetail":{"current":176,"total":176},"progress":"[==================================================\u003e]     176B/176B","id":"6fa4aedca1cb"}
{"status":"Pull complete","progressDetail":{},"id":"6fa4aedca1cb"}
{"status":"Extracting","progressDetail":{"current":65536,"total":4013521},"progress":"[\u003e                                                  ]  65.54kB/4.014MB","id":"d191c9009da0"}
{"status":"Extracting","progressDetail":{"current":4013521,"total":4013521},"progress":"[==================================================\u003e]  4.014MB/4.014MB","id":"d191c9009da0"}
{"status":"Pull complete","progressDetail":{},"id":"d191c9009da0"}
{"status":"Extracting","progressDetail":{"current":327680,"total":30328396},"progress":"[\u003e                                                  ]  327.7kB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":1310720,"total":30328396},"progress":"[==\u003e                                                ]  1.311MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":2621440,"total":30328396},"progress":"[====\u003e                                              ]  2.621MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":13762560,"total":30328396},"progress":"[======================\u003e                            ]  13.76MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":22609920,"total":30328396},"progress":"[=====================================\u003e             ]  22.61MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":25559040,"total":30328396},"progress":"[==========================================\u003e        ]  25.56MB/30.33MB","id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":30328396,"total":30328396},"progress":"[==================================================\u003e]  30.33MB/30.33MB","id":"aca4832580a2"}
{"status":"Pull complete","progressDetail":{},"id":"aca4832580a2"}
{"status":"Extracting","progressDetail":{"current":295,"total":295},"progress":"[==================================================\u003e]     295B/295B","id":"251404d7ab9a"}
{"status":"Extracting","progressDetail":{"current":295,"total":295},"progress":"[==================================================\u003e]     295B/295B","id":"251404d7ab9a"}
{"status":"Pull complete","progressDetail":{},"id":"251404d7ab9a"}
{"status":"Digest: sha256:8059bdd0058510c03ae4c808de8c4fd2c1f3c1b6d9ea75487f1e5caa5ececa02"}
{"status":"Status: Downloaded newer image for httpd:latest"}

progressDetail is implemented so it does all the difference for UI, as we know where we are ! podman by not providing the detail, makes UI cumbersome

As you see, there is an id, so you can group events by layers and display one line per layer

But for example, we should know how many layers are there at the beginning so we can display a nice UI as we know how many things need to be downloaded.

Coming back to the play kube it would be, as a first step, interesting to know something as soon as possible.

option 1:

we could have a new query parameter like ?progress=true so we can have a stream method rather than a only display result at the end

But, from UI, what I would expect is a list of items that need to be created

so Podman Desktop can display things that need to be done, and then track the progress of what will be done.

so, we receive events in the command like for docker pull

play kube command knows how many steps to do:

{"status":"Creating","progress": {"Creating resources"}, "progressDetail":{"current":1,"total":5}, "id":"id-kube"}

We know that there are 5 big steps

then each time it's doing something we have progress

{"status":"Pulling Image","progress": {"Pulling Image"},  "progressDetail":{"current":1,"total":2000},    "id":"id-image"}
{"status":"Pulling Image","progress": {"Pulling Image"},  "progressDetail":{"current":300,"total":2000},  "id":"id-image"}
{"status":"Pulling Image","progress": {"Pulling Image"},  "progressDetail":{"current":2000,"total":2000}, "id":"id-image"}

and at the end: {"result" : ...} with the existing data of what method was sending

Option 2

send an event on the events stream with data about the 'kube/play' and the number of elements that will be requested with a unique ID

and then, each individual event like 'starting a container', 'pulling an image' could contain this 'play ID'

so UI can group/attach

but it's hard to know which play kube ID has been set when calling the command (could be still an optional parameter that stream the ID first)

From Podman Desktop side, option 1 or 2 are the same.

I don't know which way you want to implement.

But basically, Podman Desktop needs the number of steps/actions as soon as possible, progress for each step So either it can display a single progress bar, or smaller chunks of progress bar, etc Displaying the raw data is usually not something important from UI at first (advanced or show details menu)

We can't wait 16s before displaying the result (16s being for a dummy yaml)

If user see a progress bar that can advance smoothly then job is done !

vrothberg commented 1 year ago

But, from UI, what I would expect is a list of items that need to be created so Podman Desktop can display things that need to be done, and then track the progress of what will be done.

I think those are two separate things. Knowing what needs to be done is the dry-run request. I am personally strongly objecting to a dry-run functionality as it's a lot of work requiring major refactoring and rewriting of the current logic; it's also a constant cost to pay since new objects need to be added to it as well and it's easy to "forget" since tests can't catch it.

If displaying progress gets us to 90% with 10% of the cost, that would be huge improvement.

[...] and then track the progress of what will be done.

That is the other part which is much easier to support.

From Podman Desktop side, option 1 or 2 are the same. I don't know which way you want to implement.

Let's stick to deciding what problem(s) to solve before going into implementation details. Just to make sure we're agreeing on what to solve.

benoitf commented 1 year ago

problem to solve is really quick as already described

To provide feedback and progress steps ( and need to provide it as soon as possible)

vrothberg commented 1 year ago

If user see a progress bar that can advance smoothly then job is done !

That is definitely doable :+1: So the goal is to opt-in to show image-pull progress.

vrothberg commented 1 year ago

@mtrmac I think we should tackle https://github.com/containers/podman/issues/12341 before addressing the issue here.

mairin commented 1 year ago

Just a note this was brought up in the Podman / Podman Desktop sync meeting today and we talked about closing the issue as don't fix (I don't think I have perms to do this.)

vrothberg commented 1 year ago

I think you mixed the issue with https://github.com/containers/podman/issues/15903 which got closed.

This issue here is still desired but we need https://github.com/containers/podman/issues/12341 done before.

mairin commented 1 year ago

Oh whoops! Sorry for the noise! 🤦🏻‍♀️