promptops / cli

um is a GPT-powered CLI assistant. Ask questions in plain English, get the perfect shell command.
https://promptops.com
GNU General Public License v3.0
122 stars 3 forks source link

Attempts to fix seem to ignore stderr output #6

Closed wgordon17 closed 1 year ago

wgordon17 commented 1 year ago

Testing out some more complex queries (after getting #4 resolves, thanks! 🎉) and I was very impressed at ums ability to query my shell history and find some really cool stuff! 🙌

Purposefully just trying to see what it would do in a failure scenario, um doesn't seem to account for (or interpret) the stderr output and instead just sees the response a generic failure (return code was not 0).

Here's the example that I was running

um: how do I query for pod restarts in all k8s namespaces

   📖 oc get pods -o custom-columns=namespace:.metadata.namespace,restarts:.status.containerStatuses[*].restartCount
 ➜︎ 📖 oc get pods -o custom-columns=CREATED:.metadata.creationTimestamp,STATUS:.status.phase,RESTARTS:.status.containerStatuses[\*].restartCount,ownerKind:.metadata.ownerReferences[\*].kind  --all-namespaces
   📖 oc get pods -o custom-columns=namespace:.metadata.namespace,restarts:.status.containerStatuses[\*].restartCount
   ✨ kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount
   💭️ don't see what you're looking for? try providing more context

loaded from shell history

> oc get pods -o custom-columns=CREATED:.metadata.creationTimestamp,STATUS:.status.phase,RESTARTS:.status.containerStatuses[\*].restartCount,ownerKind:.metadata.
ownerReferences[\*].kind  --all-namespaces
error: You must be logged in to the server (Unauthorized)
> looks like the command failed (return code was not 0), would you like us to attempt to fix it?
 ➜︎ ✨ kubectl get pods -o custom-columns=CREATED:.metadata.creationTimestamp,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,ownerKind:.metadata.ownerReferences[*].kind --all-namespaces

error: You must be logged in to the server (Unauthorized)
> looks like the command failed (return code was not 0), would you like us to attempt to fix it?
 ➜︎ ✨ kubectl get pods -o custom-columns=CREATED:.metadata.creationTimestamp,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,ownerKind:.metadata.ownerReferences[*].kind --all-namespaces

So um saw the failure, but instead of interpretting the error response (error: You must be logged in to the server (Unauthorized)), it just kept trying different iterations of the same command.

joshualwhite commented 1 year ago

Right now our model has strict instructions to only update the existing command based on the output. We will look at tweaking it to allow it to suggest commands that require a pre-condition, such as authentication.

joshualwhite commented 1 year ago

I am allowing the model to create a "precondition" script now, if you try this again the output is something along the lines of:

`kubectl config use-context <context-name> && kubectl get pods --all-namespaces -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, restarts: .status.containerStatuses[].restartCount}' ` We will continue to iterate on this, including pulling in your history to fix broken commands, and allowing users to get more general help (e.g. how do I auth with kubernetes?). Thanks for providing such great feedback!