GoogleCloudPlatform / cloud-code-vscode

Cloud Code for Visual Studio Code: Issues, Documentation and more
Other
415 stars 111 forks source link

view logs for pod created by job results in error #463

Closed jeremybramwell closed 2 years ago

jeremybramwell commented 3 years ago

Issue Type: Bug

Context

When I right-click on a pod that was created by a job and select view logs the logs Viewer tab displays the error:

Error from server (NotFound): replicasets.apps "rest-init" not found

pod manifest:

pod manifest ``` yaml apiVersion: v1 kind: Pod metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false" creationTimestamp: "2021-08-24T21:53:46Z" generateName: rest-init- labels: controller-uid: 203d02e8-52d8-4835-811a-edf8090872fd job-name: rest-init managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:cluster-autoscaler.kubernetes.io/safe-to-evict: {} f:generateName: {} f:labels: .: {} f:controller-uid: {} f:job-name: {} f:ownerReferences: .: {} k:{"uid":"203d02e8-52d8-4835-811a-edf8090872fd"}: .: {} f:apiVersion: {} f:blockOwnerDeletion: {} f:controller: {} f:kind: {} f:name: {} f:uid: {} f:spec: f:containers: k:{"name":"init-rest"}: .: {} f:args: {} f:command: {} f:env: .: {} k:{"name":"LOGFACILITY"}: .: {} f:name: {} f:value: {} k:{"name":"LOGJSONFORMAT"}: .: {} f:name: {} f:value: {} k:{"name":"LOGLEVEL"}: .: {} f:name: {} f:value: {} k:{"name":"POSTGRES_PASSWORD"}: .: {} f:name: {} f:value: {} k:{"name":"POSTGRES_USER"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_CLIENT"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_HOST"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_NAME"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_PORT"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_ROOT_DB"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_ROOT_PASS"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DB_ROOT_USER"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_DEPLOY_OUT"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_HOSTNAME"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_REDIS_HOST"}: .: {} f:name: {} f:valueFrom: .: {} f:configMapKeyRef: .: {} f:key: {} f:name: {} k:{"name":"REST_INIT_REDIS_PORT"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_TIMEOUT"}: .: {} f:name: {} f:value: {} k:{"name":"REST_INIT_TRUSTED_PROXY"}: .: {} f:name: {} f:value: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/init-disk"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:imagePullSecrets: .: {} k:{"name":"k8s-gcr-read-access"}: .: {} f:name: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"init-disk"}: .: {} f:emptyDir: .: {} f:sizeLimit: {} f:name: {} manager: kube-controller-manager operation: Update time: "2021-08-24T21:53:46Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:phase: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"10.40.1.60"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update time: "2021-08-24T21:56:00Z" name: rest-init-jm6ml namespace: default ownerReferences: - apiVersion: batch/v1 blockOwnerDeletion: true controller: true kind: Job name: rest-init uid: 203d02e8-52d8-4835-811a-edf8090872fd resourceVersion: "527844" uid: c4ea399e-0727-4fe7-9226-18d722800525 spec: containers: - args: - -euc - image: gcr.io/nth-fort-242316/cloud-platform/rest-v1.12:1.2.0-rc.1-60f099a3 imagePullPolicy: IfNotPresent name: init-rest resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/init-disk name: init-disk - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-jxk7c readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: - name: k8s-gcr-read-access nodeName: gke-dev-cluster-1-default-pool-794c8916-4vjw preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - emptyDir: sizeLimit: 16Mi name: init-disk - name: default-token-jxk7c secret: defaultMode: 420 secretName: default-token-jxk7c status: conditions: - lastProbeTime: null lastTransitionTime: "2021-08-24T21:53:46Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2021-08-24T21:56:00Z" message: 'containers with unready status: [init-rest]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2021-08-24T21:56:00Z" message: 'containers with unready status: [init-rest]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2021-08-24T21:53:46Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://ebe4eb607ef2d6b7cec3b1566596cc07c403785640a314b022300987bc2b8a3b image: gcr.io/nth-fort-242316/cloud-platform/rest-v1.12:1.2.0-rc.1-60f099a3 imageID: gcr.io/nth-fort-242316/cloud-platform/rest-v1.12@sha256:7f88ede72807e292449e1028db9ceebe304a88d66fa8ea7e2e826f54cc81738c lastState: {} name: init-rest ready: false restartCount: 0 started: false state: terminated: containerID: containerd://ebe4eb607ef2d6b7cec3b1566596cc07c403785640a314b022300987bc2b8a3b exitCode: 2 finishedAt: "2021-08-24T21:55:59Z" reason: Error startedAt: "2021-08-24T21:53:47Z" hostIP: 10.150.0.46 phase: Failed podIP: 10.40.1.60 podIPs: - ip: 10.40.1.60 qosClass: BestEffort startTime: "2021-08-24T21:53:46Z" ```
job manifest ``` yaml apiVersion: batch/v1 kind: Job metadata: name: rest-init spec: template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false" # labels: # app.kubernetes.io/name: restinfr spec: imagePullSecrets: - name: "k8s-gcr-read-access" containers: # initializes the database and deploys Rest - name: init-rest image: gcr.io/nth-fort-242316/cloud-platform/rest-v1.12:1.2.0-rc.1-60f099a3 command: - /bin/sh args: - -euc - volumeMounts: - name: init-disk mountPath: /mnt/init-disk readOnly: false restartPolicy: Never volumes: - name: init-disk emptyDir: sizeLimit: 16Mi # the container cannot be started more than once backoffLimit: 0 ```

Extension version: 1.13.2 VS Code version: Code 1.59.0 (379476f0e13988d90fab105c5c19e7abc8b1dea8, 2021-08-04T23:14:40.191Z) OS version: Darwin x64 20.6.0 Restricted Mode: No Remote OS version: Linux x64 5.11.0-27-generic

System Info |Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz (8 x 2000)| |GPU Status|2d_canvas: enabled
gpu_compositing: enabled
metal: disabled_off
multiple_raster_threads: enabled_on
oop_rasterization: enabled
opengl: enabled_on
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
webgl: enabled
webgl2: enabled| |Load (avg)|3, 3, 3| |Memory (System)|16.00GB (0.04GB free)| |Process Argv|--crash-reporter-id 1a1ed6b7-e198-43cf-9c3e-a3db36f23c78| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: bram-razor| |OS|Linux x64 5.11.0-27-generic| |CPUs|Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (8 x 2600)| |Memory (System)|15.53GB (4.72GB free)| |VM|0%|
A/B Experiments ``` vsliv368cf:30146710 vsreu685:30147344 python383cf:30185419 pythonvspyt602:30300191 vspor879:30202332 vspor708:30202333 vspor363:30204092 pythonvspyt639:30300192 pythontb:30283811 pythonptprofiler:30281270 vshan820:30294714 vstes263:30335439 pythondataviewer:30285071 pythonvsuse255:30340121 vscod805:30301674 pythonvspyt200:30340761 vscextlang:30333561 binariesv615:30325510 pythonvssor306:30344512 bridge0708:30335490 vstre464:30350172 bridge0723:30353136 vsdyn420:30354109 ```
quoctruong commented 3 years ago

Hi @jeremybramwell, if you run kubectl logs rest-init-jm6ml, do you get any results?

jeremybramwell commented 3 years ago

Yes, kubectl logs pod/rest-init-jm6ml returns logs as does kubectl logs job/rest-init

On Wed, Aug 25, 2021 at 7:05 PM Quoc Truong @.***> wrote:

Hi @jeremybramwell https://github.com/jeremybramwell, if you run kubectl logs rest-init-jm6ml, do you get any results?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/cloud-code-vscode/issues/463#issuecomment-905933572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFC736DCHDSLDRL25BO2VS3T6VZL5ANCNFSM5CXYMGIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

-- Jeremy Bramwell, Senior Computer Scientist, 404-425-7897 Medical Information Technology, Inc. Mailstop: A1E67W, One Northside 75, Atlanta, GA 30318

-- https://ehr.meditech.com/expanse           
https://www.linkedin.com/company/meditech 
https://twitter.com/MEDITECH   https://www.facebook.com/MeditechEHR Subscribe https://info.meditech.com/get-great-meditech-content?hsCtaTracking=864299ec-5abf-4004-9c6d-2d051794101f%7Cc911be42-538a-4a48-8dca-a6d4001c6326 to receive emails from MEDITECH or to change email preferences.

COVID-19 Customer Resources https://customer.meditech.com/en/d/covid19resources/homepage.htm

quoctruong commented 3 years ago

Hi @jeremybramwell, can you try enable the setting "Cloudcode: Verbose Logging", restart VSCode and retry opening the logs viewer? Once that's done, you can go to the "Cloud Code" output channel and search for kubectl logs to see the exact command we're using.

jeremybramwell commented 3 years ago

it looks like it isn't getting to the point of calling kubectl logs. The only thing in the verbose logs is:

[8/26/2021, 4:20:46 PM] Running 'kubectl get ReplicaSet rest-init --output json --namespace default' (managed) [8/26/2021, 4:20:46 PM] Error from server (NotFound): replicasets.apps "rest-init" not found

[8/26/2021, 4:20:46 PM] 'kubectl get ReplicaSet rest-init --output json --namespace default' exited with code 1 took 401ms (managed)

On Thu, Aug 26, 2021 at 3:05 PM Quoc Truong @.***> wrote:

Hi @jeremybramwell https://github.com/jeremybramwell, can you try enable the setting "Cloudcode: Verbose Logging", restart VSCode and retry opening the logs viewer? Once that's done, you can go to the "Cloud Code" output channel and search for kubectl logs to see the exact command we're using.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/cloud-code-vscode/issues/463#issuecomment-906668220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFC736CJQ4HYGZZYVFMFTVLT62GABANCNFSM5CXYMGIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Jeremy Bramwell, Senior Computer Scientist, 404-425-7897 Medical Information Technology, Inc. Mailstop: A1E67W, One Northside 75, Atlanta, GA 30318

-- https://ehr.meditech.com/expanse           
https://www.linkedin.com/company/meditech 
https://twitter.com/MEDITECH   https://www.facebook.com/MeditechEHR Subscribe https://info.meditech.com/get-great-meditech-content?hsCtaTracking=864299ec-5abf-4004-9c6d-2d051794101f%7Cc911be42-538a-4a48-8dca-a6d4001c6326 to receive emails from MEDITECH or to change email preferences.

COVID-19 Customer Resources https://customer.meditech.com/en/d/covid19resources/homepage.htm

quoctruong commented 3 years ago

Ah I see. I think we are trying to retrieve the deployment name of this pod which has an ownerReference. We are assuming here that the owner of the pod is a ReplicaSet. Can I ask what type the owner is in this case?

jeremybramwell commented 3 years ago

In this case the owner is a "job". i.e. apiVersion: batch/v1 kind: Job

I guess a replicaset isn't involved for jobs, or at least not for this simple one that only creates a single pod. On a semi-related note I noticed that the job itself doesn't show up under cloud-code anywhere. Should I create an issue for that as well?

On Mon, Aug 30, 2021 at 1:29 PM Quoc Truong @.***> wrote:

Ah I see. I think we are trying to retrieve the deployment name of this pod which has an ownerReference. We are assuming here that the owner of the pod is a ReplicaSet. Can I ask what type the owner is in this case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/cloud-code-vscode/issues/463#issuecomment-908537291, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFC736GOHAUOGJXT6D2OR7TT7O5WZANCNFSM5CXYMGIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Jeremy Bramwell, Senior Computer Scientist, 404-425-7897 Medical Information Technology, Inc. Mailstop: A1E67W, One Northside 75, Atlanta, GA 30318

-- https://ehr.meditech.com/expanse           
https://www.linkedin.com/company/meditech 
https://twitter.com/MEDITECH   https://www.facebook.com/MeditechEHR Subscribe https://info.meditech.com/get-great-meditech-content?hsCtaTracking=864299ec-5abf-4004-9c6d-2d051794101f%7Cc911be42-538a-4a48-8dca-a6d4001c6326 to receive emails from MEDITECH or to change email preferences.

COVID-19 Customer Resources https://customer.meditech.com/en/d/covid19resources/homepage.htm

quoctruong commented 3 years ago

Hi Jeremy, thanks for the information. There is a bug in our code when handling pod created by job and that's why it's not working here. Feel free to create an issue for displaying Job in the Kubernetes Explorer as that will be a separate feature.