Failed to JSON parse a line from worker stream

zou-can commented 7 months ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

After I upgrade AWX to 24.2.0, i met some JSON parse error:

Failed to JSON parse a line from worker stream. Error: Expecting ',' delimiter
Failed to JSON parse a line from worker stream. Error: Invalid \escape

I upgraded AWX due to issue - https://github.com/ansible/awx/issues/14693, and i follows the comment - https://github.com/ansible/awx/issues/14693#issuecomment-2007929031, but Failed to JSON parse a still happens, and happens more often.

Note that my AWX jobs are performed to collect configs for network devices, and job outputs are large.

AWX version

24.2.0

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[X] Other

Installation method

kubernetes

Modifications

yes

Ansible version

2.15.10

Operating system

CentOS 7

Web browser

Chrome

Steps to reproduce

Lauch a job which will produce a large output, and output contains special characters(e.g. , \ []...)

Expected results

Job result is success.

Actual results

Job result is error: Failed to JSON parse a line from worker stream. ...

Additional information

I use a custom image for default instance group, however, the image is built from quay.io/ansible/awx-ee:24.2.0

The Dockerfile looks like:

FROM quay.io/ansible/awx-ee:24.2.0

USER root

# Install Python libs
ADD python_environment.txt ./
RUN pip install --no-cache-dir -r python_environment.txt

# Install expect
RUN dnf install -y expect

# Ensures cryptographic components compatibility with legacy systems
# https://access.redhat.com/articles/3642912
# https://access.redhat.com/documentation/zh-cn/red_hat_enterprise_linux/9/html/9.1_release_notes/enhancement_security
RUN echo 'min_rsa_size@openssh = 1024' > /etc/crypto-policies/policies/modules/RSA-OPENSSH-1024.pmod
RUN update-crypto-policies --set LEGACY:RSA-OPENSSH-1024

USER 1000

# Add ssh config files
ADD --chown=1000 --chmod=600 id_rsa /runner/.ssh/
ADD --chown=1000 --chmod=644 id_rsa.pub config /runner/.ssh/

bhaskar-alagala commented 7 months ago

Same version, we are also facing same issue. our Kubernetes on AKS Failed to JSON parse a line from worker stream. Error. Expecting value: line 1 column 1 ( char0) line with invalid JSON data: b

bhaskar-alagala commented 7 months ago

kubectl logs awx-webxxxxxx -n xxxx

I am getting Imagepullbackoff and in awx jobs, I am getting below error.

Failed to JSON parse a line from worker stream. Error. Expecting value: line 1 column 1 ( char0) line with invalid JSON data: b

TheRealHaoLiu commented 6 months ago

I am getting Imagepullbackoff and in awx jobs, I am getting below error.

can u check the job detail from /api/v2/jobs/ and see if a specific result_traceback exist

zou-can commented 6 months ago

I am getting Imagepullbackoff and in awx jobs, I am getting below error.

I didn't meet above Imagepullbackoff error.

The result_traceback field in my job detail is:

{
    "result_traceback": "Receptor detail:\nPod Running"
}

Note that before i upgraded AWX to 24.2.0, the result_traceback field is:

{
    "result_traceback": "Receptor detail:\nError with pod's stdout: EOF",
}

pwalters04 commented 2 months ago

Same issue, for running against 1000 servers in an instance group container of:

apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  serviceAccountName: default
  automountServiceAccountToken: false
  containers:
    - image: awx_custom_ee
      name: worker
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      resources:
        requests:
          cpu: 1.0
          memory: 1Gi
        limits:
          cpu: 2.0
          memory: 6Gi
  tolerations:
    - effect: NoSchedule
      key: workloadisolation
      operator: Equal
      value: job
  nodeSelector:
    workloadisolation: job

I received error:

    "job_explanation": "Failed to JSON parse a line from worker stream. Error: Expecting value: line 1 column 1 (char 0) Line with invalid JSON data: b''",
    "execution_node": "",
    "controller_node": "awx-task-pod",
    "result_traceback": "Receptor detail:\nFinished",
    "event_processing_finished": true,

ansible / awx