Failed to JSON parse a line from worker stream

zou-can commented 3 weeks ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

After I upgrade AWX to 24.2.0, i met some JSON parse error:

Failed to JSON parse a line from worker stream. Error: Expecting ',' delimiter
Failed to JSON parse a line from worker stream. Error: Invalid \escape

I upgraded AWX due to issue - https://github.com/ansible/awx/issues/14693, and i follows the comment - https://github.com/ansible/awx/issues/14693#issuecomment-2007929031, but Failed to JSON parse a still happens, and happens more often.

Note that my AWX jobs are performed to collect configs for network devices, and job outputs are large.

AWX version

24.2.0

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[X] Other

Installation method

kubernetes

Modifications

yes

Ansible version

2.15.10

Operating system

CentOS 7

Web browser

Chrome

Steps to reproduce

Lauch a job which will produce a large output, and output contains special characters(e.g. , \ []...)

Expected results

Job result is success.

Actual results

Job result is error: Failed to JSON parse a line from worker stream. ...

Additional information

I use a custom image for default instance group, however, the image is built from quay.io/ansible/awx-ee:24.2.0

The Dockerfile looks like:

FROM quay.io/ansible/awx-ee:24.2.0

USER root

# Install Python libs
ADD python_environment.txt ./
RUN pip install --no-cache-dir -r python_environment.txt

# Install expect
RUN dnf install -y expect

# Ensures cryptographic components compatibility with legacy systems
# https://access.redhat.com/articles/3642912
# https://access.redhat.com/documentation/zh-cn/red_hat_enterprise_linux/9/html/9.1_release_notes/enhancement_security
RUN echo 'min_rsa_size@openssh = 1024' > /etc/crypto-policies/policies/modules/RSA-OPENSSH-1024.pmod
RUN update-crypto-policies --set LEGACY:RSA-OPENSSH-1024

USER 1000

# Add ssh config files
ADD --chown=1000 --chmod=600 id_rsa /runner/.ssh/
ADD --chown=1000 --chmod=644 id_rsa.pub config /runner/.ssh/

bhaskar-alagala commented 3 weeks ago

Same version, we are also facing same issue. our Kubernetes on AKS Failed to JSON parse a line from worker stream. Error. Expecting value: line 1 column 1 ( char0) line with invalid JSON data: b

bhaskar-alagala commented 3 weeks ago

kubectl logs awx-webxxxxxx -n xxxx

I am getting Imagepullbackoff and in awx jobs, I am getting below error.

Failed to JSON parse a line from worker stream. Error. Expecting value: line 1 column 1 ( char0) line with invalid JSON data: b

TheRealHaoLiu commented 3 days ago

I am getting Imagepullbackoff and in awx jobs, I am getting below error.

can u check the job detail from /api/v2/jobs/ and see if a specific result_traceback exist

ansible / awx