jq and sed throwing errors on missing or invalid files

[gavindidrichsen]

Description

jq and sed are throwing warnings during support_parse.sh tech_check

• No such file
• Invalid file content

For example:

➜  42993 git:(master) ✗ support_parse.sh tech_check puppet_enterprise_support_42993_lxpr0721pv_20210202135301  > tech_check.puppet_enterprise_support_42993_lxpr0721pv_20210202135301.json 
jq: Bad JSON in --slurpfile infra puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/pe_infra_status.json: Could not open puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/pe_infra_status.json: No such file or directory
jq: Bad JSON in --slurpfile modules puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/modules.json: Invalid numeric literal at line 1, column 10
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory

Further investigation into the invalid modules.json reveals:

• invalid json
• an error message "Forbidden request: /puppet/v3/environment_modules (method :get). Please see the server logs for details."

➜  42993 git:(master) ✗ cat ./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/modules.json
Forbidden request: /puppet/v3/environment_modules (method :get). Please see the server logs for details.
➜  42993 git:(master) ✗ 

The puppetserver-access.log contains the following, which may or may not be related

10.188.25.23 - - [02/Feb/2021:13:48:36 +0100] "GET /puppet/v3/environment_modules HTTP/1.1" 200 47485 "-" "Apache-HttpAsyncClient/4.1.4 (Java/11.0.8-internal)" 2063 - 2002

[m0dular]

I feel like this behavior depends on if we expect the file to be there or not. For example, a sup script from a compiler or primary with external postgres would not have a thundering herd file, so we should check if it exists. But sometimes if services are down the .json files will have errors, and I think jq pointing that out is beneficial. I'll think about the best approach for this issue.

[gavindidrichsen]

...I'll think about the best approach for this issue.

Thanks @m0dular!