XRAY_SAMPLING_RULE_MANIFEST has a length limit

[ghost]

Excepts from the logs

time="2023-09-26T17:43:05Z" level=info msg="XRAY_SAMPLING_RULE_MANIFEST is defined as /devsamples.json, merging it with the x-ray tracing config."
[2023-09-26 17:43:05.899][15][error][tracing] [source/extensions/tracers/xray/config.cc:29] Failed to read sampling rules manifest because of Invalid path: /devsamples.json.

But when I rename and set devsamples.json to just dev, it works This is in v1.25.4.0-prod

[suniltheta]

Getting error from https://github.com/envoyproxy/envoy/blob/6c748c0f71551f173ba31c16848e7c45be1ebfd9/source/extensions/tracers/xray/config.cc#L26C1-L27C1 probably because the path is not valid?

Can you put in some folder location like /tmp/devsample.json and try?

I am not sure how to worked when you just give dev. Probably it read an existing file/folder with the same name?

[ghost]

Great question. Since it worked on one envoy sidecar, but the next one also failed out with [2023-09-26 19:33:06.039][15][error][tracing] [source/extensions/tracers/xray/config.cc:29] Failed to read sampling rules manifest because of Invalid path: /dev.json.

I was trying to find out, but there's no ls in the image. :)

sh-4.2$ cat /dev.json
{
  "version": 2,
  "rules": [
    {
      "description": "noHealth",
      "host": "*",
      "http_method": "*",
      "url_path": "/grpc.health.v1.Health/Check",
      "fixed_target": 0,
      "rate": 0
    },
    {
      "description": "noHealthSchema",
      "host": "schema-registry.*",
      "http_method": "GET",
      "url_path": "/config",
      "fixed_target": 0,
      "rate": 0
    }
  ],
  "default": {
    "fixed_target": 10,
    "rate": 0.05
  }
}

the shell can read it inside the container

[ghost]

followup with

time="2023-09-26T19:33:05Z" level=info msg="XRAY_SAMPLING_RULE_MANIFEST is defined as /dev.json, merging it with the x-ray tracing config."

what's extra confusing is the non-override version I have had working for the past 2 years was set with

COPY samples.json /samples.json
ENV XRAY_SAMPLING_RULE_MANIFEST /samples.json

[suniltheta]

Do you mean this worked prior to v1.25.4.0-prod ?

[suniltheta]

For the purpose of testing you can try to build a custom image on AL2 which will have linux commands working. You just need to copy few files from Envoy image are as per this Dockerfile

FROM public.ecr.aws/appmesh/aws-appmesh-envoy:v1.25.4.0-prod as envoy

FROM public.ecr.aws/amazonlinux/amazonlinux:2
COPY --from=envoy /usr/bin/envoy /usr/bin/envoy
COPY --from=envoy /usr/bin/agent /usr/bin/agent
COPY --from=envoy /aws_appmesh_aggregate_stats.wasm /aws_appmesh_aggregate_stats.wasm
CMD [ "/usr/bin/agent" ]

[ghost]

I just updated to the recent release. No.. I had just a hard-coded /samples.json in the image.

So. I copied the files into /tmp, and forgot to fix the override filename. I get this on startup and it fails

time="2023-09-26T20:46:22Z" level=error msg="Failed to create bootstrap configuration yaml file:[/tmp/envoy-config-3188504322.yaml] could not read file XRAY_SAMPLING_RULE_MANIFEST=\"/dev.json\": open /dev.json: no such file or directory"

When I fix it,

[2023-09-26 20:50:29.452][14][info][tracing] [source/common/tracing/tracer_manager_impl.cc:42] instantiating a new tracer: envoy.tracers.xray

no more error. /shrug

I was thinking maybe permission issues. let me try the full deploy w/ the rules in /tmp

[ghost]

well. putting them in /tmp helped for now. :)

So. If the file doesn't exist, we get this helpful message:

time="2023-09-26T20:46:22Z" level=error msg="Failed to create bootstrap configuration yaml file:[/tmp/envoy-config-3188504322.yaml] could not read file XRAY_SAMPLING_RULE_MANIFEST=\"/dev.json\": open /dev.json: no such file or directory"

So, when these 2 messages appear in the logs, something is weird

time="2023-09-26T17:43:05Z" level=info msg="XRAY_SAMPLING_RULE_MANIFEST is defined as /devsamples.json, merging it with the x-ray tracing config."
[2023-09-26 17:43:05.899][15][error][tracing] [source/extensions/tracers/xray/config.cc:29] Failed to read sampling rules manifest because of Invalid path: /devsamples.json.

So bad filename, we get the first one, and it doesn't start. With the second one, the filename exists, but then it's an "invalid path" later on, which results in no tracing rules applied. I'll roll this to a few more environments and see what happens.

[suniltheta]

is this still a problem? are we good to close this issue?

[ghost]

Putting the files in /tmp worked. It's a strange issue. The image sees the file, but then can't read it. But a different file right next to it can be read. Both copied into the image the same way.
But sure, we can close since I worked around it

[suniltheta]

thank you for confirming