search

pixie-io / pixie

Instant Kubernetes-Native Application Observability
https://px.dev
Apache License 2.0
5.42k stars 414 forks source link

Table 'http_events' not found on GKE 1.28.3 #1871

Closed maxlemieux closed 2 months ago

maxlemieux commented 3 months ago

Describe the bug Scripts which use eBPF fail to run on GKE 1.28.3.

To Reproduce Steps to reproduce the behavior:

  1. Install Pixie on a new GKE Standard cluster (version 1.28.3-gke.1286000). I used machine type e2-standard-2, and installed through the New Relic chart bundle.
  2. Go to the px/cluster script in Pixie.
  3. See error: 
    Failed to execute script
Compiler error on line 33, column 22: Table 'http_events' not found..

Only the px/agent_status script seems to work.

Expected behavior Expected eBPF scripts to run.

Screenshots

Screenshot 2024-04-10 at 7 16 26 PM

Logs I couldn't use px logs to collect the logs since this is using a New Relic shadow account for Pixie. Here are the vizier-pem logs showing an error:

Click me ``` Defaulted container "pem" out of: pem, qb-wait (init) I20240411 02:00:35.018565 11276 env.cc:47] Started: /app/src/vizier/services/agent/pem/pem Started external stacktrace collection signal processor thread I20240411 02:00:35.019026 11276 kernel_version.cc:199] Found Linux kernel version using .note section. I20240411 02:00:35.019078 11276 pem_main.cc:68] Pixie PEM. Version: v0.14.8+Distribution.ce1e3f3.202312122235.1.RELEASE.jenkins, id: 23dcaa75-0167-47f1-a5bc-6a9a21038d05, kernel version: 6.1.58 I20240411 02:00:35.019224 11276 stirling.cc:958] Creating Stirling, registered sources: [process_stats, network_stats, jvm_stats, socket_tracer, perf_profiler, proc_exit_tracer, stirling_error] I20240411 02:00:35.019318 11276 system_info.cc:42] Location of proc: /proc I20240411 02:00:35.019340 11276 system_info.cc:43] Location of sysfs: /sys/fs I20240411 02:00:35.019387 11276 system_info.cc:44] Number of CPUs: 2 I20240411 02:00:35.019507 11276 system_info.cc:35] /proc/version: Linux version 6.1.58+ (builder@d367cb498058) (Chromium OS 16.0_pre484197_p20230405-r10 clang version 16.0.0 (/var/tmp/portage/sys-devel/llvm-16.0_pre484197_p20230405-r10/work/llvm-16.0_pre484197_p20230405/clang 2916b99182752b1aece8cc4479d8d6a20b5e02da), LLD 16.0.0) #1 SMP PREEMPT_DYNAMIC Sat Nov 4 14:14:29 UTC 2023 I20240411 02:00:35.019842 11276 system_info.cc:35] /host/etc/lsb-release: CHROMEOS_AUSERVER=https://tools.google.com/service/update2 CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00} CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1} CHROMEOS_DEVSERVER= CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00} CHROMEOS_RELEASE_BOARD=lakitu CHROMEOS_RELEASE_BRANCH_NUMBER=66 CHROMEOS_RELEASE_BUILD_NUMBER=17800 CHROMEOS_RELEASE_BUILD_TYPE=Official Build CHROMEOS_RELEASE_CHROME_MILESTONE=109 CHROMEOS_RELEASE_DESCRIPTION=17800.66.19 (Official Build) stable-channel lakitu CHROMEOS_RELEASE_KEYSET=v10 CHROMEOS_RELEASE_NAME=Chrome OS CHROMEOS_RELEASE_PATCH_NUMBER=19 CHROMEOS_RELEASE_TRACK=stable-channel CHROMEOS_RELEASE_VERSION=17800.66.19 DEVICETYPE=OTHER GOOGLE_RELEASE=17800.66.19 HWID_OVERRIDE=LAKITU DEFAULT I20240411 02:00:35.019901 11276 system_info.cc:35] /host/etc/os-release: NAME="Container-Optimized OS" ID=cos PRETTY_NAME="Container-Optimized OS from Google" HOME_URL="https://cloud.google.com/container-optimized-os/docs" BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us" GOOGLE_METRICS_PRODUCT_ID=26 KERNEL_COMMIT_ID=77d913a39ed76e488be139379dffe44e172527f9 GOOGLE_CRASH_ID=Lakitu VERSION=109 VERSION_ID=109 BUILD_ID=17800.66.19 I20240411 02:00:35.019943 11276 probe_cleaner.cc:102] Cleaning probes from /sys/kernel/debug/tracing/kprobe_events with the following marker: __pixie__ I20240411 02:00:35.020022 11276 probe_cleaner.cc:117] All Stirling probes removed (count=0) I20240411 02:00:35.020035 11276 probe_cleaner.cc:102] Cleaning probes from /sys/kernel/debug/tracing/uprobe_events with the following marker: __pixie__ I20240411 02:00:35.020079 11276 probe_cleaner.cc:117] All Stirling probes removed (count=0) I20240411 02:00:35.020092 11276 source_connector.cc:35] Initializing source connector: process_stats I20240411 02:00:35.020104 11276 stirling.cc:438] Adding info class: [process_stats/process_stats] I20240411 02:00:35.020120 11276 source_connector.cc:35] Initializing source connector: network_stats I20240411 02:00:35.020133 11276 stirling.cc:438] Adding info class: [network_stats/network_stats] I20240411 02:00:35.020159 11276 source_connector.cc:35] Initializing source connector: jvm_stats I20240411 02:00:35.020172 11276 stirling.cc:438] Adding info class: [jvm_stats/jvm_stats] I20240411 02:00:35.020443 11276 source_connector.cc:35] Initializing source connector: socket_tracer I20240411 02:00:35.020495 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+ I20240411 02:00:35.020514 11276 linux_headers.cc:385] Detected kernel release (uname -r): 6.1.58+ I20240411 02:00:35.021205 11276 linux_headers.cc:400] Not Found : Could not find 'source' or 'build' under /lib/modules/6.1.58+. I20240411 02:00:35.021323 11276 linux_headers.cc:248] Looking for host Linux headers at /host/lib/modules/6.1.58+/build. I20240411 02:00:35.021364 11276 linux_headers.cc:403] Not Found : Did not find host headers at path: /host/lib/modules/6.1.58+/build. I20240411 02:00:35.021394 11276 linux_headers.cc:346] Attempting to install packaged headers. W20240411 02:00:35.021690 11276 linux_headers.cc:319] Ignoring /px/linux-headers-x86_64-4.14.304.tar.gz since it does not conform to the naming convention W20240411 02:00:35.021770 11276 linux_headers.cc:319] Ignoring /px/linux-headers-x86_64-4.19.271.tar.gz since it does not conform to the naming convention I20240411 02:00:35.021883 11276 linux_headers.cc:352] Using packaged header: /px/linux-headers-x86_64-6.1.8.tar.gz I20240411 02:00:35.707615 11276 linux_headers.cc:56] Overriding linux version code to 393530 I20240411 02:00:35.708001 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+ I20240411 02:00:35.708133 11276 linux_headers.cc:98] Found kernel config at: /proc/config.gz. I20240411 02:00:35.712800 11276 linux_headers.cc:377] Successfully installed packaged copy of headers at /lib/modules/6.1.58+/build I20240411 02:00:35.712833 11276 bcc_wrapper.cc:94] Resolving task_struct offsets. I20240411 02:00:35.801630 11290 bcc_wrapper.cc:166] Initializing BPF program ... I20240411 02:00:36.804966 11290 scoped_timer.h:48] Timer(init_bpf_program) : 1.00 s cannot create /var/tmp/bcc WARNING: cannot get prog tag, ignore saving source with program tag I20240411 02:00:36.817472 11276 bcc_wrapper.cc:166] Initializing BPF program ... I20240411 02:00:37.834250 11276 scoped_timer.h:48] Timer(init_bpf_program) : 1.02 s cannot create /var/tmp/bcc WARNING: cannot get prog tag, ignore saving source with program tag I20240411 02:00:37.849454 11276 bcc_wrapper.cc:97] Successfully resolved task_struct offsets: {real_start_time=1832, group_leader=1584, exit_code=1416} I20240411 02:00:37.849560 11276 bcc_wrapper.cc:166] Initializing BPF program ... src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1300:25: error: expected ')' if (({ typeof(void btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) { ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1300:18: note: to match this '(' if (({ typeof(void btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) { ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1300:51: error: variable has incomplete type 'typeof(void)' (aka 'void') if (({ typeof(void btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) { ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1303:43: error: expected ')' connect_args.addr = ({ typeof(void btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1303:36: note: to match this '(' connect_args.addr = ({ typeof(void btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1303:69: error: variable has incomplete type 'typeof(void)' (aka 'void') connect_args.addr = ({ typeof(void btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1311:46: error: expected ')' write_args.iov = ({ typeof(struct iovec btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1311:31: note: to match this '(' write_args.iov = ({ typeof(struct iovec btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1311:20: error: assigning to 'const struct iovec *' from incompatible type 'typeof(struct iovec)' (aka 'struct iovec') write_args.iov = ({ typeof(struct iovec btf_type_tag(user)*const) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; }); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1424:25: error: expected ')' if (({ typeof(void btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) { ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1424:18: note: to match this '(' if (({ typeof(void btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) { ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1424:46: error: variable has incomplete type 'typeof(void)' (aka 'void') if (({ typeof(void btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }) != NULL) { ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1427:43: error: expected ')' connect_args.addr = ({ typeof(void btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1427:36: note: to match this '(' connect_args.addr = ({ typeof(void btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1427:64: error: variable has incomplete type 'typeof(void)' (aka 'void') connect_args.addr = ({ typeof(void btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_name); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1435:45: error: expected ')' read_args.iov = ({ typeof(struct iovec btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1435:30: note: to match this '(' read_args.iov = ({ typeof(struct iovec btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; }); ^ src/stirling/source_connectors/socket_tracer/bcc_bpf/socket_trace.c:1435:19: error: assigning to 'const struct iovec *' from incompatible type 'typeof(struct iovec)' (aka 'struct iovec') read_args.iov = ({ typeof(struct iovec btf_type_tag(user)*) _val; __builtin_memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (void *)&msghdr->msg_iov); _val; }); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 12 errors generated. I20240411 02:00:41.324113 11276 scoped_timer.h:48] Timer(init_bpf_program) : 3.47 s W20240411 02:00:41.326061 11276 stirling.cc:416] Source Connector (registry name=socket_tracer) not instantiated, error: Internal : Unable to initialize BCC BPF program: Unable to initialize BPF program I20240411 02:00:41.326159 11276 source_connector.cc:35] Initializing source connector: perf_profiler I20240411 02:00:41.326224 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+ I20240411 02:00:41.326244 11276 linux_headers.cc:385] Detected kernel release (uname -r): 6.1.58+ I20240411 02:00:41.326279 11276 linux_headers.cc:206] Using Linux headers from: /lib/modules/6.1.58+/build. I20240411 02:00:41.326292 11276 bcc_wrapper.cc:90] Returning the previously resolved TaskStructOffsets object I20240411 02:00:41.326350 11276 bcc_wrapper.cc:166] Initializing BPF program ... I20240411 02:00:43.133134 11276 scoped_timer.h:48] Timer(init_bpf_program) : 1.81 s cannot create /var/tmp/bcc WARNING: cannot get prog tag, ignore saving source with program tag I20240411 02:00:43.134491 11276 perf_profile_connector.cc:153] PerfProfiler: Stack trace profiling sampling probe successfully deployed. I20240411 02:00:43.134653 11276 perf_profile_connector.cc:169] PerfProfiler: Java symbolization enabled. I20240411 02:00:43.134789 11276 java_symbolizer.cc:233] JavaSymbolizer found agent lib /px/libpx-java-agent.so. I20240411 02:00:43.134922 11276 stirling.cc:438] Adding info class: [perf_profiler/stack_traces.beta] I20240411 02:00:43.135090 11276 source_connector.cc:35] Initializing source connector: proc_exit_tracer I20240411 02:00:43.135188 11276 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.58+ I20240411 02:00:43.135290 11276 linux_headers.cc:385] Detected kernel release (uname -r): 6.1.58+ I20240411 02:00:43.135391 11276 linux_headers.cc:206] Using Linux headers from: /lib/modules/6.1.58+/build. I20240411 02:00:43.135548 11276 bcc_wrapper.cc:90] Returning the previously resolved TaskStructOffsets object I20240411 02:00:43.135702 11276 bcc_wrapper.cc:166] Initializing BPF program ... I20240411 02:00:43.976743 11276 scoped_timer.h:48] Timer(init_bpf_program) : 840.96 ms I20240411 02:00:43.976873 11276 proc_exit_connector.cc:86] Writing exit_code's offset to BPF array: offset=1416 bpf_array=control_values index=0 cannot create /var/tmp/bcc WARNING: cannot get prog tag, ignore saving source with program tag I20240411 02:00:43.982354 11276 stirling.cc:438] Adding info class: [proc_exit_tracer/proc_exit_events] I20240411 02:00:43.982455 11276 source_connector.cc:35] Initializing source connector: stirling_error I20240411 02:00:43.982564 11276 stirling.cc:438] Adding info class: [stirling_error/stirling_error] I20240411 02:00:43.982631 11276 stirling.cc:438] Adding info class: [stirling_error/probe_status] I20240411 02:00:43.982671 11276 stirling.cc:419] Stirling successfully initialized. E0411 02:00:43.983477093 11276 dns_resolver_ares.cc:456] no server name supplied in dns URI E0411 02:00:43.983625510 11276 channel.cc:120] channel stack builder failed: UNKNOWN: the target uri is not valid: dns:/// I20240411 02:00:43.994021 11276 manager.cc:154] Hostname: gke-mlemieux-gke-pool-1-e24203b4-vksd I20240411 02:01:07.767810 11276 cgroup_path_resolver.cc:141] Auto-discovered CGroup base path: /sys/fs/cgroup I20240411 02:01:07.773299 11276 cgroup_path_resolver.cc:144] Auto-discovered example path: /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581052f4_3e86_404a_87c6_267ac4c2f39a.slice/cri-containerd-e59fc320752e647cd963a1561602e3b0f744f209801def33a1259175ef182714.scope/cgroup.procs I20240411 02:01:07.773522 11276 cgroup_path_resolver.cc:148] Auto-discovered template: /sys/fs/cgroup/kubepods.slice/kubepods-$2.slice/kubepods-$2-pod$0.slice/cri-containerd-$1.scope/cgroup.procs I20240411 02:01:07.773583 11276 cgroup_metadata_reader.cc:41] Using path_resolver with configuration: template=/sys/fs/cgroup/kubepods.slice/kubepods-$2.slice/kubepods-$2-pod$0.slice/cri-containerd-$1.scope/cgroup.procs pod_id_separators=_ qos_spearator=- I20240411 02:01:07.774121 11656 stirling.cc:792] Stirling is running. I20240411 02:01:07.775919 11656 perf_profile_connector.cc:428] PerfProfileConnector statistics: kBPFMapSwitchoverEvent=1 kCumulativeSumOfAllStackTraces=4480 kLossHistoEvent=0 I20240411 02:01:07.775952 11656 perf_profile_connector.cc:442] PerfProfileConnector u_symbolizer num_symbols_cached=0 hits=0 accesses=0 hit_rate=0 I20240411 02:01:07.775966 11656 perf_profile_connector.cc:445] PerfProfileConnector k_symbolizer num_symbols_cached=0 hits=0 accesses=0 hit_rate=0 I20240411 02:01:07.850221 11276 heartbeat.cc:143] Heartbeat ACK latency moving average: 3 ms I20240411 02:01:07.875310 11656 run_core_stats.cc:111] |main_loop_iters,no_work_iters,useful_iters,push+transfer,transfer,push,min_push+transfer,max_push+transfer,total_0.00_ms,total_0.01_ms,total_0.02_ms,total_0.03_ms,total_0.06_ms,total_0.10_ms,total_0.18_ms,total_0.32_ms,total_0.56_ms,total_1.00_ms,total_1.78_ms,total_3.16_ms,total_5.62_ms,total_10.00_ms,total_17.78_ms,total_31.62_ms,total_56.23_ms,total_100.00_ms,total_177.83_ms,total_316.23_ms,total_562.34_ms,total_1000.00_ms,total_10000000.00_ms,no_work_0.00_ms,no_work_0.01_ms,no_work_0.02_ms,no_work_0.03_ms,no_work_0.06_ms,no_work_0.10_ms,no_work_0.18_ms,no_work_0.32_ms,no_work_0.56_ms,no_work_1.00_ms,no_work_1.78_ms,no_work_3.16_ms,no_work_5.62_ms,no_work_10.00_ms,no_work_17.78_ms,no_work_31.62_ms,no_work_56.23_ms,no_work_100.00_ms,no_work_177.83_ms,no_work_316.23_ms,no_work_562.34_ms,no_work_1000.00_ms,no_work_10000000.00_ms I20240411 02:01:08.927611 11276 exec.cc:97] Queries in flight: 0 I20240411 02:01:08.928225 11663 exec.cc:51] Executing query: id=39cc10f9-ac60-4869-a9bb-73e15cb7b03a I20240411 02:01:08.945636 11663 exec.cc:63] Completed query: id=39cc10f9-ac60-4869-a9bb-73e15cb7b03a W20240411 02:01:12.769654 11276 state_manager.cc:276] Failed to read PID info for pod=882acabe-2a97-44cd-a81d-af3ead9e3915, cid=ed9e71ab6216ed2de124fa02e755c7910220ad446b6aaeed546669d5d3136c30 [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882acabe_2a97_44cd_a81d_af3ead9e3915.slice/cri-containerd-ed9e71ab6216ed2de124fa02e755c7910220ad446b6aaeed546669d5d3136c30.scope/cgroup.procs] W20240411 02:01:12.770524 11276 state_manager.cc:276] Failed to read PID info for pod=9cc0bf12-a9d1-4fe7-98b6-4b0342d970f3, cid=59f0291a730159a7c735e1a0577f2927f8aac63263d9cb02523bdab585782ee6 [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc0bf12_a9d1_4fe7_98b6_4b0342d970f3.slice/cri-containerd-59f0291a730159a7c735e1a0577f2927f8aac63263d9cb02523bdab585782ee6.scope/cgroup.procs] W20240411 02:01:12.770881 11276 state_manager.cc:276] Failed to read PID info for pod=581052f4-3e86-404a-87c6-267ac4c2f39a, cid= [msg=Failed to open file /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581052f4_3e86_404a_87c6_267ac4c2f39a.slice/cri-containerd-.scope/cgroup.procs] ```

App information (please complete the following information):

Additional context Please let me know if I can provide any more information about this issue. It's also affecting 0.14.6, but did not reportedly happen in the same environment with 0.12.x.

ddelnano commented 2 months ago

@maxlemieux this will be fixed in the next Pixie release v0.14.10 and should be available sometime early this week. This issue was addressed in #1909.

ddelnano commented 2 months ago

v0.14.10 is released.