K3s Raspberry pi container issue

[HighKingPhoenix]

Describe the bug A clear and concise description of what the bug is. Please do not use an Issue to ask for help deploying the stack in the cluster. Troubleshoot your cluster configuration first.

Troubleshooting

1. Which kind of Kubernetes cluster are you using? (Kubernetes, K3s, etc)
2. Are all pods in "Running" state? If any is in CrashLoopback or Error, check it's logs.
3. You cluster already works with other applications that have HTTP/HTTPS? If not, first deploy an example NGINX and test it's access thru the created URL.
4. If you enabled persistence, do your cluster already provides persistent storage (PVs) to other applications?
5. Does it provides dynamic storage thru StorageClass?
6. If you deployed the monitoring stack and some targets are not available or showing no metrics in Grafana, make sure you don't have IPTables rules or use a firewall on your nodes before deploying Kubernetes.

Customizations

1. Did you customize vars.jsonnet? Put the contents below:

{
  _config+:: {
    namespace: 'monitoring',
  },
  // Enable or disable additional modules
  modules: [
    {
      // After deployment, run the create_gmail_auth.sh script from scripts dir.
      name: 'smtpRelay',
      enabled: false,
      file: import 'modules/smtp_relay.jsonnet',
    },
    {
      name: 'armExporter',
      enabled: true,
      file: import 'modules/arm_exporter.jsonnet',
    },
    {
      name: 'upsExporter',
      enabled: false,
      file: import 'modules/ups_exporter.jsonnet',
    },
    {
      name: 'metallbExporter',
      enabled: false,
      file: import 'modules/metallb.jsonnet',
    },
    {
      name: 'nginxExporter',
      enabled: false,
      file: import 'modules/nginx.jsonnet',
    },
    {
      name: 'traefikExporter',
      enabled: true,
      file: import 'modules/traefik.jsonnet',
    },
    {
      name: 'elasticExporter',
      enabled: false,
      file: import 'modules/elasticsearch_exporter.jsonnet',
    },
    {
      name: 'speedtestExporter',
      enabled: true,
      file: import 'modules/speedtest_exporter.jsonnet',
    },
  ],

  k3s: {
    enabled: true,
    master_ip: ['192.168.1.29'],
  },

  // Domain suffix for the ingresses
  suffixDomain: '192.168.1.48.nip.io',
  // Additional domain suffixes for the ingresses.
  // For example suffixDomain could be an external one and this a local domain.
  additionalDomains: [],
  // If TLSingress is true, a self-signed HTTPS ingress with redirect will be created
  TLSingress: true,
  // If UseProvidedCerts is true, provided files will be used on created HTTPS ingresses.
  // Use a wildcard certificate for the domain like ex. "*.192.168.99.100.nip.io"
  UseProvidedCerts: false,
  TLSCertificate: importstr 'server.crt',
  TLSKey: importstr 'server.key',

  // Persistent volume configuration
  enablePersistence: {
    // Setting these to false, defaults to emptyDirs.
    prometheus: false,
    grafana: false,
    // If using a pre-created PV, fill in the names below. If blank, they will use the default Storag>    prometheusPV: '',
    grafanaPV: '',
    // If required to use a specific storageClass, keep the PV names above blank and fill the storage>    storageClass: '',
    // Define the PV sizes below
    prometheusSizePV: '2Gi',
    grafanaSizePV: '20Gi',
  },

  // Configuration for Prometheus deployment
  prometheus: {
    retention: '15d',
    scrapeInterval: '30s',
    scrapeTimeout: '30s',
  },
  grafana: {
    // Grafana "from" email
    from_address: 'myemail@gmail.com',
    // Plugins to be installed at runtime.
    //Ex. plugins: ['grafana-piechart-panel', 'grafana-clock-panel'],
    plugins: [],
    //Ex. env: [ { name: 'http_proxy', value: 'host:8080' } ]
    env: []
  },
}

2. Did you change any other file? Put the contents below:

# file x.yz content

What did you see when trying to access Grafana and Prometheus web GUI root@KingMaster:~/cluster-monitoring# kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE arm-exporter-wcssl 2/2 Running 0 26m node-exporter-s9k5p 2/2 Running 0 25m speedtest-exporter-5bbc667795-8bg5z 0/1 ContainerCreating 0 25m grafana-5854b958d8-gnl2v 0/1 Running 0 25m prometheus-adapter-65b6669bd6-7rgbd 1/1 Running 0 25m arm-exporter-xsdjk 0/2 CreateContainerError 0 26m prometheus-operator-bc686679c-ppgmp 1/2 CreateContainerError 2 26m node-exporter-c5brt 0/2 CreateContainerError 2 25m kube-state-metrics-6bf46964c4-bnbdl 0/3 CreateContainerError 1 25m root@KingMaster:~/cluster-monitoring# kubectl get ingress -n monitoring NAME CLASS HOSTS ADDRESS PORTS AGE alertmanager-main alertmanager.192.168.1.48.nip.io 192.168.1.29,192.168.1.48 80, 443 27m grafana grafana.192.168.1.48.nip.io 192.168.1.29,192.168.1.48 80, 443 27m prometheus-k8s prometheus.192.168.1.48.nip.io 192.168.1.29,192.168.1.48 80, 443

Additional context Add any other context about the problem here.

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring prometheus-operator-bc686679c-ppgmp -c prometheus-operator
Error from server (BadRequest): container "prometheus-operator" in pod "prometheus-operator-bc686679c-ppgmp" is waiting to start: CreateContainerError

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring prometheus-operator-bc686679c-ppgmp -c kube-rbac-proxy
I0531 12:00:25.593877       1 main.go:186] Valid token audiences:
I0531 12:00:25.594063       1 main.go:232] Generating self signed cert as no cert is provided
I0531 12:00:26.767849       1 main.go:281] Starting TCP socket on :8443
I0531 12:00:26.768889       1 main.go:288] Listening securely on :8443

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring kube-state-metrics-6bf46964c4-bnbdl -c kube-state-metrics
Error from server (BadRequest): container "kube-state-metrics" in pod "kube-state-metrics-6bf46964c4-bnbdl" is waiting to start: CreateContainerError

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring kube-state-metrics-6bf46964c4-bnbdl -c kube-rbac-proxy-main
Error from server (BadRequest): container "kube-rbac-proxy-main" in pod "kube-state-metrics-6bf46964c4-bnbdl" is waiting to start: CreateContainerError
root@KingMaster:~/cluster-monitoring#

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring kube-state-metrics-6bf46964c4-bnbdl -c kube-rbac-proxy-self
Error from server (BadRequest): container "kube-rbac-proxy-self" in pod "kube-state-metrics-6bf46964c4-bnbdl" is waiting to start: CreateContainerError

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring node-exporter-c5brt -c node-export
er
Error from server (BadRequest): container "node-exporter" in pod "node-exporter-c5brt" is waiting to start: CreateContainerError

root@KingMaster:~/cluster-monitoring# kubectl logs -f -n monitoring node-exporter-c5brt -c kube-rbac-proxy
Error from server (BadRequest): container "kube-rbac-proxy" in pod "node-exporter-c5brt" is waiting to start: CreateContainerError