search

CloudHealth / helm

Apache License 2.0
8 stars 15 forks source link

403 on new install #118

Open bn-cforkner opened 3 months ago

bn-cforkner commented 3 months ago

Did a fresh helm install on EKS

=========================================================================

  CHT Containers Collector Environment

  CHT_API_TOKEN: ****

  CHT_CLUSTER_NAME: dev-us-east-2

  JAVA_OPTS:
    -XX:+ExitOnOutOfMemoryError -Xms10M -Xmx891M

  CHT_INTERVAL: 900

=========================================================================

2024-04-24T22:41:32.215Z [main] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
CHT Containers Collector : version 5f1b5a2 starting
D, [2024-04-24T22:41:46.848458 #10] DEBUG -- : Ensuring cache directory is present: /tmp/cache
D, [2024-04-24T22:41:46.862460 #10] DEBUG -- : Ensuring metrics cache directory is present: /tmp/metrics_cache
D, [2024-04-24T22:41:46.935558 #10] DEBUG -- : Fetching collector agent version
D, [2024-04-24T22:41:46.943248 #10] DEBUG -- : Kubernetes collector agent version found : 1398
D, [2024-04-24T22:41:47.028912 #10] DEBUG -- : loaded K8S config from  with master @ https://172.22.0.1:443/ with ca certificate /var/run/secrets/kubernetes.io/serviceaccount/ca.crt with client_cert_file  with client key file  with trust_certs false with trust store file  with proxy username 
D, [2024-04-24T22:41:47.635460 #10] DEBUG -- : Fetching Cluster UID
D, [2024-04-24T22:41:47.640977 #10] DEBUG -- : Connecting to URL: https://172.22.0.1:443/api/v1/namespaces/kube-system
E, [2024-04-24T22:41:48.137155 #10] ERROR -- : [Java::JavaIo::IOException]: Server returned HTTP response code: 403 for URL: https://172.22.0.1:443/api/v1/namespaces/kube-system
E, [2024-04-24T22:41:48.138684 #10] ERROR -- : Backtrace:
    jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:77)
    jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)
    java.lang.reflect.Constructor.newInstanceWithCaller(java/lang/reflect/Constructor.java:499)
    java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:480)
    sun.net.www.protocol.http.HttpURLConnection$10.run(sun/net/www/protocol/http/HttpURLConnection.java:2057)
    sun.net.www.protocol.http.HttpURLConnection$10.run(sun/net/www/protocol/http/HttpURLConnection.java:2052)
    java.security.AccessController.doPrivileged(java/security/AccessController.java:569)
    sun.net.www.protocol.http.HttpURLConnection.getChainedException(sun/net/www/protocol/http/HttpURLConnection.java:2051)
    sun.net.www.protocol.http.HttpURLConnection.getInputStream0(sun/net/www/protocol/http/HttpURLConnection.java:1609)
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(sun/net/www/protocol/http/HttpURLConnection.java:1589)
    sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(sun/net/www/protocol/https/HttpsURLConnectionImpl.java:224)
    jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)
    jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)
    org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:329)
    org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:193)
    RUBY.get_input_stream(uri:classloader:/mesos-collector/lib/common/fetcher.rb:37)
    RUBY.fetch_uid(uri:classloader:/mesos-collector/lib/common/fetcher.rb:50)
    RUBY.do_fetch_cluster_uid(uri:classloader:/mesos-collector/lib/common/collection_cycle.rb:104)
    RUBY.do_upload(uri:classloader:/mesos-collector/lib/common/collection_cycle.rb:24)
    RUBY.upload_k8s_state_v4(uri:classloader:/mesos-collector/bin/container-collector.rb:88)
    org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:522)
    org.jruby.RubyBasicObject.send(org/jruby/RubyBasicObject.java:1687)
    org.jruby.RubyKernel.send(org/jruby/RubyKernel.java:2315)
    org.jruby.RubyKernel$INVOKER$s$send.call(org/jruby/RubyKernel$INVOKER$s$send.gen)
    RUBY.delegate_runner_methods(uri:classloader:/gems/boson-1.3.0/lib/boson/runner_library.rb:12)
    org.jruby.RubyProc.call(org/jruby/RubyProc.java:378)
    org.jruby.RubyMethod.call(org/jruby/RubyMethod.java:144)
    org.jruby.RubyMethod$INVOKER$i$call.call(org/jruby/RubyMethod$INVOKER$i$call.gen)
    RUBY.redefine_command_block(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:87)
    org.jruby.RubyProc.call(org/jruby/RubyProc.java:333)
    org.jruby.RubyProc$INVOKER$i$call.call(org/jruby/RubyProc$INVOKER$i$call.gen)
    RUBY.call_original_command(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:123)
    RUBY.during_analyze(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:115)
    RUBY.analyze(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:107)
    RUBY.redefine_command_block(uri:classloader:/gems/boson-1.3.0/lib/boson/scientist.rb:86)
    org.jruby.RubyProc.call(org/jruby/RubyProc.java:378)
    org.jruby.RubyClass.finvokeWithRefinements(org/jruby/RubyClass.java:522)
    org.jruby.RubyBasicObject.send(org/jruby/RubyBasicObject.java:1687)
    org.jruby.RubyKernel.send(org/jruby/RubyKernel.java:2315)
    org.jruby.RubyKernel$INVOKER$s$send.call(org/jruby/RubyKernel$INVOKER$s$send.gen)
    RUBY.full_invoke(uri:classloader:/gems/boson-1.3.0/lib/boson.rb:79)
    RUBY.execute_command(uri:classloader:/gems/boson-1.3.0/lib/boson/bare_runner.rb:42)
    RUBY.execute_command(uri:classloader:/gems/boson-1.3.0/lib/boson/runner.rb:35)
    RUBY.execute(uri:classloader:/gems/boson-1.3.0/lib/boson/runner.rb:31)
    RUBY.start(uri:classloader:/gems/boson-1.3.0/lib/boson/runner.rb:26)
    RUBY.<main>(uri:classloader:/mesos-collector/bin/container-collector.rb:120)
    org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1290)
    org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2964)
    org.jruby.RubyKernel.loadCommon(org/jruby/RubyKernel.java:1121)
    org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1091)
    org.jruby.RubyKernel$INVOKER$s$load.call(org/jruby/RubyKernel$INVOKER$s$load.gen)
    RUBY.<main>(uri:classloader:/META-INF/main.rb:1)
    org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1290)
    org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2964)
    org.jruby.RubyKernel.requireCommon(org/jruby/RubyKernel.java:1064)
    org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1057)
    org.jruby.RubyKernel$INVOKER$s$1$0$require.call(org/jruby/RubyKernel$INVOKER$s$1$0$require.gen)
    RUBY.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85)
    RUBY.<main>(<script>:3)
    org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1312)
    org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:1316)
    org.jruby.embed.internal.EmbedEvalUnitImpl.run(org/jruby/embed/internal/EmbedEvalUnitImpl.java:119)
    org.jruby.embed.ScriptingContainer.runUnit(org/jruby/embed/ScriptingContainer.java:1296)
    org.jruby.embed.ScriptingContainer.runScriptlet(org/jruby/embed/ScriptingContainer.java:1289)
    jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)
    jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)
    JarMain.invokeMethod(JarMain.java:263)
    JarMain.invokeMethod(JarMain.java:256)
    JarMain.launchJRuby(JarMain.java:141)
    JarMain.start(JarMain.java:158)
    JarMain.doStart(JarMain.java:233)
    JarMain.main(JarMain.java:227)
E, [2024-04-24T22:41:48.139484 #10] ERROR -- : Could not open an input stream to cluster's Kubernetes API: Server returned HTTP response code: 403 for URL: https://172.22.0.1:443/api/v1/namespaces/kube-system

I've validated that the service account and bindings were made as expected in the chart

kscherme commented 1 month ago

@bn-cforkner Hi! From the line Kubernetes collector agent version found : 1398, it looks like you may be using an old version of the helm chart. Can you double check that you are using helm chart https://github.com/CloudHealth/helm/releases/tag/cloudhealth-collector-4.6.2?