leezhihui
commented
3 months ago Hello @thomasdraebing, Could you help take a look what happened, why it will report class not found, even though I install with helm.
Open leezhihui opened 3 months ago
leezhihui
commented
3 months ago Hello @thomasdraebing, Could you help take a look what happened, why it will report class not found, even though I install with helm.
thomasdraebing
commented
3 months ago @leezhihui First of all, for questions regarding k8s-gerrit, please use the Gerrit mailing list. The GitHub repo is just a mirror and is not actively monitored.
What helm chart are you trying to install? From where do you pull the container images? Which Gerrit version do you use? Which k8s-gerrit version (commit ID) do you use? What is your configuration in the values.yaml (especially which plugins and how does your gerrit.config look like)? What are the logs of the gerrit-init container?
Best Regards, Thomas
leezhihui
commented
3 months ago Hi @thomasdraebing, thank you for your reply. 1 gerrit under /helm-charts/gerrit 2 install it on GKE 3 the latest i think, I directly pull the main branch code to local and deploy with helm 4 this it the valule file
gerrit:
images:
gerritInit: k8sgerrit/gerrit-init
gerrit: gerritcodereview/gerrit
tolerations: []
topologySpreadConstraints: {}
nodeSelector: {}
# kubernetes.io/arch: arm64
affinity: {}
additionalAnnotations: {}
additionalPodLabels: {}
replicas: 1
updatePartition: 0
# The memory limit has to be higher than the configures heap-size for Java!
resources:
requests:
cpu: 300m
memory: 8Gi
limits:
cpu: 300m
memory: 8Gi
persistence:
enabled: true
size: 10Gi
# If no value for probeScheme, the probe will use the default HTTP
probeScheme: HTTP
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 5
readinessProbe:
initialDelaySeconds: 5
periodSeconds: 1
startupProbe:
initialDelaySeconds: 10
periodSeconds: 30
gracefulStopTimeout: 90
service:
additionalAnnotations: {}
loadBalancerSourceRanges: []
type: NodePort
externalTrafficPolicy: Cluster
http:
port: 80
ssh:
enabled: true
port: 29418
priorityClassName:
etc:
# Some values are expected to have a specific value for the deployment installed
# by this chart to work. These are marked with `# FIXED`.
# Do not change them!
config:
gerrit.config: |-
[gerrit]
basePath = git # FIXED
serverId = gerrit-1
# The canonical web URL has to be set to the Ingress host, if an Ingress
# is used. If a LoadBalancer-service is used, this should be set to the
# LoadBalancer's external IP. This can only be done manually after installing
# the chart, when you know the external IP the LoadBalancer got from the
# cluster.
canonicalWebUrl =
disableReverseDnsLookup = true
[index]
type = LUCENE
[auth]
type = DEVELOPMENT_BECOME_ANY_ACCOUNT
[httpd]
# If using an ingress use proxy-http or proxy-https
listenUrl = proxy-https://*:8080/
requestLog = true
gracefulStopTimeout = 1m
[sshd]
listenAddress = off
[transfer]
timeout = 120 s
[user]
name = phoenix
email = phoenix-ci@arm.com
anonymousCoward = Unnamed User
[cache]
directory = cache
[container]
user = gerrit # FIXED
javaHome = /usr/lib/jvm/java-17-openjdk # FIXED
javaOptions = -Djavax.net.ssl.trustStore=/var/gerrit/etc/keystore # FIXED
javaOptions = -Xms200m
# Has to be lower than 'gerrit.resources.limits.memory'. Also
# consider memories used by other applications in the container.
javaOptions = -Xmx4g
replication.config: |-
[gerrit]
autoReload = false
replicateOnStartup = true
defaultForceUpdate = true
kubectl get all -n gerrit
NAME READY STATUS RESTARTS AGE
pod/gerrit-gerrit-stateful-set-0 0/1 CrashLoopBackOff 5 (47s ago) 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/gerrit-gerrit-service NodePort <none> 80:30616/TCP,29418:30785/TCP 10m
NAME READY AGE
statefulset.apps/gerrit-gerrit-stateful-set 0/1 10m
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/gerrit-git-gc 0 6,18 * * * False 0 <none> 10mFor the mail list, I need login with another account, organise account are restrict and can not see the issue there.
thomasdraebing
commented
3 months ago You are using the gerritcodereview/gerrit image for the main Gerrit container. That won't work. First of all the entrypoint script is not made to work with the k8sgerrit setup and more importantly, the latest version of that image is based on a newer Gerrit version, which is likely the cause for your issue. Please use k8sgerrit/gerrit
leezhihui
commented
3 months ago Hi @thomasdraebing Has changed to k8sgerrit/gerrit as the image, the new error looks this.
Defaulted container "gerrit" out of: gerrit, gerrit-init (init)
Jun 12, 2024 7:57:29 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
WARNING: AssistedInject factory com.google.gerrit.server.api.changes.ChangeApiImpl$Factory will be slow because class com.google.gerrit.server.api.changes.ChangeApiImpl has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!) The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.server.api.changes.ChangeApiImpl.<init>()[48]]
[2024-06-12 07:57:38,966] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'WorkQueue' queue
[2024-06-12 07:57:39,071] [main] INFO com.google.gerrit.server.cache.PersistentCacheBaseFactory : Enabling disk cache /var/gerrit/cache
[2024-06-12 07:57:40,268] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Interactive' queue
[2024-06-12 07:57:40,269] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Batch' queue
[2024-06-12 07:57:40,569] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'ReceiveCommits' queue
[2024-06-12 07:57:40,570] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SendEmail' queue
[2024-06-12 07:57:48,870] [main] INFO com.google.gerrit.server.rules.prolog.PrologEnvironment : reductionLimit: 100000, compileLimit: 1000000
[2024-06-12 07:57:48,873] [main] INFO com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "gc".
[2024-06-12 07:57:48,874] [main] INFO com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "changeCleanup".
[2024-06-12 07:57:48,874] [main] INFO com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "attentionSet".
[2024-06-12T07:57:49.070Z] [main] WARN com.google.gerrit.server.config.GitwebCgiConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)
[2024-06-12T07:57:50.973Z] [main] INFO org.eclipse.jetty.util.log : Logging initialized @32420ms to org.eclipse.jetty.util.log.Slf4jLog
[2024-06-12T07:57:51.772Z] [main] INFO com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /var/gerrit/etc/jgit.config
[2024-06-12T07:57:51.865Z] [main] INFO com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 990m
[2024-06-12T07:57:51.868Z] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Failed to read NoteDb schema version
1 error
at com.google.gerrit.server.schema.NoteDbSchemaVersionCheck.start(NoteDbSchemaVersionCheck.java:90)
at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
at com.google.gerrit.pgm.Daemon.start(Daemon.java:404)
at com.google.gerrit.pgm.Daemon.run(Daemon.java:297)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:252)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:148)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:93)
at Main.main(Main.java:30)
Caused by: com.google.gerrit.exceptions.StorageException: Failed to read refs/meta/version
at com.google.gerrit.server.schema.NoteDbSchemaVersionManager.read(NoteDbSchemaVersionManager.java:52)
at com.google.gerrit.server.schema.NoteDbSchemaVersionCheck.start(NoteDbSchemaVersionCheck.java:57)
... 12 more
Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository All-Projects
at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:186)
at com.google.gerrit.server.schema.NoteDbSchemaVersionManager.read(NoteDbSchemaVersionManager.java:49)
... 13 more
Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /var/mnt/git/All-Projects
at com.google.gerrit.server.git.DynamicRefDbRepository$FileKey.open(DynamicRefDbRepository.java:55)
at org.eclipse.jgit.lib.RepositoryCache.openRepository(RepositoryCache.java:245)
at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:88)
at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:61)
at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:182)
... 14 more
Hi @leezhihui ,
the issue is that the Gerrit version in the previous image uses a newer index schema than the one in k8sgerrit/gerrit. A schema downgrade is not supported. SInce the site was never actively used, the easiest way would be to uninstall the chart, delete the persistent volumes and install again
leezhihui
commented
3 months ago looks ready now, Thank you @thomasdraebing
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
[2024-06-12T08:10:01.769Z] [main] INFO com.google.gerrit.server.plugins.PluginLoader : Loaded plugin healthcheck (w/ ApiModule), version v3.5.6-20-g2432849168
[2024-06-12T08:10:01.863Z] [main] INFO com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "accountDeactivation".
[2024-06-12T08:10:01.867Z] [main] INFO org.eclipse.jetty.server.Server : jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.11+9-alpine-r0
[2024-06-12T08:10:02.072Z] [main] INFO org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0
[2024-06-12T08:10:02.163Z] [main] INFO org.eclipse.jetty.server.session : No SessionScavenger set, using defaults
[2024-06-12T08:10:02.165Z] [main] INFO org.eclipse.jetty.server.session : node0 Scavenging every 660000ms
[2024-06-12T08:10:03.069Z] [main] INFO org.eclipse.jetty.server.handler.ContextHandler : Started o.e.j.s.ServletContextHandler@221383c9{/,null,AVAILABLE}
[2024-06-12T08:10:03.363Z] [main] INFO org.eclipse.jetty.server.AbstractConnector : Started ServerConnector@5eba0cc5{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
[2024-06-12T08:10:03.364Z] [main] INFO org.eclipse.jetty.server.Server : Started @41495ms
[2024-06-12T08:10:03.366Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.9.4 ready
[2024-06-12T08:10:14.171Z] [HTTP GET /config/server/healthcheck~status (N/A from 100.98.7.1)] INFO com.googlesource.gerrit.plugins.healthcheck.check.BlockedThreadsConfigurator : Default blocked threads check is configured with 50% thresholdHi @thomasdraebing, the UI looks like this after I create a repo backstage-gerrit, it lacks ssh clone url and https clone URL, so it means lacks the plugins realated? ![Uploading Screenshot 2024-06-12 at 17.14.56.png…]()
leezhihui
commented
3 months ago Hi @thomasdraebing when I enable the sshd from off to :29418 it will report the below error, if I can directly configure here:
# `gerrit.keystore` expects a base64-encoded Java-keystore
# Since Java keystores are binary files, adding the unencoded content and
# automatic encoding using helm does not work here.
keystore: [httpd]
# If using an ingress use proxy-http or proxy-https
listenUrl = proxy-https://*:8080/
requestLog = true
gracefulStopTimeout = 1m
[sshd]
**listenAddress = *:29418**[2024-06-13 01:33:58,966] [main] INFO com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "attentionSet".
Jun 13, 2024 1:33:59 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
WARNING: AssistedInject factory com.google.gerrit.sshd.DispatchCommand$Factory will be slow because class com.google.gerrit.sshd.DispatchCommand has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!) The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.sshd.BaseCommand.injector]
[2024-06-13T01:34:01.375Z] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SshCommandStart' queue
[2024-06-13T01:34:01.463Z] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Stream-Worker' queue
[2024-06-13T01:34:01.465Z] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Interactive-Worker' queue
[2024-06-13T01:34:01.466Z] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Batch-Worker' queue
[2024-06-13T01:34:01.467Z] [main] INFO com.google.gerrit.server.git.WorkQueue : Adding metrics for 'MigrateExternalIdCase' queue
[2024-06-13T01:34:01.469Z] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
com.google.inject.CreationException: Unable to create injector, see the following errors:
1) No SSH keys under /var/gerrit/etc
while locating HostKeyProvider
at SshHostKeyModule.configure(SshHostKeyModule.java:25)
at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
\_ for 6th parameter hostKeyProvider
at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
\_ for 1st parameter authenticator
at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
while locating CachingPublicKeyAuthenticator
at SshDaemon.<init>(SshDaemon.java:168)
\_ for 3rd parameter userAuth
at SshDaemon.class(SshDaemon.java:140)
while locating SshDaemon
at SshModule.configure(SshModule.java:76)
while locating SshInfo
2) No SSH keys under /var/gerrit/etc
while locating HostKeyProvider
at SshHostKeyModule.configure(SshHostKeyModule.java:25)
at DatabasePubKeyAuth.<init>(DatabasePubKeyAuth.java:75)
\_ for 6th parameter hostKeyProvider
at CachingPublicKeyAuthenticator.<init>(CachingPublicKeyAuthenticator.java:26)
\_ for 1st parameter authenticator
at CachingPublicKeyAuthenticator.class(CachingPublicKeyAuthenticator.java:26)
while locating CachingPublicKeyAuthenticator
while locating PublickeyAuthenticatorI use
keytool -keystore keystore -alias jetty -genkey -keyalg RSA generate a file named keystore and then use
base64 keystore -w 0 > keystore.base64 to write it to keystore.base64,
and then copy the content in keystore.base64 and paste as the value for gerrit.keystore. but still got the error No SSH keys under /var/gerrit/etc/
msohn
commented
3 months ago Please use the Gerrit issue tracker for reporting issues about k8s-gerrit. See https://www.gerritcodereview.com/issues.html Open issues for k8s-gerrit here https://issues.gerritcodereview.com/issues/new?component=1432071&template=0
leezhihui
commented
3 months ago HI @msohn, just created one https://issues.gerritcodereview.com/issues/346898507
Could you help explain more about this field in the value.yaml
# `gerrit.keystore` expects a base64-encoded Java-keystore
# Since Java keystores are binary files, adding the unencoded content and
# automatic encoding using helm does not work here.
keystore: the keystore, I put content with base64 encoded, but it got error.
thomasdraebing
commented
2 months ago @leezhihui What you are missing are the SSH keys. That is unrelated to the keystore, which provides SSL keypairs. By default Gerrit creates its own SSH keys. However, that does not work in Kubernetes with scaled Gerrit instances, since then every instance would have a different key and clients would not be able to reliably verify the server's identity. Thus, you will have to create the keys yourself and provide them under .Values.etc.secret
leezhihui
commented
2 months ago Hi @thomasdraebing, so, you mean leave the keystore blank, and generate key pair and put private key and public key in the ssh_host_ecdsa_key: and ssh_host_ecdsa_key.pub: ecdsa-sha2-nistp256...
# automatic encoding using helm does not work here.
keystore:At that moment the secret looks like this:
secret:
secure.config: |-
# Password for the keystore added as value for 'gerritReplica.keystore'
# Only needed, if SSL is enabled.
[httpd]
sslKeyPassword = gerrit
# Credentials for replication targets
# [remote "replica"]
# username = git
# password = secret
# ssh_host_ecdsa_key: |-
# -----BEGIN EC PRIVATE KEY-----
# -----END EC PRIVATE KEY-----
# ssh_host_ecdsa_key.pub: ecdsa-sha2-nistp256...Since SSL is terminated at the load balancer and not by Gerrit itself, the keystore is only used, if Gerrit needs to provide or check a certificate when communicating with other services, e.g. an LDAP server. Thus, you might need it.
Regarding the SSH keys, you can add any key value pair under etc.secret and it will be mounted as a file in the etc directory. Gerrit expects the name to be of the format ssh_host_<algorithm>_key. An easy way to obtain them, is to run gerrit once locally, let it generate the keys and add them to the configuration.
leezhihui
commented
2 months ago @thomasdraebing, Oh, I see, but when LDAP server communicating with gerrit, if it need through the load balancer? because I think the gerrit is always behind the load balancer.
Why it need to run the gerrit locally to get the keys? usually we can use ssh-keygen command on ubuntu to generate keypairs, and then paste the private key and public key to the location.
Another question, when we use gerrit, we usually update ssh public key in the User setting part. Here why we need first put it as part of gerrit deployment vaule?
msohn
commented
2 months ago Depends on where your LDAP server runs.
You don't need to run gerrit to generate the ssh host keys, but you can, and that's a convenient way to create them. If you prefer ssh-keygen you can use that instead.
I think you are confusing public ssh keys of clients used by end users and the host keys of the gerrit deployment.
when install gerrit using helm it report the class not found error