artemiscloud / activemq-artemis-operator

Apache License 2.0
66 stars 62 forks source link

activemq-artemis-broker-init 1.0.27 generates bootstrap.xml that throws exception that prevents broker 2.34.0 from starting #955

Closed pshields2 closed 3 weeks ago

pshields2 commented 3 months ago

Describe the bug testing latest ActiveMQ Artemis v 2.34.0 in kubernetes. Using these docker images: quay.io/artemiscloud/activemq-artemis-operator 1.2.2 5e0cc862ad52 6 days ago 532MB quay.io/artemiscloud/activemq-artemis-broker-init 1.0.27 af2dc8181546 7 days ago 1.08GB quay.io/artemiscloud/activemq-artemis-broker-kubernetes 1.0.26 917b2cde2da0 7 days ago 893MB

I build a custom broker-init image to configure the broker. Which adds config parameters to the broker.xml and bootstrap.xml files. The init container runs my customizations with out any problems but when the container is started it reports an exception and the container is stopped/terminated. Here is the contents of the broker log from the failed container.

kubectl -n dvs logs cray-dvs-mqtt-ss-0 
Removing provided -XX:+UseParallelOldGC in favour of artemis.profile provided option
Platform is x86_64
Running server env: home: /home/jboss AMQ_HOME /opt/amq CONFIG_BROKER false RUN_BROKER 
NO RUN_BROKER defined
Using custom configuration. Copy from /amq/init/config to /home/jboss/amq-broker
bin
etc
lib
log
tmp
Running Broker in /home/jboss/amq-broker
The Prometheus plugin already configured.
Using default logging configuration(console only)
ERROR StatusLogger Reconfiguration failed: No configuration found for '5a07e868' at 'null' in 'null'
javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 38; columnNumber: 155; cvc-complex-type.3.2.2: Attribute 'sniHostCheck' is not allowed to appear in element 'binding'.]
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:453)
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:387)
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:356)
    at org.apache.activemq.artemis.dto.XmlUtil.decode(XmlUtil.java:114)
    at org.apache.activemq.artemis.cli.factory.xml.XmlBrokerFactoryHandler.createBroker(XmlBrokerFactoryHandler.java:35)
    at org.apache.activemq.artemis.cli.factory.BrokerFactory.createBrokerConfiguration(BrokerFactory.java:47)
    at org.apache.activemq.artemis.cli.factory.BrokerFactory.createBrokerConfiguration(BrokerFactory.java:54)
    at org.apache.activemq.artemis.cli.commands.Configurable.getBrokerDTO(Configurable.java:122)
    at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
    at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:212)
    at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:162)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:144)
    at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:61)
Caused by: org.xml.sax.SAXParseException; lineNumber: 38; columnNumber: 155; cvc-complex-type.3.2.2: Attribute 'sniHostCheck' is not allowed to appear in element 'binding'.
    at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
    at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:135)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:396)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
    at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:284)
    at java.xml/com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(XMLSchemaValidator.java:512)
    at java.xml/com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.reportSchemaError(XMLSchemaValidator.java:3600)
    at java.xml/com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.processAttributes(XMLSchemaValidator.java:2993)
    at java.xml/com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:2287)
    at java.xml/com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:830)
    at java.xml/com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorHandlerImpl.startElement(ValidatorHandlerImpl.java:571)
    at com.sun.xml.bind.v2.runtime.unmarshaller.ValidatingUnmarshaller.startElement(ValidatingUnmarshaller.java:71)
    at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:45)
    at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:216)
    at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:150)
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:385)
    ... 15 more

And the log from my running of my custom post-config.sh script in the broker-init container:

kubectl -n dvs logs cray-dvs-mqtt-ss-0 -c cray-dvs-mqtt-container-init
Removing provided -XX:+UseParallelOldGC in favour of artemis.profile provided option
Platform is x86_64
Running server env: home: /home/jboss AMQ_HOME /opt/amq CONFIG_BROKER  RUN_BROKER false
NO CONFIG_BROKER defined
Configuring Broker at /amq/init/config
config Using instanceDir: /home/jboss/amq-broker
Broker will be clustered
Setting journal type to nio
Creating Broker with args --silent --role admin --name amq-broker --http-host cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local --java-options=-Djava.net.preferIPv4Stack=true  --user XXXXX --password XXXXX  --allow-anonymous --data /opt/cray-dvs-mqtt/data --no-amqp-acceptor --no-hornetq-acceptor --no-mqtt-acceptor --no-stomp-acceptor --no-autotune --clustered --cluster-user XXXXX --cluster-password XXXXX --host cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local --nio at /home/jboss/amq-broker
Creating ActiveMQ Artemis instance at: /home/jboss/amq-broker

You can now start the broker by executing:  

   "/home/jboss/amq-broker/bin/artemis" run

Or you can run the broker in the background using:

   "/home/jboss/amq-broker/bin/artemis-service" start

Checking yacfg file under dir: 
Generating jgroups-ping.xml, current dir is: /tmp/remote_source/app, AMQHOME: /opt/amq
APPLICATION_NAME is not set
Setting redistribution-delay to zero.
Using acceptors from environment and removing existing entries
Removing hardcoded -Xms -Xmx from artemis.profile in favour of JAVA_OPTS in log above
Configure logging
Enable artemis metrics plugin
Adding artemis metrics plugin
Copying Config files from S2I build
'/opt/amq/conf/jgroups-ping.xml' -> '/home/jboss/amq-broker/etc/jgroups-ping.xml'
'/opt/amq/conf/log4j2.properties' -> '/home/jboss/amq-broker/etc/log4j2.properties'
Custom Configuration file 'BROKER_XML' is disabled
Custom Configuration file 'LOG4J2_PROPERTIES' is disabled
user defined CONFIG_INSTANCE_DIR, copying
amq-broker
exposing env var CONFIG_INSTANCE_DIR for custom init
CONFIG_INSTANCE_DIR value from /amq/init/config
Exported value of CONFIG_INSTANCE_DIR: /amq/init/config/amq-broker
Finding default custom script at /amq/scripts/post-config.sh
Found custom script /amq/scripts/post-config.sh, executing it
/amq/scripts/post-config.sh
post-config.sh: Start
post-config.sh: JWKS_SPIRE_URLS: http://spire-jwks.spire/keys
post-config.sh: JWKS_AUDIENCE: system-compute
post-config.sh: JWKS_ISSUERS: http://spire.local/shasta/vshastaio
post-config.sh: JWKS_CLOCK_SKEW_SECONDS: 60
post-config.sh: JWKS_SPIRE_CONTACT_RATE: 60
post-config.sh: JWT_XNAME_POLICY_ENABLED: true
post-config.sh: LOG4J2_PROPERTIES: 
post-config.sh: ACTIVEMQ_LOGGING: 
post-config.sh: Config dir: /amq/init/config/amq-broker
post-config.sh: Copy jars
'/cray/lib/hpc-activemq-artemis-security-manager-0.0.1.jar' -> '/amq/init/config/amq-broker/lib/hpc-activemq-artemis-security-manager-0.0.1.jar'
'/cray/lib/jose4j-0.9.3.jar' -> '/amq/init/config/amq-broker/lib/jose4j-0.9.3.jar'
post-config.sh: Add properties to bootstrap.xml
post-config.sh: Replace <jaas-security domain="activemq"/> in /amq/init/config/amq-broker/etc/bootstrap.xml
post-config.sh: Configuring HA for cray-dvs-mqtt-ss-0
post-config.sh: Configuring storage directories for cray-dvs-mqtt-ss-0
post-config.sh: contents of: /cray/etc/bootstrap-security-manager.xml
------------------------
   <security-manager class-name="com.hpe.hpc.activemq.JwtJaasSecurityManager">
      <property key="domain" value="activemq"/>
      <property key="jwks-spire-urls" value="http://spire-jwks.spire/keys"/>
      <property key="jwks-audience" value="system-compute"/>
      <property key="jwks-issuers" value="http://spire.local/shasta/vshastaio"/>
      <property key="jwks-clock-skew-seconds" value="60"/>
      <property key="jwks-spire-contact-rate" value="60"/>
      <property key="jwt-xname-policy-enabled" value="true"/>
   </security-manager>
------------------------
post-config.sh: contents of: /amq/init/config/amq-broker/etc/bootstrap.xml
------------------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!--
  ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements. See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to You under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License. You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<broker xmlns="http://activemq.apache.org/schema">

   <security-manager class-name="com.hpe.hpc.activemq.JwtJaasSecurityManager">
      <property key="domain" value="activemq"/>
      <property key="jwks-spire-urls" value="http://spire-jwks.spire/keys"/>
      <property key="jwks-audience" value="system-compute"/>
      <property key="jwks-issuers" value="http://spire.local/shasta/vshastaio"/>
      <property key="jwks-clock-skew-seconds" value="60"/>
      <property key="jwks-spire-contact-rate" value="60"/>
      <property key="jwt-xname-policy-enabled" value="true"/>
   </security-manager>

   <!-- artemis.URI.instance is parsed from artemis.instance by the CLI startup.
        This is to avoid situations where you could have spaces or special characters on this URI -->
   <server configuration="file:/home/jboss/amq-broker/etc//broker.xml"/>

   <!-- The web server is only bound to localhost by default -->
   <web customizer="org.eclipse.jetty.server.ForwardedRequestCustomizer" path="web" rootRedirectLocation="console">
       <binding sniHostCheck="false" sniRequired="false" name="artemis" uri="http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161">
           <app name="branding" url="activemq-branding" war="activemq-branding.war"/>
           <app name="plugin" url="artemis-plugin" war="artemis-plugin.war"/>
           <app name="console" url="console" war="console.war"/>
       <app url="metrics" war="metrics.war"/></binding>
   </web>

</broker>

------------------------
post-config.sh: contents of: /amq/init/config/amq-broker/etc/broker.xml
------------------------

=== OMITTED ===

------------------------
post-config.sh: SUCCESS

Note the contents of the bootstrap.xml in the output above. In particular these two lines are generating the exception.

   <web customizer="org.eclipse.jetty.server.ForwardedRequestCustomizer" path="web" rootRedirectLocation="console">
       <binding sniHostCheck="false" sniRequired="false" name="artemis" uri="http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161">

This is code provided from the activemq-artemis-broker-init imag. My post-config.sh adds the contents to the bootstrap.xml file.

I see from the Artemis documentation, https://activemq.apache.org/components/artemis/documentation/latest/web-server.html, that web syntax looks ok.

[!TIP] Vote this issue reacting with :+1: or :-1:

gtully commented 3 months ago

is your custom init container image rebased on the 1.0.27 init container. It looks like possibly there is an older xsd in play.

pshields2 commented 3 months ago

I thought it was but I'll double check.

gtully commented 3 months ago

This interdependence on the xml schema between init container and run container is one of the reasons for moving lots of the configuration to brokerProperties.

For the web container you are limited to system properties to augment the config, those can be in JAVA_ARGS_APPEND but it should be possible to configure your custom security manager via broker properties.

In short, I think you may be able to drop your init container to avoid this sort of problem into the future and avoid the need to manage a dependent container.

pshields2 commented 3 months ago

That will be a future exercise. I also need to add jar file to the broker for my security manager implementation. So I will still need the custom broker-init. For now I am trying to get the xsd in sync with the newer run container. But I don't see where it is getting out of sync. I verified my docker file. See bellow.

FROM arti.hpc.amslabs.hpecorp.net/docker-remote/maven:3.9.1-amazoncorretto-11 AS BUILDER
COPY security-manager /security-manager
RUN cd /security-manager; mvn clean install --settings settings.xml

FROM quay.io/artemiscloud/activemq-artemis-broker-init:1.0.27
USER root
RUN set -x \
    && mkdir -p /amq/scripts \
    && mkdir -p /cray/etc \
    && mkdir -p /cray/lib
RUN set -x \
    && chmod a+w /cray/etc
USER 185
COPY scripts/post-config.sh /amq/scripts
COPY config/bootstrap-security-manager.xml /cray/etc
COPY --from=builder /security-manager/target/hpc-activemq-artemis-security-manager-0.0.1.jar /cray/lib/
COPY --from=builder /security-manager/target/lib/jose4j-0.9.3.jar /cray/lib/

And I have loaded the images into my local repo for the custom build, so I am at a loss to where the old xsd creeps in?

gtully commented 3 months ago

There will be a need for an init container or volume mount to make your custom classes available, but that can isolated and version independent.

pshields2 commented 3 months ago

I am upgrading a system where an older version has been running. Could the older xsd be picked up somehow?

gtully commented 3 months ago

It seems so. The schema(s) are in the /opt/amq/schema/ directory, the relevant bindings type in activemq.xsd

brusdev commented 2 months ago

@pshields2 did you fix your issue?