Gravitee API and Gravitee Gateway suffers from vert.x dns bug that prevent correct working on a K8S cluster.

[chicco785]

Expected Behavior

When an endpoint for an API is registered, Gravitee should be able to resolve it.

Current Behavior

DNS resolution fails, because netty version used by vert.x 3.5 does not support the advanced dns configuration created by k8s in resolv.conf on container deployment, as documented here: https://github.com/eclipse/vert.x/issues/2031

Possible Solution

Include the following option in the bash scripts launching gravitee, to disable vert.x dns resolution and use the operating system one:

        -Dvertx.disableDnsResolver=true \

full code for the gateway (for api manager, you need to apply the same change):

#!/bin/sh

DIRNAME=`dirname $0`
PROGNAME=`basename $0`

# OS specific support (must be 'true' or 'false').
cygwin=false;
darwin=false;
linux=false;
case "`uname`" in
    CYGWIN*)
        cygwin=true
        ;;

    Darwin*)
        darwin=true
        ;;

    Linux)
        linux=true
        ;;
esac

# Force IPv4 on Linux systems since IPv6 doesn't work correctly with jdk5 and lower
if [ "$linux" = "true" ]; then
   JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
fi

# Searching for configuration 
GRAVITEE_OPTS=""
if [ -f "/etc/gravitee-gateway/gravitee.yml" ]
then
    GRAVITEE_OPTS="-Dgravitee.conf=/etc/gravitee-gateway/gravitee.yml"
fi

# Setup GRAVITEE_HOME
if [ "x$GRAVITEE_HOME" = "x" ]; then
    # get the full path (without any relative bits)
    GRAVITEE_HOME=`cd $DIRNAME/..; pwd -P`
fi

export GRAVITEE_HOME

# Move to the context home
cd $GRAVITEE_HOME

export JAVA_OPTS

# Setup the JVM
if [ "x$JAVA" = "x" ]; then
    if [ "x$JAVA_HOME" != "x" ]; then
    JAVA="$JAVA_HOME/bin/java"
    else
    JAVA="java"
    fi
fi

# Setup the classpath
runjar=`find $GRAVITEE_HOME -name "gravitee-gateway-standalone-bootstrap*.jar"`
if [ ! -f "$runjar" ]; then
    die "Missing required file: $runjar"
fi
GRAVITEE_BOOT_CLASSPATH="$runjar"

if [ "x$GIO_MIN_MEM" = "x" ]; then
    GIO_MIN_MEM=512m
fi
if [ "x$GIO_MAX_MEM" = "x" ]; then
    GIO_MAX_MEM=512m
fi

# min and max heap sizes should be set to the same value to avoid
# stop-the-world GC pauses during resize
JAVA_OPTS="$JAVA_OPTS -Xms${GIO_MIN_MEM}"
JAVA_OPTS="$JAVA_OPTS -Xmx${GIO_MAX_MEM}"

# set to headless, just in case
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

# Force the JVM to use IPv4 stack
if [ "x$GIO_USE_IPV4" != "x" ]; then
  JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
fi

# Causes the JVM to dump its heap on OutOfMemory.
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
# The path to the heap dump location, note directory must exists and have enough
# space for a full heap dump.
#JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=$GRAVITEE_HOME/logs/heapdump.hprof"

# Disables explicit GC
JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC"

# Ensure UTF-8 encoding by default (e.g. filenames)
JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF-8"

# Convert paths for Java on Windows.
if $cygwin; then
    GRAVITEE_BOOT_CLASSPATH=$(cygpath -w $GRAVITEE_BOOT_CLASSPATH)
    GRAVITEE_HOME=$(cygpath -w $GRAVITEE_HOME)
fi

# Display our environment
echo "========================================================================="
echo ""
echo "  Gravitee.IO Standalone Runtime Bootstrap Environment"
echo ""
echo "  GRAVITEE_HOME: $GRAVITEE_HOME"
echo ""
echo "  GRAVITEE_OPTS: $GRAVITEE_OPTS"
echo ""
echo "  JAVA: $JAVA"
echo ""
echo "  JAVA_OPTS: $JAVA_OPTS"
echo ""
echo "  CLASSPATH: $GRAVITEE_BOOT_CLASSPATH"
echo ""
echo "========================================================================="
echo ""

# Execute the JVM in the foreground
daemon=`echo $* | egrep -- '(^-d |-d$| -d |--daemon$|--daemon )'`
if [ -z "$daemon" ] ; then
    exec "$JAVA" $JAVA_OPTS \
        -javaagent:$GRAVITEE_HOME/boot/jetty-alpn-agent-2.0.6.jar \
        -Dvertx.disableFileCaching=true \
        -Dvertx.disableFileCPResolving=true \
        -Dvertx.disableContextTiming=true \
        -Dvertx.disableWebsockets=true \
        -Dvertx.disableDnsResolver=true \
        -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory \
        -cp "$GRAVITEE_BOOT_CLASSPATH" \
        $GRAVITEE_OPTS \
        io.gravitee.gateway.standalone.boostrap.Bootstrap \
        "$@"
else
    while [ $# -gt 0 ]; do
        case "$1" in
            -p=*)
                pid_file="${1#*=}"
                ;;
        esac
        shift
    done

    if [ -z "$pid_file" ] ; then
        pid_file=/var/run/graviteeio-apim-gw.pid
    fi

    exec "$JAVA" $JAVA_OPTS \
        -javaagent:$GRAVITEE_HOME/boot/jetty-alpn-agent-2.0.6.jar \
        -Dvertx.disableFileCaching=true \
        -Dvertx.disableFileCPResolving=true \
        -Dvertx.disableContextTiming=true \
        -Dvertx.disableWebsockets=true \
        -Dvertx.disableDnsResolver=true \
        -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory \
        -cp "$GRAVITEE_BOOT_CLASSPATH" \
        $GRAVITEE_OPTS \
        io.gravitee.gateway.standalone.boostrap.Bootstrap \
        "$@" <&- &

    retval=$?
    pid=$!

    [ $retval -eq 0 ] || exit $retval
    if ! ps -p $pid > /dev/null ; then
        exit 1
    fi

    echo $pid > $pid_file
    exit 0
fi

exit $?

Steps to Reproduce (for bugs)

I can provide an example chart. But the solution (tested) is already above. So, I am not sure it is needed (and I am bit in overloaded to package the code). Still, if relevant for the community, I can put later on a pull request with the helm chart somewhere (where?).

Context

Trying to run gravitee cluster on k8s.

Your Environment

• Operating System and version: K8S on Linux Debian

[brasseld]

Hi @chicco785,

This one is already a well-known bug from the Gravitee team, but thanks for sharing links. IMHO, if we have to apply such fix, it must be done for a certain period only (I did not have a look to Netty roadmap), to wait for the bug to be fixed by the Netty team. So we can apply the change for next version (1.13) but it will be removed in a near future to move again to Netty DNS resolver.

Also, you talk about k8s. We are thinking about providing all required descriptors to support Gravitee.io on k!s. Perhaps you can help us and share some parts with us ? Your help will be greatly appreciated!

Thanks a lot.

[chicco785]

HI @brasseld,

I have a sort of helm chart (some aspects may be custom, due to the need for example to have a custom image to solve the dns issue).

I am happy to share it. Where and how?

Thx, Federico

[brasseld]

Hi @chicco785

There are already some work around Gravitee.io and Openshift support: https://github.com/barkbay/openshift-ansible-gravitee I think the best to start would be to create your own github repository to push your stuff and then we will exchange together about next steps.

WDYT ?

[chicco785]

Fair enough, given that the official way of “packaging” for Kubernetes is helm charts, I will fork the official kubernetes charts repo and add my gravitee templates under incubator. Then you can review them, and once you think they are good, we can make a pull request to the official repo.

Is that good?

Il giorno 2 feb 2018, alle ore 18:17, Brassely David notifications@github.com ha scritto:

Hi @chicco785

There are already some work around Gravitee.io and Openshift support: https://github.com/barkbay/openshift-ansible-gravitee I think the best to start would be to create your own github repository to push your stuff and then we will exchange together about next steps.

WDYT ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[brasseld]

LGTM, perfect!

[brasseld]

Upgrading Vert.x to 3.5.1 fix the issue.

Will be done in 1.14.x : https://github.com/gravitee-io/issues/issues/1092

[brasseld]

Hello @chicco785

What about k8s support ?

Cheers

[PierrickLozach]

Hi @chicco785 , I am interested in your helm chart to deploy gravitee on kubernetes. Would you be ok to share it with me? Have you created a repo maybe? I could not find anything under your profile.

[chicco785]

hi pierrick

due to a project requirement we are using gitlab for it. my plan was to clean it up and make it a proper chart. if you are in a rush, i will just copy it somewhere.

On 22 June 2018 at 16:54, Pierrick Lozach notifications@github.com wrote:

Hi @chicco785 https://github.com/chicco785 , I am interested in your helm chart to deploy gravitee on kubernetes. Would you be ok to share it with me? Have you created a repo maybe? I could not find anything under your profile.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gravitee-io/issues/issues/1040#issuecomment-399470297, or mute the thread https://github.com/notifications/unsubscribe-auth/AAvcAk_xddYey1kygsR-pIdZNBGs_wqFks5t_QUlgaJpZM4RufXQ .

[PierrickLozach]

Thanks @chicco785 . I would definitely appreciate if you could send me a copy. I can help you clean it up too.

[brasseld]

@chicco785 can you push the code / conf files into a dedicated branch in the new repository gravitee-kubernetes ? I will ask the other guys to do so, it will be a good starting point to merge all of them and make a proper chart.

thanks a lot.

[chicco785]

as mentioned in the other ticket, it seems I don't have access to any action on the repo.

On 26 June 2018 at 12:58, Brassely David notifications@github.com wrote:

@chicco785 https://github.com/chicco785 can you push the code / conf files into a dedicated branch in the new repository gravitee-kubernetes ? I will ask the other guys to do so, it will be a good starting point to merge all of them and make a proper chart.

thanks a lot.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gravitee-io/issues/issues/1040#issuecomment-400251772, or mute the thread https://github.com/notifications/unsubscribe-auth/AAvcAjHSPq8gOGqPpQ8y9S3PgoJUT3a3ks5uAgXFgaJpZM4RufXQ .

[NicolasGeraud]

I've just pushed a first commit.

Tell me if it's working.

[brasseld]

May I close this issue to avoid duplicates on the same subject ?

[chicco785]

(y)

On 26 June 2018 at 16:14, Brassely David notifications@github.com wrote:

May I close this issue to avoid duplicates on the same subject ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gravitee-io/issues/issues/1040#issuecomment-400302988, or mute the thread https://github.com/notifications/unsubscribe-auth/AAvcAm25veX881a48pePOdf5Rr-02hsPks5uAjOdgaJpZM4RufXQ .