search

reactiverse / es4x

🚀 fast JavaScript 4 Eclipse Vert.x
https://reactiverse.io/es4x/
Apache License 2.0
884 stars 75 forks source link

ES4X Docker requires GraalVM #433

Closed UglyHobbitFeet closed 4 years ago

UglyHobbitFeet commented 4 years ago

This is in continuation to this ticket: https://github.com/reactiverse/es4x/issues/425

ES4X seems to have GraalVM dependencies when using the dockerfile. If I change the dockerfile mentioned in the above issue to use a different JVM like this:

# Third stage (contain)
#FROM $BASEIMAGE
FROM maslick/minimalka:jdk11

I see tons of these errors in the console regarding graalvm dependency issues when running the docker. 


| Exception in thread "main" java.lang.NoClassDefFoundError: org/graalvm/polyglot/proxy/ProxyObject
|   at java.base/java.lang.ClassLoader.defineClass1(Native Method)
|   at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
|   at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
|   at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
|   at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
|   at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
|   at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
|   at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
|   at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
|   at io.vertx.core.impl.launcher.commands.RunCommand.run(RunCommand.java:245)
|   at io.vertx.core.impl.launcher.VertxCommandLauncher.execute(VertxCommandLauncher.java:248)
|   at io.vertx.core.impl.launcher.VertxCommandLauncher.dispatch(VertxCommandLauncher.java:402)
|   at io.vertx.core.impl.launcher.VertxCommandLauncher.dispatch(VertxCommandLauncher.java:346)
|   at io.reactiverse.es4x.ES4X.main(ES4X.java:74)
| Caused by: java.lang.ClassNotFoundException: org.graalvm.polyglot.proxy.ProxyObject
|   at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
|   at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
|   at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
|   ... 14 more```
UglyHobbitFeet commented 4 years ago

If I modify the code instead to use the same JVM for both steps:

ARG BASEIMAGE=maslick/minimalka:jdk11
...<same code until>...
FROM $BASEIMAGE AS JVM
RUN apk --no-cache add curl && apk --no-cache add bash
...<same code until end>...

It results in a: Cannot install GraalVM MBean due to Receiver class org.graalvm.compiler.hotspot.management.HotSpotGraalManagement does not define or inherit an implementation of the resolved method 'abstract void initialize(org.graalvm.compiler.hotspot.HotSpotGraalRuntime, org.graalvm.compiler.hotspot.GraalHotSpotVMConfig)' of interface org.graalvm.compiler.hotspot.HotSpotGraalManagementRegistration.

frank-dspeed commented 4 years ago

@UglyHobbitFeet it has a dependencie for graal(compiler) for speed and truffle-api, truffle-js(graaljs) you can use jdk 11 but only the graal compatible versions like 11.02 11.08 not 11.06 and not 4

The most easy and fast way to use this is to use the most current graalvm suite which is JVM+JVMCI+GRAAL+GRAALJS with consistent versions.

The parts that i listed have also many moving subparts

es4x works with stock labs jdk a open jdk that is build with jvmci support

jvmci is the java virtual machine compiler interface that is the most importent as graal uses that when you do not have that and you want to use es4x without graal compiler support it will be up to 10x slower.

pmlopes commented 4 years ago

@UglyHobbitFeet the dockerfile is a 3 stage build, say for example that you build with:

docker build -t myapp:1.0.0 --build-arg BASEIMAGE=openjdk:11 .

It should build a working container based on openjdk 11. Indeed you spotted a small issue, that the 2nd stage requires:

An alternative to this is to replace the download + untar from 2nd stage, to the 1st stage, by either forcing the @es4x/create package to be always installed, however this has a few side effects as pulling all dev dependencies to the runtime, which may not be needed/desired as it will increase the final container image in ~3Mb.

Or by ensuring that the 1st stage has all needed OS dependencies and copy the untar'ed files to the 2nd stage (which will require no other dependencies that the JVM + bash)

UglyHobbitFeet commented 4 years ago

Thanks to both for your reply/insights. I tried building with openjdk11 as pmlopes mentioned in the docker build command but I get the same error about 'Cannot install GraalVM MBean due to Receiver class'. Can either of you verify this to be the case on your end?
On a side-note is there any better way to bring the size of the image down? My 'hello-world' test image using openjdk11 is coming in at 900MB (or 1.4Gb with GraalVM) which is way too high for a bare-bones project with no custom code :(

UglyHobbitFeet commented 4 years ago

I have also tried with amazoncorretto:11-alpine-jdk which is jdk 11.08 but I get the same issue

pmlopes commented 4 years ago

To bring the size down, then the jlink command is the best option. It picks a graalvm/openjdk distribution and strips all the unnecessary parts to a minimal jdk

UglyHobbitFeet commented 4 years ago

I adjusted the code to use jlink similar to below:


# Switched to v11 instead of default of v8 so we could use jlink. It outputs to /opt/graalvm-ce-java11-20.2.0
ARG BASEIMAGE=oracle/graalvm-ce:20.2.0-java11
...
RUN ${GRAALVM_DIR}/bin/jlink \
    --verbose \
    --add-modules \
 <typical java deps then>
 ,org.graalvm.locator,org.graalvm.truffle,org.graalvm.sdk,org.graalvm.js.scriptengine,com.oracle.graal.graal_enterprise \
    --compress 2 \
    --strip-debug \
    --no-header-files \
    --no-man-pages \
    --output "/someDir"

When I run the code I get the following output 

./node_modules/.bin/es4x-launcher: line 33: /opt/graalvm-ce-java11-20.2.0/bin/java: No such file or directory

However, the file is there. I can see it when I inspect my docker. Any ideas?

pmlopes commented 4 years ago

Try es4x jlink instead. The command is quite complex and requires more than 1 step.

pmlopes commented 4 years ago

Francesco told me that he combined jlink and distress to make a small image I'll check his work tomorrow and see how to integrate it

UglyHobbitFeet commented 4 years ago

FWIW when I use es4x jlink I get the same error.

RUN es4x jlink --target=/someDir
./node_modules/.bin/es4x-launcher: line 33: /opt/graalvm-ce-java11-20.2.0/bin/java: No such file or directory

Nevertheless I'll wait for his image. Thanks!

UglyHobbitFeet commented 4 years ago

FWIW Here's a comment stripped version of where I'm at (based on your original file). Maybe it can be integrated somehow with yours/his?

ARG BASEIMAGE=oracle/graalvm-ce:20.2.0-java11
ARG BASEIMAGE_DIR=/opt/graalvm-ce-java11-20.2.0
ARG APP_DIR=/foo
ARG JVM_MIN_DIR=/bar

# Build app via npm clean install
FROM node:lts-alpine AS NPM_LAYER
ARG APP_DIR
WORKDIR "${APP_DIR}"
COPY . .
RUN npm --unsafe-perm ci

# Build the JVM related code
FROM $BASEIMAGE AS ES4X_LAYER
ARG ES4X_VERSION=0.13.2
ARG ES4X_OPTS=""
ARG APP_DIR
WORKDIR "${APP_DIR}"
RUN curl -sL https://github.com/reactiverse/es4x/releases/download/${ES4X_VERSION}/es4x-pm-${ES4X_VERSION}-bin.tar.gz | tar zx --strip-components=1 -C /usr/local
ENV ES4X_ENV=production
COPY --from=NPM_LAYER "${APP_DIR}" .
RUN es4x install $ES4X_OPTS

# Build minimal JVM via jlink
ARG BASEIMAGE_DIR
ARG JVM_MIN_DIR
RUN es4x jlink --target="${JVM_MIN_DIR}"

# Build final slim
FROM alpine:latest as FINAL_IMAGE
ARG BASEIMAGE_DIR
ARG APP_DIR
ARG JVM_MIN_DIR
ENV PATH="${PATH}:${BASEIMAGE_DIR}/bin"
WORKDIR "${APP_DIR}"
COPY --from=ES4X_LAYER "${APP_DIR}" .
COPY --from=ES4X_LAYER "${JVM_MIN_DIR}" "${BASEIMAGE_DIR}"
ENV JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+UseContainerSupport"
RUN apk add --no-cache bash
ENV JAVA_HOME="${BASEIMAGE_DIR}"
ENTRYPOINT [ "/bin/bash", "-c", "./node_modules/.bin/es4x-launcher" ]
pmlopes commented 4 years ago

@UglyHobbitFeet you can use this one for now if you like:

ARG BASEIMAGE=openjdk:11-jdk-slim
ARG RUNTIMEIMAGE=gcr.io/distroless/java:11
# Use official node for build
FROM node:lts AS NPM
# Create app directory
WORKDIR /usr/src/app
# Add the application to the container
COPY . .
# Install app dependencies
# npm is run with unsafe permissions because the default docker user is root
ENV NODE_ENV=production
RUN npm --unsafe-perm ci

# Second stage (build the JVM related code)
FROM $BASEIMAGE AS JVM
ARG ES4X_VERSION=0.13.2
ARG ES4X_OPTS=""
# Download the ES4X runtime tool
ADD https://github.com/reactiverse/es4x/releases/download/${ES4X_VERSION}/es4x-pm-${ES4X_VERSION}-bin.tar.gz /tmp
RUN tar zxf /tmp/es4x-pm-${ES4X_VERSION}-bin.tar.gz --strip-components=1 -C /usr/local
# force es4x maven resolution only to consider production dependencies
ENV ES4X_ENV=production
# Copy the previous build step
COPY --from=NPM /usr/src/app /usr/src/app
# use the copied workspace
WORKDIR /usr/src/app
# Install the maven dependencies
RUN es4x install $ES4X_OPTS

# Third stage (contain)
FROM $RUNTIMEIMAGE
# Collect the jars from the previous step
COPY --from=JVM /usr/src/app /usr/src/app
#COPY --from=JVM /tmp/jre /usr/local
# use the copied workspace
WORKDIR /usr/src/app
# define the entrypoint
ENTRYPOINT [  "/usr/bin/java", "-XX:+IgnoreUnrecognizedVMOptions", "--module-path=node_modules/.jvmci", "-XX:+UnlockExperimentalVMOptions", "-XX:+EnableJVMCI", "--upgrade-module-path=node_modules/.jvmci/compiler.jar", "-XX:+UseCGroupMemoryLimitForHeap", "-XX:+UseContainerSupport", "-jar", "./node_modules/.bin/es4x-launcher.jar" ]

On my machine an hello world container takes: 

myapp                                     0.0.1               4afe8481614f        2 minutes ago       275MB

For comparison:

paulo:~$ docker images|grep node
node                                      lts                 1f560ce4ce7e        7 days ago          918MB
paulo:~$ docker images|grep java
oracle/graalvm-ce                         20.2.0-java11       fbecb9923cd0        2 months ago        1.33GB
oracle/graalvm-ce                         20.1.0-java11       2adef7aab8f9        5 months ago        1.27GB
gcr.io/distroless/java                    11                  56c1d100a082        50 years ago        197MB
paulo:~$

Note that the app is a docker layer, so you won't be really downloading 275Mb all the time, as the base layer gcr.io/distroless/java doesn't change.

The upside of this container is that since it's distroless there's no shell so in theory it is safer, the downside is that, since there's no shell you can't pass arguments, for example, changing number of instances, etc... plus the application layer is bigger as it needs to include all the graaljs dependencies.

pmlopes commented 4 years ago

A further improvement is to run jlink and instead of gcr.io/distroless/java use gcr.io/distroless/debian however it is not a trivial task as the base glibc/zlib versions must match which makes it more error prone.

UglyHobbitFeet commented 4 years ago

I just tested it out and got similar results. Thanks so much for helping to reduce the overall image size! Can't wait to see a jlink version.

UglyHobbitFeet commented 4 years ago

I notice that my package.json has a postinstall for es4x install. Does this duplicate anything being done in the es4x install called within the docker?

I also notice that the node_modules/.jvmci folder contains duplicate graal and truffle jars already contained in node_modules/.lib.

As a potential improvement, modifying the following can help reduce the node_modules folder which can get quite large by using a 3rd party app (like npm-prune and/or modclean) along with npms production mode 

FROM node:lts AS NPM
...
ADD https://gobinaries.com/tj/node-prune /tmp
RUN npm install modclean -g \
 && npm --unsafe-perm ci --only=production \
 && modclean -r -n default:safe,default:caution \
 && sh /tmp/node-prune
pmlopes commented 4 years ago

@UglyHobbitFeet this is what I got:

ARG BASEIMAGE=adoptopenjdk/openjdk11
ARG RUNTIMEIMAGE=gcr.io/distroless/base
# Use official node for build
FROM node:lts AS NPM
# Create app directory
WORKDIR /usr/src/app
# Add the application to the container
COPY . .
# Install app dependencies
ENV NODE_ENV=production
# npm is run with unsafe permissions because the default docker user is root
RUN npm --unsafe-perm ci

# Second stage (build the JVM related code)
FROM $BASEIMAGE AS JVM
ARG ES4X_VERSION=0.13.3
# Download the ES4X runtime tool
RUN curl -sL https://github.com/reactiverse/es4x/releases/download/${ES4X_VERSION}/es4x-pm-${ES4X_VERSION}-bin.tar.gz | tar zx --strip-components=1 -C /usr/local
# force es4x maven resolution only to consider production dependencies
ENV ES4X_ENV=production
# Copy the previous build step
COPY --from=NPM /usr/src/app /usr/src/app
# use the copied workspace
WORKDIR /usr/src/app
# Install the maven dependencies
RUN es4x install
# Build Custom runtime
RUN es4x jlink --target=/es4x

# Third stage (contain)
FROM $RUNTIMEIMAGE
# Collect the jars from the previous step
COPY --from=JVM /usr/src/app /usr/src/app
# Collect the runtime
COPY --from=JVM /es4x /usr
# We also need libz
COPY --from=JVM /lib/x86_64-linux-gnu/libz.so.1 /usr/lib
# use the copied workspace
WORKDIR /usr/src/app
# define the entrypoint
ENTRYPOINT [  "/usr/bin/java", "-XX:+IgnoreUnrecognizedVMOptions", "--module-path=node_modules/.jvmci", "-XX:+UnlockExperimentalVMOptions", "-XX:+EnableJVMCI", "--upgrade-module-path=node_modules/.jvmci/compiler.jar", "-XX:+UseCGroupMemoryLimitForHeap", "-XX:+UseContainerSupport", "-jar", "./node_modules/.bin/es4x-launcher.jar" ]

It's a openjdk 11 build, which means it will need to enable the JVMCI interface which on openjdk is still marked as experimental but it's the same as graalvm. This brings a few extra MBs of dependencies otherwise there's no optimized js engine.

With this dockerfile I get:

paulo (develop):~/Projects/reactiverse/es4x$ docker images|grep myapp
myapp                                     0.0.1               e5249a9ed0a6        11 minutes ago      149MB

Which is better than the original 1.1GB, but remember in terms of layers, this is worse. On the other hand, it now uses distroless, so it should be safer (no root, no shell).

This is interesting in terms of size, as a working app is even smaller than the minimum distroless java image as jlink got rid off all the non used modules such as awt, swing (UI), etc...

You can still pass arguments to this container, for example, deploy multiple instances:

docker run -it myapp:0.0.1 -instances 2

And the -instances 2 will be passed to the entry point

UglyHobbitFeet commented 4 years ago

Looks good, I'll check it out. When I used es4x jlink before I got multi-stage? errors. I see you fixed that in the latest version. Will I have to wait for that to be released for the jlink to work?

On Wed, Oct 21, 2020, 4:37 AM Paulo Lopes notifications@github.com wrote:

@UglyHobbitFeet https://github.com/UglyHobbitFeet this is what I got:

ARG BASEIMAGE=adoptopenjdk/openjdk11ARG RUNTIMEIMAGE=gcr.io/distroless/base# Use official node for buildFROM node:lts AS NPM# Create app directoryWORKDIR /usr/src/app# Add the application to the containerCOPY . .# Install app dependenciesENV NODE_ENV=production# npm is run with unsafe permissions because the default docker user is rootRUN npm --unsafe-perm ci

Second stage (build the JVM related code)FROM $BASEIMAGE AS JVMARG ES4X_VERSION=0.13.3# Download the ES4X runtime toolRUN curl -sL https://github.com/reactiverse/es4x/releases/download/${ES4X_VERSION}/es4x-pm-${ES4X_VERSION}-bin.tar.gz | tar zx --strip-components=1 -C /usr/local# force es4x maven resolution only to consider production dependenciesENV ES4X_ENV=production# Copy the previous build stepCOPY --from=NPM /usr/src/app /usr/src/app# use the copied workspaceWORKDIR /usr/src/app# Install the maven dependenciesRUN es4x install# Build Custom runtimeRUN es4x jlink --target=/es4x

Third stage (contain)FROM $RUNTIMEIMAGE# Collect the jars from the previous stepCOPY --from=JVM /usr/src/app /usr/src/app# Collect the runtimeCOPY --from=JVM /es4x /usr# We also need libzCOPY --from=JVM /lib/x86_64-linux-gnu/libz.so.1 /usr/lib# use the copied workspaceWORKDIR /usr/src/app# define the entrypointENTRYPOINT [ "/usr/bin/java", "-XX:+IgnoreUnrecognizedVMOptions", "--module-path=node_modules/.jvmci", "-XX:+UnlockExperimentalVMOptions", "-XX:+EnableJVMCI", "--upgrade-module-path=node_modules/.jvmci/compiler.jar", "-XX:+UseCGroupMemoryLimitForHeap", "-XX:+UseContainerSupport", "-jar", "./node_modules/.bin/es4x-launcher.jar" ]

It's a openjdk 11 build, which means it will need to enable the JVMCI interface which on openjdk is still marked as experimental but it's the same as graalvm. This brings a few extra MBs of dependencies otherwise there's no optimized js engine.

With this dockerfile I get:

paulo (develop):~/Projects/reactiverse/es4x$ docker images|grep myapp myapp 0.0.1 e5249a9ed0a6 11 minutes ago 149MB

Which is better than the original 1.1GB, but remember in terms of layers, this is worse. On the other hand, it now uses distroless, so it should be safer (no root, no shell).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/reactiverse/es4x/issues/433#issuecomment-713407460, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALBDS7QIKNO6DU4QDYR2K4DSL2MWHANCNFSM4STPNHQQ .

UglyHobbitFeet commented 4 years ago

Maybe the flag was --multi-version. In can't remember offhand

On Wed, Oct 21, 2020, 6:54 AM Jeremy D jeremy.disten.work@gmail.com wrote:

Looks good, I'll check it out. When I used es4x jlink before I got multi-stage? errors. I see you fixed that in the latest version. Will I have to wait for that to be released for the jlink to work?

On Wed, Oct 21, 2020, 4:37 AM Paulo Lopes notifications@github.com wrote:

@UglyHobbitFeet https://github.com/UglyHobbitFeet this is what I got:

ARG BASEIMAGE=adoptopenjdk/openjdk11ARG RUNTIMEIMAGE=gcr.io/distroless/base# Use official node for buildFROM node:lts AS NPM# Create app directoryWORKDIR /usr/src/app# Add the application to the containerCOPY . .# Install app dependenciesENV NODE_ENV=production# npm is run with unsafe permissions because the default docker user is rootRUN npm --unsafe-perm ci

Second stage (build the JVM related code)FROM $BASEIMAGE AS JVMARG ES4X_VERSION=0.13.3# Download the ES4X runtime toolRUN curl -sL https://github.com/reactiverse/es4x/releases/download/${ES4X_VERSION}/es4x-pm-${ES4X_VERSION}-bin.tar.gz | tar zx --strip-components=1 -C /usr/local# force es4x maven resolution only to consider production dependenciesENV ES4X_ENV=production# Copy the previous build stepCOPY --from=NPM /usr/src/app /usr/src/app# use the copied workspaceWORKDIR /usr/src/app# Install the maven dependenciesRUN es4x install# Build Custom runtimeRUN es4x jlink --target=/es4x

Third stage (contain)FROM $RUNTIMEIMAGE# Collect the jars from the previous stepCOPY --from=JVM /usr/src/app /usr/src/app# Collect the runtimeCOPY --from=JVM /es4x /usr# We also need libzCOPY --from=JVM /lib/x86_64-linux-gnu/libz.so.1 /usr/lib# use the copied workspaceWORKDIR /usr/src/app# define the entrypointENTRYPOINT [ "/usr/bin/java", "-XX:+IgnoreUnrecognizedVMOptions", "--module-path=node_modules/.jvmci", "-XX:+UnlockExperimentalVMOptions", "-XX:+EnableJVMCI", "--upgrade-module-path=node_modules/.jvmci/compiler.jar", "-XX:+UseCGroupMemoryLimitForHeap", "-XX:+UseContainerSupport", "-jar", "./node_modules/.bin/es4x-launcher.jar" ]

It's a openjdk 11 build, which means it will need to enable the JVMCI interface which on openjdk is still marked as experimental but it's the same as graalvm. This brings a few extra MBs of dependencies otherwise there's no optimized js engine.

With this dockerfile I get:

paulo (develop):~/Projects/reactiverse/es4x$ docker images|grep myapp myapp 0.0.1 e5249a9ed0a6 11 minutes ago 149MB

Which is better than the original 1.1GB, but remember in terms of layers, this is worse. On the other hand, it now uses distroless, so it should be safer (no root, no shell).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/reactiverse/es4x/issues/433#issuecomment-713407460, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALBDS7QIKNO6DU4QDYR2K4DSL2MWHANCNFSM4STPNHQQ .

pmlopes commented 4 years ago

Yes that flag was missing. I've added it to 0.13.3

UglyHobbitFeet commented 4 years ago

Ahh, just saw you released. Great, I'll download and check it out!

On Wed, Oct 21, 2020 at 7:48 AM Paulo Lopes notifications@github.com wrote:

Yes that flag was missing. I've added it to 0.13.3

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/reactiverse/es4x/issues/433#issuecomment-713512028, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALBDS7RKHRCRGIL52EHIFWDSL3DCVANCNFSM4STPNHQQ .

UglyHobbitFeet commented 4 years ago

Works great, thanks again!!

UglyHobbitFeet commented 4 years ago

Just curious, but does the 'es4x install' command that appears in the package.json postinstall section clash/duplicate anything going on in the dockers 'RUN es4x install' command? In other words, is there a need to use both, or should they be combined? For example, I see the .jvmci and .lib folders both have the truffle-api and graal-sdk jars)

pmlopes commented 4 years ago

I've noticed that the 2 jars can be removed from .lib if there is .jvmci and it is used, but I'm not sure if that is fine or not. For example that will break on openj9. Sure it will remove 6Mb to the final image.

Regarding the multiple es4x install, on the first container (node) i disable it, and it only install prod dependencies so it should not be called. On the second I manually install from github so it doesn't polute the node_modules dir.

What I can do extra is if both .lib and .jvmci exist then remove the 2 jars from .lib

pmlopes commented 4 years ago

So after the jlink action I added:

# Strip
RUN rm node_modules/.lib/graal-sdk-*.jar
RUN rm node_modules/.lib/truffle-api-*.jar
RUN rm node_modules/.lib/profiler-*.jar
RUN rm node_modules/.lib/chromeinspector-*.jar

And the docker images now list:

$ docker images|grep myapp
myapp                    0.0.1               4d07d19d8d52        25 seconds ago       137MB

And the app still runs... but you won't be able to debug/profile it and i'm not sure if we broke anything else either...

UglyHobbitFeet commented 4 years ago

Gotcha. I'll test it out and if I run across anything odd, I'll report back. Thanks again for all your hard work!