multiple jar files

[rowlandwatkins]

Hi folks,

I know that this updated version of Capstan supports old Capstan files, which let you specify exact files to upload. In the case of Java, we've been putting everything into one jar file which can slow the build time considerably. One issue with specifying each jar is that it might change after a dependency upgrade. Is there any way to simply do the following:

target/libs/*

To include a load of jars required for the main application?

Many thanks in advance!

Rowland

[miha-plesko]

Hi @rowlandwatkins , I think you need to use two * like this: target/libs/**.

But I would suggest you to make use of the more recent Capstan approach that uploads all files from your current directory to the unikernel by default; no need to list them all (but you can define which ones to exclude). If you have directory structure like this:

my-application
  |- meta
  |    |- package.yaml
  |    |- run.yaml
  |- target
       |- libs
            |- lib1.jar
            |- lib2.jar
            |- my-main-jar.jar

then all you need to do is to run:

$ capstan package compose myunikernel --pull-missing
$ capstan run myunikernel

Where meta/package.yaml is as simple as:

name: my-super-package
title: My Super Package
author: I am

and meta/run.yaml looks like this:

runtime: java

config_set:
   app:
      main: /target/libs/my-main-jar.jar  # <--- which jar file is the main entrypoint to your application
   classpath:
      - /target/libs  # <--- where are other jars located
config_set_default: app

Please let me know if it works!

[rowlandwatkins]

Hi @miha-plesko , wow this is cool, thanks for the example, I'll give this a whirl and get back to you!

Rowland

[rowlandwatkins]

Ok, so the fun begins. I've created the necessary meta/package.yaml and meta/run.yaml, and specified as suggested. But there may be a problem with AWS S3:

capstan package compose someunikernel --pull-missing yields:

[[[ Prepending 'java' runtime dependencies to dep list: [eu.mikelangelo-project.osv.java] package eu.mikelangelo-project.osv.java is not available in the given repository (https://mikelangelo-capstan.s3.amazon.com/) ]]]

In other closed tickets, I noticed that there's an openjdk8-zulu-full.mpm, it's not clear how to use this or import into capstan.

Any tips for moving forward?

Cheers,

Rowland

[miha-plesko]

Hi @rowlandwatkins, could you please install latest Capstan:

$ wget -O - https://raw.githubusercontent.com/mikelangelo-project/capstan/master/scripts/download | bash
$ capstan --version
capstan version 'v0.3.0'

I apologize for delay on providing the more recent Capstan binary before (we released it just now), but we were always compiling it from source so it just passed by...

[miha-plesko]

So by default package openjdk8-zulu-compact1 is used if you specify your meta/package.yaml and meta/run.yaml as shown above.

The -compact1 package means that not all Java classes are included (which results in smaller unikernel size). There is also -compact3-with-java-beans package available and openjdk8-zulu-full, each containing more and more built-in Java classes. You can choose which one you want in meta/package.yaml like this:

name: my-super-package
title: My Super Package
author: I am
require:
  - openjdk8-zulu-full  # <----------- here

Having this in place, just build your unikernel like before:

$ capstan package compose myunikernel --pull-missing
$ capstan run myunikernel

and viola, your application should be started inside unikernel! 😄

[rowlandwatkins]

Very cool, thanks, the new version has downloaded correctly. Now, --pull-missing....

I'm now uploading 81503 files to the base image - is this expected?

[[[ runtime: java

config_set: app: main: /target/scala-2.12/pack/lib/myjar.jar classpath:

• /target/scala-2.12/pack/lib config_set_default: app ]]]

I'm assuming /target is a relative path to the project directory and not root. Checking target/scala-2.12/pack/lib/ there are 243 jars to add. Am I missing anything here?

Cheers,

Rowland

[rowlandwatkins]

With 81k files it'll take around 50 mins to complete ;)

[rowlandwatkins]

Hmm, even removing beginning / yields an upload size of 81k

[rowlandwatkins]

Ahhhhh, capstan is pulling the entire compiler output directory.....

It seems to take everything in target/ not target/pack/lib

[rowlandwatkins]

So continuing my voyage, I added -v to get some debug stuff going:

[[[ .capstanignore: ignore /.capstanignore .capstanignore: ignore /.git (entire folder) .capstanignore: ignore /.gitignore .capstanignore: ignore /meta/package.yaml .capstanignore: ignore /mpm-pkg (entire folder) ]]]

the last entry is a little odd because when we start to upload, most files are being copied from mpm-pkg ;)

I've also added a .capstanignore to the project but not sure if that's working...

[miha-plesko]

@rowlandwatkins so the most important thing is that capstan package compose ... uploads ALL what is within your current directory into the unikernel (unless what is listed in your .capstanignore). Are there 81k files in your project?

You have two options: either remove stuff that you don't need prior invoking capstan package compose or specify .capstanignore file so that those files will be omitted. You can consult this document to see how .capstanignore file can be written.

Please let me know if it works!

[rowlandwatkins]

@miha-plesko cools, yes it did help, many thanks! I'm now reduced to a balmy 536 files to upload ;) Compose completes successfully, but nothing runs yet because of some jar naming hiccups.

Ok, so I have another question about adding a java agent - is there a way to specify further file mappings so a java agent can be uploaded and executed via command line args?

Many thanks in advance!

Rowland

[miha-plesko]

Unfortunately I have no idea what "java agent" is, is this a jar file that you want to run? Please take a look at our documentation on how to run Java in unikernel: https://github.com/mikelangelo-project/capstan/blob/master/Documentation/RuntimeJava.md.

Please let me know if your java agent has any specifics that prevents it from running.

[rowlandwatkins]

Hi @miha-plesko , so I need to specify the following when running Java:

1) logging config 2) application config (injected from cloud-init) 3) java agent files (including shared native libs) 4) jks keystores for rabbitmq, disabledAlgorithm props list

Java agent example:

/java.so -agent /lib/myagent.jar -Dlogger.file=/conf/logging/logback.xml -Dconfig.file=/conf/application.conf -cp /lib/* -jar some.jar

What I'd like is to have the ability to map statically some files which are always known: logging, app config, encryption keys, agent. But then also be able to pull in the entire lib directory so I don't have to specify exactly what the runtime libs are.

Hope that makes sense!

Rowland

[miha-plesko]

Hi @rowlandwatkins , could you please try following meta/run.yaml:

runtime: native
config_set:
  agent:
    bootcmd: /java.so -agent /lib/myagent.jar -Dlogger.file=/conf/logging/logback.xml -Dconfig.file=/conf/application.conf -cp /lib/* -jar some.jar
config_set_default: agent

Runtime "native" means that you specify bootcmd as-it-is while runtime "java" tries to build command for you. Since you have your bootcmd well defined, you can just use it directly as shown above (but please don't forget to include "openjdk8-zulu-..." in your meta/run.yaml).

Looking forward to hear how it works!

[rowlandwatkins]

Hi @miha-plesko, nice this works great, thanks!

[miha-plesko]

No problem! Please close the issue if it works now.

[aitorhh]

Since, it's not closed yet... I was wondering if the "!" exclude functionality as in the .gitignore will be implemented.

I personally prefer the old version where just selected paths/files were uploaded (as in Dockerfile). Otherwise, the unikernels won't be optimized in terms of space.

Thanks!

[miha-plesko]

This issue is already scrooling nightmare 😄 So I'm closing it and opening a new one https://github.com/mikelangelo-project/capstan/issues/92 for the feature requested by @aitorhh