Discovering deployables

[mdutoo]

Service discovery allow comparison for sanity check purposes. Versioned services allow to check that the "right" service is used along an EasySOA-integrated SOA development process (ex. in dev, according to design).

However they do not allow that when dealing with what already exists in the entreprise, because the version of the consumer is not known. It's possible to do a discovery of consumed services, but all too often consumed service definitions point to an unversioned, runtime only definition provided by the provider's service stack - so even if they had this information, it would be about the provider.

The solution is discovery of versioned service consumer implementations, that is applications, and at a grain both finer and more manageable (because that's their implementation i.e. what they're made of), deployables (or "modules"). Deployables can be dependency artifacts (ex. in Java Maven artifacts or OSGi bundles, in Ruby Gems...), or be more specific to platforms.

If versioned deployables's internals have information (typically configuration ex. SCA, OSGi, but also processes ex. BPEL, XPDL) :

• about which services they consume, then the deployable version can be checked against the consumed service's and allow to check things like that "the service('s version or definition) just changed, so consumer implementations should also change, or else they'll break"
• about services they expose, then it allows to check that "the definition's just changed, the implementation should also, else both are not related i.e. wrong (EasySOA) configuration"

Else all deployables (or a root deployable) can be used as signature / trace of the application version, and the same checks be made, albeit on a smaller perimeter.

in Java, with Maven

Each maven artifact (jar, war, pom...) is a versioned deployable, and to each ApplicationImpl is mapped a root artifact (typically a pom). Maven snapshots can also be used it they're further versioned by a timestamp. Beyond, maven artifacts are also OK for most Java deployables (Nuxeo bundles, OSGi bundles, JBI, Bonita's .bar archives...) and dependency managers (Ivy...).

Maven artifacts can be discovered :

• after release, Maven repository discovery : by indexing a Maven repository, typically the entreprise's during Continuous Integration (but the local one would also have uses for the developer) :
• at runtime (be it dev, staging...), classpath discovery : classpath can be introspected to report which versioned maven jar artifacts are used ; if no locally indexed Maven repository is available to get artifact name and group, it can still work with only jar names. It can even work for non-maven jars using a few default "classical" versioned jar naming pattern, but with a degraded behaviour since their version can be provided with an unsupported pattern or not at all. The case of application servers & OSGi have to be handled right, because they have specific classloader behaviour but also because they may already provide this feature at some level (ex. OSGi can provide bundle information & version at runtime).
• during build : (more focused on the developer) a maven plugin that would say (for the configured environment, and possibly application being developed) when a new artifact version has been released. Snapshots also have uses. Discovering all artifacts used at this stage would be too much though.
• before build : (less interesting) maven artifact information of a yet unbuilt project could still be told to EasySOA when reporting use of a service (ex. a business process designer adds a new task calling a service).

Doing Maven artifact discovery :

• may be only done on a restricted perimeter (ex. artifact group) for accuracy and performance, configured in AppliImpl and used in discovery probe.
• extracts group, artifact name and version, also original timestamp for snapshot
• optionally, a flat list of resolved dependency (within the configured perimeter) is useful below. A locally indexed Maven repository can be used to get it now, or at a later step if not yet available.
• pluggable internals discoveries can be triggered or reused here to also get info about provided and consumed services or more (process definition...).Ex. the already available #28 FraSCAti composites discovery
• reports all of that to EasySOA / Nuxeo
• if it has a use context (ex. dev of a project), it is used to impact the corresponding AppliImpl
• besides, also impacted are AppliImpls whose configured root artifact (pom, war...) have changed
• otherwise, let deployable discovery probe display the mapping problem and possibly suggestions
• to also support the case of snapshots : impacted AppliImpls are those whose root artifact depends of a changed artifact. For that, only a flat, resolved dependency list (within the configured restricted perimeter) is required, and deployables can be appropriately stored in a flat manner also in the Nuxeo model. Artifact changes can be detected using versions, and original timestamp (for snapshots).
• EasySOA does sanity checks when asked, comparing the root deployable version, or if none or snapshot comparing all deployable versions (or to make it quicker, a hash of an ordered list of them).

  Service stacks & ESB already managing such info

They can use the same EasySOA remote Service & Deployable discovery API to feed EasySOA with information they already manage : their versioned deployables, the services they expose and consume and further internals info. They can do so by pull (EasySOA asking them) or push. Pull can be done by Jasmine probes, or by EasySOA (triggered by user action, scheduling or their registration).

Priorities

• Quick win demo : classpath discovery & first sanity checks (on alphabetically ordered versioned jar list stored in string metadata).
• Full model, unification with Nuxeo bundles, maven repository discovery (with and for Improve, Talend), unification with FraSCAti composite discovery, unification with FraSCAti apps (especially demos and Studio apps).
• If used : Pull using Jasmine, high-level process internals support. If useful : snapshot support.

[mkalam-alami]

Specs for the Discovery by Classpath Analysis prototype

Configuration/launch

• A propertiies file is needed to configure the probe, to provide:
  • The name of the current environment
  • The URL (ID?) of the application
  • The URL of EasySOA
  • (Other ideas: name filter, delay, frequency)
• When launching the server, the user must somehow call EasySOAClasspathAnalysis.discover() after the classpath is set up to trigger the discovery.

Discovery behavior

1. Load the configuration
2. Explore the classloader to find all Jars and gather their names
3. Send a notification to EasySOA, to store the jar list in the configured app (in configured environment)
4. In EasySOA, store in a list property of AppliImpl the jar list, while making sure there are no duplicates (in case several notifications are fired)

Validation improvement

• Make the Jar list appear in the validation dashboard for both the work environment and the reference environment
• (Mock?) Improve validation to handle the following checks :
  • If an application consumes a service whose definition has been updated, check that this application's implementation has been udpated too.
  • If a service definition has been updated, check that its implementation has been updated too.
  • (Other use cases?)

(Discovery note: in general, if we don't know all that we need to send a discovery notification, the pending data should be stored and the user action required on the probe side - not on the EasySOA core)