Decide where and how the supported quickstarts will be maintained

[gastaldi]

The backend needs to know what quickstarts are available to be displayed in the UI. Where should we keep that data and how?

[gastaldi]

I have a suggestion of having a separate Github repository having one file for each quickstart (YAML or JSON or XML) containing the metadata needed for the backend (supported language runtime, features, patterns, etc)

[gastaldi]

Just got an idea: to have a ProjectType that when executed will download the project content from a Github repository and do the appropriate changes (change groupId/artifactId for example) before returning the ZIP.

Pros: no need to update anything besides the github repository when updates to the quickstart are needed. Cons: We depend on Github being online (but we can also cache that)

[cmoulliard]

ProjectType

So a project type will be mapped to a quickstart & will be used to download the artefact in order to populate the zip file after replacing the gav, package name ? @gastaldi

[gastaldi]

@cmoulliard when you talk about artifact, you mean something like https://github.com/obsidian-toaster/quick_rest_springboot-tomcat/archive/master.zip right? My idea is to grab that ZIP and unzip locally (or git clone), do the necessary changes, ZIP the directory and return to the user.

[cmoulliard]

right

Yes

[cmoulliard]

So the yaml file could include such info

obsidian.yaml
  git:  https://github.com/obsidian-toaster/quick_rest_springboot-tomcat/archive/master.zip
  type: spring-boot
  patterns: 
    - jax-rs
    - secured-jax-rs
    - jpa
    - circuitbreaker
    - health
    - loadbalancing
  openshift: 
    - version: 3.3
    - components:
       - sso
       - metrics
       - apm

[gastaldi]

Yes, the git value could be only organization/repository as we could assume the rest of the values(they are the same for any other repository)

[jstrachan]

its maybe worth stepping back a little - what kind of projects are the quickstarts? For java based quickstarts in fabric8 we went with using archetypes - as its got all the logic to replace group / archetype / version metadata and do the package rename stuff.

So we generate archetypes for all our quickstarts: https://github.com/fabric8-quickstarts which are then reusable in any mvn tool or IDE stand alone - then the Forge wizard can create them.

We then create an archetype catalog to load them all: http://central.maven.org/maven2/io/fabric8/archetypes/archetypes-catalog/

we've a pending issue to make it easy to configure the fabric8 console / fabric8-forge microservice to allow users to register dynamic archetypes (to extend the system with customer generated archetypes for example) https://github.com/fabric8io/fabric8-forge/issues/509 which we need for iPaaS

If we find we need more metadata than is in archetype-catalog we could create our own YAML / XML / file too with more metadata. For the create wizard and funktion wizard we just went with simple naming conventions (separating tags with '-' in the archetype name and filtering out 'archetype' or 'example' from the list of tags) which works pretty well. But am sure more metadata might be handy to help pick them. Once you've picked one we can put any extra metadata inside the archetype jar

[gastaldi]

I also like the idea of having them published as Maven archetypes. My idea of grabbing straight from the Github repository was to have a quick solution until no archetype is published yet, but I agree that a Maven archetype/archetype catalog would be the best solution in this case.

[cmoulliard]

quickstarts

The quickstarts that we are developing will complete the existing Fabric8 Quickstarts. They will support a specific use case (Expose a REST endpoint, Secure a REST endpoint using Keycloak, Persist the data using JPA/JDBC in a DB, ....) for Spring Boot, Eclipse Vert.x & WildFly Swarm

Here is a Spring Boot example which is targeted to be deployed in OpenShift, uses f-m-p & locally minishift

[jstrachan]

BTW we have a CI / CD system to build all the fabric8 quickstarts; it should be easy to fork to create obsidian quickstarts the same way - or we could just include obsidian quickstarts into the fabric8 release process which already creates archetypes + catalog for everything: https://github.com/fabric8io/ipaas-quickstarts

e.g. right now every quickstart in this org is included: https://github.com/fabric8-quickstarts/

we could easily add whatever obsidian organisation you wanted too?

[cmoulliard]

Maven archetype/archetype catalog

We could reuse what Fabric8 has developed to convert our quickstarts as archetypes.

[jstrachan]

I'm still not sure why we can't just share the same library of quickstarts really - though its no biggie to put them in separate organisations if need be.

Am not sure why a quickstart needs a shell script to install minishift though; thats generic stuff right? We can solve that with "mvn fabric8:cluster-start" or something? https://maven.fabric8.io/#fabric8:cluster-start

or at least have a canonical way to do it on any quickstart without copy/pasting shell scripts?

[cmoulliard]

we could easily add whatever obsidian organisation you wanted too?

I will talk about your idea during our next weekly call ;-)

[jstrachan]

@cmoulliard no need to convert any quickstarts to archetypes - thats done automatically - though if the tests fail then there may need to be a minor code change ;)

we do need better visibility from the tests (e.g. to test the generated archtypes individually on minishift/minikube and report back to the project if/when it breaks)

[cmoulliard]

thats generic stuff right?

Yes. The project that I shared is still a early version and some clean-ups are still required. We haven't yet discussed how we will manage the documentation explaining how to setup/install OpenShift locally. As the quickstart could also be used from almighty and deployed within OpenShift online, I admit that we should revisit the way that the README file is designed.

[cmoulliard]

copy/pasting shell scripts?

This is also a point that we haven't yet discussed. It could be possible that we will keep some shell scripts within the early releases till we will find a more decent strategy to install & configure some services. Example : To secure a REST endpoint (Spring-Boot, Vert.x or WildFly, Red Hat SSO must be installed on Openshift, a realm created, like a clientID + url to access the Web Resource, a user/password created with some roles and the keycloak.json file of the quickstart should be updated with the public key of the realm.

[jstrachan]

I'd rather make a suite of Forge commands to do things like that really - its so easy for some scripts/docs on a specific quickstart to go stale.

Whereas most of all this stuff (install minishift, secure an endpoint) are all generic tools that should be easy to share across any maven project really.

e.g. why not add the help docs into forge too on how to install minishift, secure a REST endpoint, whatever. Then forge could be the wizards to do this kinda stuff (usable from CLI, REST, web app, IDE etc)

forge help install-minishift
forge help secure-endpoint

etc

[cmoulliard]

why not add the help docs into forge too on how to install minishift, secure a REST endpoint, whatever.

@gastaldi What do you think about this idea ?

[jstrachan]

@cmoulliard @gastaldi am hoping the help docs can all be in some canonical markdown/asciidoc files somewhere & we could convert them into a nice textual console format and include them in forge - along with linking to them online in nicer HTML format?

again I'm hoping we can share docs between fabric8 / minishift (which is becoming CDK3) and obsidian as most of this is all common horizontal stuff. e.g. http://fabric8.io/guide/getStarted/gofabric8.html

[cmoulliard]

some canonical markdown/asciidoc

Ideally, we should decompose a Quickstart README into a series of sections @gastaldi, @jstrachan, @jclingan :

Section	Description
Title	define according to this convention as container type + patterns (eg. Quickstart Spring Boot - Rest endpoint, Quickstart Eclipse Vert.x - Secured Rest endpoint, Quickstart WildFly Swarm - Camel Integration - Publish/Subscribe)
Introduction	use case description with diagrams, links to supported documentation
PreRequisites	Contains pre requisites info like JDK, Maven, Minishift, .... required locally or specific knowledge. The prereq could be defined as a table containing the name of the framework, technology or even pattern (EIP, Microservices), version/description, link to the web site containing more info to install/setup
Service (we should find a better name ?)	This is where we will describe the additional steps required to by example use/setup Red Hat SSO, A-MQ, Decision Server, ...
Build	maven build instructions
Launch	How to run locally the project
Test	How to test, debug the project
Run on OpenShift	Depending how the quickstart will be used and maybe generated; different options will be possible like deploy/run the project on OpenShift Online vs OpenShift Enterprise vs ... and could include additional prereq steps
What else ???	Do we need another section

[jstrachan]

apart from the title + introduction; isn't all the rest of that list the same for every quickstart? The main point of fabric8 + openshift is that you can run & test things in exactly the same way - irrespective of whether folks use spring boot, vertx, WARs, swarm, wildfly, tomcat, tomee et al.

So I'd rather point all the quickstarts to a canonical place with a great website of polished (rather than stale) docs on how to build / run / test / debug / scale / manage etc.

e.g. to run any quickstart its

mvn fabric8:run

the prerequisites are java + maven + a kubernetes/openshift cluster. To debug its

mvn fabric8:debug

etc.

We have many many quickstarts already: https://github.com/fabric8-quickstarts/ and that list is only going to get bigger and bigger.

Maybe we can add a canonical button/image on the bottom of the ReadMe that links to the prerequisites to build/run/test/deploy/debug - plus links to tools where folks can visualise what the exact dependencies are etc. (e.g. stuff that depends on AMQ or SSO or Kafka or Elasticsearch or whatever).

At some point through the Service Catalog we should have a canonical way to specify dependent services in metadata https://github.com/kubernetes-incubator/service-catalog so using that metadata (rather than docs that go stale fast) sounds a nicer idea; then we can have tooling to visualise dependencies & validate that dependent services are properly installed and configured before we can run/debug things etc

[gastaldi]

I think that makes sense. We could have a Forge command that displays the README based on the project type (Quickstart) chosen (writing to the UIOuput.out() stream). Let me know if there is anything that needs to be added/enhanced in Forge core to help accomplish that.

[cmoulliard]

isn't all the rest of that list the same for every quickstart?

I like this idea. I think that we should talk about that (centralized doc site) with Almighty project ?

[jstrachan]

@cmoulliard we already have centralised docs for fabric8; so why not use those & reuse/enhance as required?

BTW AFAIK fabric8 is becoming the upstream OSS project replacing Almighty

[gastaldi]

For now we are using the archetype-catalog.xml that is deployed on https://repository.jboss.org/nexus/service/local/artifact/maven/redirect?r=snapshots&g=io.obsidian&a=archetypes-catalog&v=1.0.0-SNAPSHOT&e=xml&c=archetype-catalog

[jstrachan]

So the manually maintained YAML file per quickstart isn't really required right? My worry is this doc will get super stale super fast since its of little use to developers maintaining the quickstarts - and so become a major burden creating all that YAML and validating its all correct and up to date.

We can instead just use the pom.xml <name> & <description> for most metadata along with a README.md - then put any useful tags (like spring-boot, vertx, REST, camel or whatever) in the archetype ID). Then we've pretty much this metadata https://github.com/obsidian-toaster/platform/issues/1#issuecomment-262913157 and we can provide a tag based UI to select quickstarts like we do in the current fabric8 console from all quickstarts:

(we can obviously update the UI to make it nicer & provide a nicer tag cloud selection mechanism etc).

The only missing piece of metadata is describing the required other services that need to be installed first (e.g. kafka, activemq or whatnot). But then the service catalog is the long term mechanism to specify those kinds of things anyway https://github.com/kubernetes-incubator/service-catalog

Until then we could come up with some annotations we can use to describe dependent services; e.g. names of services we expect to be running (like kafka, elasticsearch, activemq) and have some UI integration the fabric8 console to give developers a hint they need to deploy those apps first before running their quickstart.

We could also use this metadata to automate running system tests (e.g. knowing a system test needs an activemq broker or whatnot).