1 - Store and replay

Store HTTP exchanges using monit's Message API, then use it for simple replay and simulation :

OK HTTPMonit / Servlet : forward(Message), discoverService(Message), record(Message), monit(Message, conf) LATER ArchivedExchangeImporter.import(HAR) LATER WS/RSMonit / CXF : discoverService(), record(), monit() on Message ; AND / OR on Object ? ArchivedExchangeStore.store(Message), store(decoratedMessage)

2 - RunDriver

Allow it to be used in the release through the RunDriver by adding a replay/rerun() (and load() & list() if required)

3 - TemplateService

Develop a TemplateService on FraSCAti providing one operation per template.

As of 20120106 it's been done with a TemplateDefinition consisting of field definitions : { name, type, defaultValue, fieldSetterInfo }, allowing to make a template out of a record's request by using fieldSetterInfos to set this request's fields (within path, headers or content of various formats) to user values, see ExchangeServiceImpl.replayWithTemplate().

4 - Template Builder & Suggester

Separate it in different parts : field suggestion, template building, template use.

Template Suggester

A TemplateFieldSuggester works on anything (a request or record, a whole record store, a service definition...) to suggest fields that would be interesting to make template variables of. Therefore its output is similar to the one of the first TemplateService above.

Template Builder

The TemplateBuilder applies a selection of TemplateFieldSuggestions on a record (request) and uses their fieldSetterInfos to replace those fields' values by template variables (ex. in velocity : $myFieldName), and saves the result in a different record. Note that the TemplateBuilder can therefore be used (or tested) with hand-defined TemplateFieldSuggestions. OLD ExchangeTemplateDefinition : { field : { variableName, originalValue, userLabel=variableName, possibly type }, templatizedExchangeRequest } where templatizedExchangeRequest is an exchange in json format where some parts (fields) have been replaced by a velocity reference to a defined field.

Template Renderer

The TemplateRenderer's idea is to execute a record (request) template by loading its record and rendering it in the chosen template engine, with template variables set to user provided values first, doing the corresponding request call and returning its result. Note that the TemplateRenderer can therefore be used (or tested) with hand-defined exchange (request) templates. OLD The TemplateService sends the json request (loaded as json from the ExchangeStore) rendered by velocity using field values provided in the user request.

The TemplateRenderer works thanks to FraSCAti's implementation.velocity, which allows for a Servlet interface but also for any Java custom interface (see example) thanks to (ProxyImplementationVelocity). ProxyImplementationVelocity puts the SCA props and references in the velocityContext, then calls the velocimacro having the same name as the call method, on the template (see exemple) also with the same name (or the default one if not found). Here we use a custom ExchangeTemplateItf interface having a renderRequest(path, reqArgs...), and a hacked ProxyImplementationVelocity (for now copied & changed here, LATER to be contributed to FraSCAti) using rather the nth call argument as the template path where n is the value of a dedicated pathArgIndex SCA property. The request call part can also usefully be externalized in a TemplateExecutor, and theTemplateRenderer therefore be a mere binding.http providing velocity on top of stored, built template records. Ex. template :

#macro ( renderReq $path $argMap )
    where argMap is a java map
#end

On the reponse side, first provide in the same way a template rendering the original response (could be to HTML for replay from the scaffolder, or to JSON or XML in service simulation cases), LATER add a handler that requires less coding (ex. extraction of fields from suggested pathes). Ex :

macro ( renderRes $path $outHeadersAndContentInJavaModelOrJsonOr $outHeaders $outContent $othersExPerfs )

default :
$outHeadersAndContentInJavaModelOrJsonOr
otherwise something like a parametrized version of :
{"headers":null,"messageContent":{"content":"{\"user\":\"toto\",\"lastTweet\":\"This is the last tweet\"}","encoding":"","mimeType":"application/octet-stream","size":52},"status":200,"statusText":"OK"}

end

Improve ease of use by : reset "size" (and make sure it is recomputed when loaded or at worst replayed), LATER handle json or XML providing libs (will solve the inner quote escaping pb), remove null or empty fields in default storage and (EXCEPT interesting ones like headers) when building a template that has to be modifiable by a developer...

5 - Template as SOAP

Develop a component that uses it to provide web services that are a "business" version of a templateRecord's templates. This could be done either in REST ex. instead of having to do GET /template/meteoLille1details { fields:{ name:"id", value:2 }, do GET /meteoLille1/details { id:2 }, and provides the corresponding service definition ; or in WSDL ex. MyTemplateRecordService.templateName(field names and values). First do it for WSDL by copying FraSCAti's ServletImplementationVelocity locally and hacking it to provide, thanks to the above developed TemplateDefinitionService and TemplateExecutor :

Preserve the original behavior of the ServletImplementationVelocity by adding code activated only when a call to /target is detected
for / an HTML list of links to myTemplateStore.wsdl for all template recordStores (listed by ExchangeStoreManager.getStoreList(), or better the TemplateDefinitionService, see below),
for myTemplateStore.wsdl a WSDL generated (including XSD default values ; using XSL or velocity...) from all of the the Store's .fld files (gotten from ExchangeStoreManager.loadJson(.fld), or better move it to a TemplateDefinitionService.list/loadDef() which is an ExchangeStoreManager also aware of .fld files of templateRecords) with for each template one operation with the same name and its fields as parameters,
and for other POST requests a simple implementation of calling these WSDLs (extract service and operation name, then call the TemplateDefinitionService with them to find the template and its .fld, then extract field values as required in.fld from request XML content, then use TemplateExecutor to execute the templated request with these).

Then provide it to INRIA as basis of a (possibly new implementation) contribution & for patches, improvements and feedback on how to develop such things ("meta" service, "proxy" implementation... all also ex. for providing Nuxeo Content Automation as SOAP) for FraSCAti.

OLD On ?wsdl request, it generates a WSDL where input params are said fields, and originalValues are default values. Evaluate whether to do that in a mere binding.http Servlet, or in CXF (ex. with a custom Invoker), or more in FraSCAti (one on-demand proxy for each template and generated code for extraction and routing to them ?), or (YES) first as hacks in frascati-velocity.

LATER Make it easily usable from the ScaffolderService, notably by initializing its form fields using WSDL default values. LATER do the same for the response, by applying the right content model (json, xml, HTTP...) & expression for each output field to be extracted and returned. LATER do it all for REST / json exchanges.

6 - Flexible content (& query, path) modeling - supporting XML & JSON content fields

whatever the request content i.e. query / path / xml or json content, using more languages

7 - Assertions

Assertions are about to check a replay's response using the originally recorded response. They can return a status (OK, MAYBE, KO), a metric of validity (mostly if MAYBE), possibly a model or UI of delta, and additional logs. (TODO or maybe separate metrics from assertions, and let these reference those applicable when in MAYBE ?)

They are developed each in its own class, and configured and run in an AssertionEngine, which can be plugged in #72 Validation / sanity check, or possibly in unit testing (ex. on Java development platforms). Here are a few ideas of assertions :

TODO trimmed string equality of (string version ex. json of) full recorded response and actual one. Metric could be length of diff, difference in length or or other string distance algorithms ; however beware, some like Lehvenstein distance (see example impl source ) are too costly to be done beyond a few words, rather use ex. Hamming distance. Delta is diff (UI). Not very interesting in itself but a basis for further features and easy to do.
trimmed string equality of Response property per property, or only of some properties (ex. status, content), or only of some "fields" (see templating above). Metric (ex. distances) & delta (ex. diff) are aggregates of the previous ones, and / or could / should be further adapted to content kind ex. XML (ex. becomes xmldiff), JSON LATER.
NEXT contains(string), on whole response or per property. Same comments. Simple and very useful.
matches(pattern), on whole response or per property. Same comments.
concerning complex Response properties such as URL and Content : if they are broken up in primitive fields along a model (ex. json or xml content), all of the above can be done more usefully field per field. This requires flexible content modeling, see above. TODO field equality is a particular case, see in autosuggestions below.
custom scripted or coded assertions composiing the previous ones

Those are all string-based assertions.

LATER An alternative is Model-based assertions. They require a service definition, such as FraSCAti requires / provides or templatized requests provide. They allow coded assertions i.e. classical unit tests.

Just as for templates, assertions can be suggested automatically by an AssertionSuggestionService, or submitted to user's approval. Suggestions are stored in .asr JSON files. NB. fieldequality assertions could also be stored as references to fields define in .fld files, or fully in .fld files as mere booleans enabling equality assertion for each field ; this will have to be decided when we'll make a difference between suggested fields and approved fields (belonging to a template, or even tagging semantically the service). Here are a few ideas of assertion suggestions :

stringequality over full response
TODO stringequality of (hopefully) non-variable "business" properties / parts of response : status, mimetype, content TODO more ?
TODO equals(fields that where extracted during automatically suggested templatizing)
contains(longest common substring), for whole Response, per property or even per content field. Requires implementing LCS algorithm, however beware of cost (NP-complex, O(mn)) without improvements or "friendly" data sets
- theory : see http://en.wikipedia.org/wiki/Longest_common_subsequence_problem
- code : any maven ? else ex. BSD by Hunsicker http://www.koders.com/java/fidA84C24D1A7E1A08BC888853E967609B2DFE683EF.aspx
If there are already existing assertion test reports, parse them to see whether metrics are improving, and if not handle it as another assertion failure
LATER other pattern deduction / machine learning techniques...

Finally, with the appropriate UI(s), the user can help define assertions, by :

coding them (in XML / JSON conf, script, java...)
choosing / approving / fine-tuning autosuggested assertions
including notably setting up OK levels for metrics-based assertions (like String distance...), especially by running the assertions a first time to get a metrics level and approving the resulting metrics as OK levels. NB. tracking metrics over time is something Sonar does well also.

TODO in a first time, stringequality assertion as a boolean marker for each field in .fld files (LATER with other assertions in .asr files, maybe mutualize field definition), handling of response content hardcoded for JSON (LATER also XML and pluggable), output in "test report" logs (LATER LogEngine) telling about those & Levenshtein distance. Then fieldequality based on the same field definitions (.fld) as replay & template as soap (i.e. output by the same test).

DONE AssertionEngineImpl, AssertionSuggestionService suggesting AssertionSuggestions on whole response (equality, LCS, Levenshtein BUT both too long because in O(nm)) or Message (MessageAssertion : status, mimetype, encoding) to .asr files. Simple LogEngine.

TODO make result objects model serializable in a simple manner to text or xml. LATER enrich Sonar (& Jenkins first if required ?) with a plugin aggregating those, similar to ex. JMeter's : see image of HTML version or xml report in this page, and image of Sonar aggregated results in this page.

TODO study alternatives to also have JSON / XML & possibly object versions of content, or why not also to have fine model of content type : HashMap(propertyPath_format,finer model) in Message OR in a wrapping MessageHandler object, phased handling to build those models (deserialize) or back (serialize) à la CXF OR hardcoded, take advantage of CXF and extend it or from scratch (depends if same goal & used for same things or not)...

8 - Simulation

A simulation is a server that, for a given request, provides a given response according to simulation data. We'll focus on the case where simulation data is an exchange record store, or anything extracted manually or automatically (suggestion, just as templates and asserts) out of it.

The idea is that the returned response is rendered from a response template filled with known fields found in the request, and that the appropriate response template is chosen among others within a response template store / simulation store by comparing recorded values of known fields of the request related to the template response for each simulation store entry. Just as for templates and asserts, "interesting" fields can be automatically suggested using a correlation algorithm such as the previous request-response correlation one.

templatize responses : complete field definition (& .fld file) to have : (fields { (name/id), req(Extraction)Path, res(ReplaceForTemplatization)Path, (req)foundValue, (resFoundValueNOALLCURRENTCORRELATIONSHAVEEQUALVALUE), correlLevel }, resTemplate) dans un simulStore
write TemplateSimulationServlet.service() : get simulStore from simulStoreProvider (2 different possible impls : existing simulStore vs suggested simulStore generated automatically from existing recordStore using correlation algo) for each exchange in simulStore { try to match all fields together (AND) () .. (try to match some fields) ... try to match one of each field independently, ordered by correlLevel (*) if match add to matches (with correl) } choose better match according to correl if none return "pb" else return templateEngine.render(matchedResponseTemplate, fieldsExtractedFromReq)

(**) for each field in simulExchangeFields : if extractField(res, field.path) == field.value then return field else continue

(*) for each field in simulExchangeFields { if extractField(res, field.path) == field.value then continue else return null; } return simulExchangeFields

More interesting algo for (*) : correlatedFields = new list(); for each field in simulExchangeFields ordered by correlLevel decr { if extractField(res, field.path) == field.value then correlatedFields.add(field) else break; } return correlatedFields;

9 - Further

This is about related stuff that will be done elsewhere, and old specifications where some ideas are still left.

Allow to choose fields using : expression languages (allow to edit the current expression, whatever the request content i.e. query / path / xml or json content, using more languages), "tag / marker" value fed through the client business application, LATER (inferred) definition.

Support HAR import.

LATER Do it within CXF ?

then improve it according to the following further uses and prototype them :

DefinitionInferrer : inferDefinition(restMessages)

All components below rely on message contents (req, res, other ex. timings) having a model able to load & parse them and address (get/set) primitive value elements within. This model can rely on the (possibly inferred) definition, but not necessarily (ex. in req/res body, xpath or jsonpath is enough). Definition is mostly useful for working WITHOUT sample messages (by allowing to generate some, like SOAPUI does ; or to select an element / field).

The best client is always the actual client application. By listening to its exchanges, it allows to input service parameters, but also select a field by inputting a well known value.

ImplicitCorrelator : correlate(Messages, env, conf{ valuesToLookFor }) return correlatedFields Correlations : reqResCorrelation, envCorrelation (IP / host), conf(ex. userInputValue)Correlation, (, apiCodeCorrelation) Decorator : decorate(Messages, correlatedIdentifyingFields, correlatedSegmentingFields) Asserts : (schemaAssert, ) correlatedFieldAssert, contains/regexpAssert, templateAssert, (codeAssert,) slaAssert Templatizer : (re)templatize(Messages, correlator : correlatedIdFields, variables : someCorrelatedFields, asserts : someOtherCorrelatedFields) TemplateEditor : setEditable/load/edit/save(reqOrResTpl)

ReplayTemplatizer : many templatize() TemplatizedExchangeStore : store(PlayableMessages) ExchangeReplayClient.replay(one PlayableMessage { reqTpl, correlationsBesidesReqRes, asserts }, env(, userOrConfParams)), .start(DefinitionFieldChooser, CorrelationFieldChooser)

SimulationTemplatizer : many templatize() ExchangeSimulator.start(set of PlayableMessage { correlator, resTpl, variableCorrelationsBesidesUserInput }, env(, confParams) ), .start(DefinitionFieldChooser, CorrelationFieldChooser) ExchangeReplayClientStore, ExchangeSimulatorStore ?

easysoa / EasySOA

Message API : store and reuse for replay & simulation #73