Runtime Capabilities and the Catalogue

pchainho commented 7 years ago

So far we only use the Browser Runtime and have not used yet RuntimeCapabilities to filter Catalogue objects to be loaded from the Runtime Catalogue.

But now we also have the NodeJS and some components including protostubs and Hyperties may be different from the ones used for Browser runtime.

For example, the default protostub is addressed with its well known URI but the object to be loaded in the runtime will be different depending on the the Runtime Capabilities ie whether it is a Browser or a NodeJS runtime.

The proposal is to have at the Service Framework a new "RuntimeCapabilitiesManager" component that is in charge of collecting all data from the Runtime and generate a .RuntimeCapabilities JSON object. Then the Runtime Catalogue can query the RuntimeCapabilitiesManager to decide which Hyperty or Protostub to be loaded according to the constraints defined in its descriptor.

The "RuntimeCapabilitiesManager" component implementation would be different according to the runtime type.

Pls provide your feedback during this week.

emmelmann-fokus commented 7 years ago

Initial thought as it come up right away when looking at the spec:

RuntimeCapabilities is not mandatory. Do we have in the spec an interpretation in case the field is not present / set? I would assume that in such case, the object is valid for all runtimes, right?

I'll add @Endebert to the assignees to sync as well.

pchainho commented 7 years ago

Right, we should be clear about that in the specs

emmelmann-fokus commented 7 years ago

@Endebert and I discussed this issue. Some feedback here:

For example, the default protostub is addressed with its well known URI but the object to be loaded in the runtime will be different depending on the the Runtime Capabilities i.e. whether it is a Browser or a NodeJS runtime.

We need to distinguish here between the interface to the catalogue and any potential internal interpretation of URI.

So far, a URI represents a unique object. So the inherent assumption in

the object to be loaded in the runtime will be different depending on the the Runtime Capabilities i.e. whether it is a Browser or a NodeJS runtime.

is not according to existing specs.

You use the URI to retrieve a catalogue object. Part of this objection is the optional RuntimeCapabilites element. That allow the receiver of the object to identify if he is able to use the object in its local runtime.

So at the catalogue, there will be no "interpretation" of incoming requests to send difference answers depending on the runtime of the requesting client. You will get exactly the object that you request and that object is uniquely identified by the ...../.well-known/..... URL path.

To proactively strive for a solution that we can achieve within reTHINK, @Endebert and I suggest that you use and create, e.g., a "default-browser" prototstub (or any other catalogue object having this "default-broswer" name); you can have in parallel a "default-nodejs" protostub. Then you may request at the client side the default-browser object if you know that you are a browser runtime. Such "default-xxx" catalogue objects would have to be correctly named and provided by the programmer when adding the object to the catalogue database.

Please be aware that this approach requires agreement on the naming scheme used and that providers of protostubs strictly follow this naming scheme when including their objects in the catalogue database.

Once such a naming scheme is agreed, the mentioned "RuntimeCapabilitiesManager" could provide means to exactly output the suffix (e.g. -browser or -nodejs) that is to be used as an appendix for the url to be retrieved from the catalogue to directly retrieve the runtime-specific catalogue object.

pchainho commented 7 years ago

This solution could work with "atomic" constraints ie when constraints only contains one constraint that is "true".

However it would require to agree / standardise on URL naming paths for all possible combinations of constraints. One example is a Node protostub that is constrained for WebRTC datachannel.

Probably the best is to distinguish between Runtime Type and RuntimeCapability as specified before. In that case the Well Known URI could have the Runtime Type in the path eg ...\.well-known\protocolstub\node\defaultwould be the default protocol stub for node without any constraints and the ...\.well-known\protocolstub\node\default-datachannel would be the default protocol stub for node constrained by datachannel. The name used to identify the constraint should be aligned with the names used in RuntimeCapabilities class.

For complex constraints we could also have a constraints file with a list of catalogue URLs for each supported constraints, something like:

{
    constraints : [{
        datachannel : true,
        webrtc : true,
        windowSandbox : true
    },
    catalogueURL : "hyperty-catalogue://catalogue.hybroker.rethink.ptinovacao.pt/.well-known/protostubs/node/windowSandbox-Webrtc"]
}

The well-known URI for this file could be ...\.well-known\protostubs\node\default-constraints

emmelmann-fokus commented 7 years ago

Hi @pchainho, thanks for the discussion.

I am not sure if it is feasible to standardize (all relevant) combinations of resource paths.

In your example above: the catalogueURL should start a after ./well-known, i.e., it should not contain "hyperty-catalogue://catalogue.hybroker.rethink.ptinovacao.pt/.well-known/ as this the first part is set by the catalogue hosting the object, and the .well-known part is obvious as always present for all objects.

Besides, when you mention ...\.well-known\protocolstub\node\default-datachannel this means changing the path within the catalogue and I need to check with @Endebert first, if this effects the implementation. Due to the state we are in the project, we need to maintain backward compatibility.

pchainho commented 7 years ago

I am not sure if it is feasible to standardize (all relevant) combinations of resource paths.

yes, that's my point

In your example above: the catalogueURL should start a after ./well-known, i.e., it should not contain "hyperty-catalogue://catalogue.hybroker.rethink.ptinovacao.pt/.well-known/ as this the first part is set by the catalogue hosting the object, and the .well-known part is obvious as always present for all objects.

Right

Endebert commented 7 years ago

After some deliberation, I believe a url-based approach is not a feasible solution, especially if we want to keep backward compatibility.

Instead, I would suggest a solution using POST requests, which serves as an extension to the already implemented solution regarding default descriptors.

The Catalogue Broker would have to be configured with a set of contraints, similar to how @pchainho suggested it in a previous comment. Since I would like to still maintain the launch option capabilities of the Catalogue Broker, declaring a constraint could look something like this with a launch option:

Currently, a default protocolstub is defined like this:
java -jar broker.jar -default "protoclstub/SomeStub"

I suggest defining constraints like this:
java -jar broker.jar -default "protocolstub/SomeStub/{'datachannel':true, 'webrtc':true}"

In a brokerconf.json it could look like this:

{
  "defaultDescriptors": {
    "protocolstub": "SomeStub"
  },
  "defaultConstraints": {
    "protocolstub": [
      {
        "datachannel": true,
        "webrtc": true,
        "instance": "SomeStub"
      },
      {
        "datachannel": false,
        "webrtc": true,
        "windowSandbox": true,
        "instance": "SomeOtherStub"
      }
    ]
  }
}

(defaultConstraints has been introduced to the config, because defaultDescriptors can not be reused without losing backward compatibility)

So when requesting e.g. the default protostub, a user could send a POST request to /.well-known/protocolstub/default with additional constraints in the request payload.

Please provide feedback on how this constraint matching is supposed to work. For example, when we look at how getUserMedia() is handled in browsers, constraints can be defined as mandatory or optional. I believe reducing our constraints to only include mandatory entries would suffice, but I would like to hear what you think.

What I mean by mandatory only constraints, is that, if a user requests a default descriptor with some constraints, all of them have to be fulfilled by a "constraint set" defined in the broker config, in order to be chosen. But, a "constraint set" in the config can have more entries than than what the user has requested.

Please note that defaultDescriptors and defaultConstraints are not dependant on each other. Basically, defaultDescriptors is for GET requests, while defaultConstraints is for POST requests. A POST request without a payload would either be rejected or handled like a GET request.

pchainho commented 7 years ago

Not sure if I fully understood but pls note that constraints are required for other catalogue objects not only for default protostobs. Pls see here

If you are able to filter queries at the server why not use the descriptors for that?

At this stage, where the catalogue broker is being only used in a couple of testbeds for some adhoc demos, not being backward compatibility is not a big problem.

Endebert commented 7 years ago

constraints are required for other catalogue objects not only for default protostobs.

Could you please elaborate on that? Do you mean to have multiple catalogue objects with the same name and the Catalogue returns the one matching the constraints? Or to not return a catalogue object if the provided constraints do not match with what is specified in that object?

pchainho commented 7 years ago

I mean, having different catalogue objects that can be retrieved from the same catalogue URL depending on provided runtime constraints.

Endebert commented 7 years ago

I mean, having different catalogue objects that can be retrieved from the same catalogue URL depending on provided runtime constraints.

This is possible under the assumption that we use POST requests that include (in addition to the catalogueURL) a set of constraints.

Your suggested approach:

The well-known URI for this file could be ...\.well-known\protostubs\node\default-constraints

is not a feasible solution for the reason I already mentioned (backwards compatibility).

To move forward, we suggest to adopt the POST request based solution. This means that:

we will have to extend the Broker to parse the constrains stored in the catalogue objects
modify the Broker's all internal mechanisms/datastructures to support multiple catalogue objects with the same name on the same URL
extend the Broker to check the constraints in order to return the first catalogue object whose constraints satisfy the constraints provided in the POST request
extend the RuntimeCatalogue and the RuntimeUA to provide the correct constraints based on the runtime/environment that is being used / constraints provided by a to-be-changed API

These extensions provide additional features which were not covered by the specification of the Catalogue service in phase 1. Since this modification is a significant amount of work, we would consider these work items as substantial achievements in phase 2.

Note that the original work plan did not consider such change in a requested functionality and as a result, other phase 2 work items might have to be deprecated.

pchainho commented 7 years ago

is not a feasible solution for the reason I already mentioned (backwards compatibility).

At this point I don't think it will be a big problem. Anyway I think we can adapt the solution to make it backward compatibility by not changing the path URLs and just introducing one or more constraints file pointing to the right objects according to Runtime constraints.

emmelmann-fokus commented 7 years ago

Hi, am I correct in assuming that we agree on pursuing the POST-based approach that @Endebert suggests?

Please note that this update will unlikely occur before end of the year; as this is a new functional requirement and our work and resource plan blocks us right now in progressing with QoS related implementations.

Endebert commented 7 years ago

@pchainho as far as I understood, additional files or filters for constraints are not needed since I can create them dynamically from the constraints defined in the descriptors, or am I wrong?

For example, this is the descriptor of the runtime for the catalogue-database-reTHINKdefault docker image:

{
  "cguid": "30002",
  "runtimeType": "browser",
  "version": "1.0",
  "objectName": "Runtime",
  "description": "This is a hyperty runtime descriptor",
  "language": "Javascript ECMA6",
  "hypertyCapabilities": {
    "mic": true,
    "camera": true,
    "sensor": false,
    "webrtc": true,
    "ortc": true
  },
  "protocolCapabilities": {
    "http": true,
    "https": true,
    "ws": true,
    "wss": true,
    "coap": false,
    "datachannel": false
  }
}

Then you only need to send a POST request to /.well-known/runtime/default with the payload:

{
  "hypertyCapabilities": {
    "webrtc": true
  },
  "protocolCapabilities": {
    "datachannel": false
  }
}

This would match the provided runtime, so the broker responds to the request with that runtime.

pchainho commented 7 years ago

@Endebert just a note: for simplification purposes there is, now, only one constraints field. Pls check here

pchainho commented 7 years ago

I agree with your proposal @Endebert but I would like to clarify whether there is any change needed in the httpRequest lib and if yes who will take care of that

emmelmann-fokus commented 7 years ago

Status of face-to-face Lisbon meeting:

Confirmed agreement reached in telco last week to proceed as proposed by @Endebert (i.e. to support http post requests on the catalogue server side).

Open issue: are changes needed in the httpRequest lib?

Note: If changes are needed, a partner in addition to Fraunhofer needs to join the team to develop this change in functional requirement specification.

@Endebert to start changing the catalogue-server side of the implementation. If changes in the httpRequest lib are necessary, a partner besides Franhofer committing to implement those changes needs to be volunteered to keep the timing.

Endebert commented 7 years ago

Changes to the Request class in dev-runtime-browser and dev-runtime-nodejs are definitely needed.

functions for the different HTTP request types are provided (.get(), .post(), etc.), but both implementations fall back to a hardcoded GET request.
there is no way to provide a payload for the relevant request types

Endebert commented 7 years ago

The Catalogue has been updated to support the requested functionality, it can be found in the dev-constraints branch: https://github.com/reTHINK-project/dev-catalogue/tree/dev_constraints

The corresponding version of the RuntimeCatalogue can be found in the develop-constraints branch: https://github.com/reTHINK-project/dev-service-framework/blob/develop-constraints/src/runtime-catalogue/RuntimeCatalogue.js

They were pushed into their own branches just to be sure, although it should be backwards compatible.

RuntimeCatalogue and interaction between Catalogue Broker and RuntimeCatalogue is mostly untested, please provide feedback if you find any problems.

Short explanation:

Catalogue:

can handle multiple descriptors with the same objectName, although with a GET request, only the first one is returned
can handle multiple default descriptors for the same type, although with a GET request, only the first one is returned

constraints have to be enclosed in a JSON object, e.g.:

{'configuration':{'url':'wss://msg-node.fokus.fraunhofer.de/ws'}}

respects constraints when requesting a list of instances for a type (e.g. when requesting /.well-known/protocolstub/)
- in such a case, the returned list only consists of instances that meet the constraints
md5 hashes are used for the URLs pointing to sourcePackages, in order to ensure uniqueness

RuntimeCatalogue:

supports getting descriptors using constraints
supports getting list of descriptors for a type using constraints
should be backward-compatible

emmelmann-fokus commented 7 years ago

Issue resolved. All other related issues are closed. closing this as well

reTHINK-project / specs

Runtime Capabilities and the Catalogue #13