Graph Connector initial version

[pchainho]

The Graph Connector source code is breaking the unit tests and preventing 0.2 release. Pls remove Graph Connector source code from branch dev-0.2. Later, when 0.2 is released, dev-0.3 branch will be created and you can push your source code to this branch.

[fbeierle]

Can you specify how the Graph Connector is breaking the unit tests? So far, it does not interact with other components. Here, the unit tests of the Graph Connecter are running fine. There are some tests failing from Registry, Runtime Catalogue, RuntimeUA, and SyncherManager.

[vitormsilva]

Yes, you are right the tests are failing in that components. But you didn't include in a package.json the GraphConnector dependencies like jsrsasign, sjcl, bip39, bitcoinjs-lib, for that reason we can't run the karma and fixing the errors.

I'm fixing the tests are failing.

[fbeierle]

Ah, ok, thanks. I added the missing dependencies, I hope, it works now.

[fbeierle]

In order to update reachability information, the Graph Connector needs to make calls to the REST-Interface of the Global Registry.

Should the Graph Connector interact with the Global Registry directly or is there some intermediary step necessary, e.g., through the Message Bus?

[pchainho]

The Graph Connector or any other Hyperty Runtime component don't directly interface with external functionalities from the Runtime. Protostubs are the only ones that interface with the "external world". The exception is the runtime catalogue since it is the entry point used to download protostubs. Communication with protostubs are handled by the Message Bus.

However, it is not clear to me that the Hyperty Runtime should directly interact with the Global Registry but through the Domain Registry (this was already discussed somewhere).

For performance reasons we should minimise the number of protostubs deployed in the runtime.

One question: what kind of reachability information do you want to update?

[fbeierle]

With reachability information I was referring to the data stored in the Global Registry: https://github.com/reTHINK-project/dev-registry-global#dataset

Regarding the interaction with the Global Registry you were referring to the discussion here: https://github.com/reTHINK-project/core-framework/issues/129 The result of the discussion was to offer both options: 1) Let the user, i.e., the Hyperty Runtime, handle the interaction with the Global Registry directly. 2) Have a service provider, i.e., a Domain Registry, handle the interaction.

Another point is the contact data the Graph Connector handles. I think, we need the possibility to store it at some cloud storage provider and retrieve it from there, e.g., when the user is changing to a different computer or different browser. I guess, we need some Protostub here as well. Additionally, the Persistence Manager could be useful when re-using the same computer and same browser (https://github.com/reTHINK-project/dev-runtime-core/issues/39).

[pchainho]

With reachability information I was referring to the data stored in the Global Registry: https://github.com/reTHINK-project/dev-registry-global#dataset

nice. However, to be consistent with other data specs, this spec should be moved to the data model repo. One comment regarding this model: to be compliant with the address spec the user URLs should have "user" scheme and not "rethink" eg "user://facebook.com/fluffy123"

Regarding the interaction with the Global Registry you were referring to the discussion here: reTHINK-project/core-framework#129 The result of the discussion was to offer both options: 1) Let the user, i.e., the Hyperty Runtime, handle the interaction with the Global Registry directly. 2) Have a service provider, i.e., a Domain Registry, handle the interaction.

That was a long thread that I missed. If I understood it correctly, the Global Registry client can be anywhere including in the runtime or in some back-end server like the domain registry, which makes sense to me. This also pushes for a single Global Reg protostub that could be deployed in different places including the Message Node and Domain Reg. In this way, the impact of future changes would be minimised.

Another point is the contact data the Graph Connector handles. I think, we need the possibility to store it at some cloud storage provider and retrieve it from there, e.g., when the user is changing to a different computer or different browser. I guess, we need some Protostub here as well. Additionally, the Persistence Manager could be useful when re-using the same computer and same browser (#39).

This sounds like a generic backup service that could be provided by a backend Hyperty on behalf of the User.

[fbeierle]

nice. However, to be consistent with other data specs, this spec should be moved to the data model repo. One comment regarding this model: to be compliant with the address spec the user URLs should have "user" scheme and not "rethink" eg "user://facebook.com/fluffy123"

Ok, I will add it there.

That was a long thread that I missed. If I understood it correctly, the Global Registry client can be anywhere including in the runtime or in some back-end server like the domain registry, which makes sense to me. This also pushes for a single Global Reg protostub that could be deployed in different places including the Message Node and Domain Reg. In this way, the impact of future changes would be minimised.

I'm afraid I'm not familiar with all the details of Protostubs. From what I can tell, your suggestion sounds good. The connection with the Global Registry is needed

• for bootstrapping, the Global Registry Record must be sent
• when calling someone, the GUID needs to be resolved

So, this is an essential part that needs to be done in order for the Graph Connector to work. Can you point me to an existing implementation of a Protostub?

This sounds like a generic backup service that could be provided by a backend Hyperty on behalf of the User.

I consider getting the contact data an integral part of bootstrapping. If we go with a backup Hyperty, this Hyperty would be required to be installed in order to finish bootstrapping. I am unsure of the implications it has opting to implement this as a Hyperty. To me, it sounds like we create a dependency between the Hyperty Runtime / GraphConnector and one specific Hyperty (except for the case someone does not want to load her contacts). What do you think?

[pchainho]

I'm afraid I'm not familiar with all the details of Protostubs. From what I can tell, your suggestion sounds good.

Interaction between the Graph Conector and the GReg protostub will be carried on messages compliant with the Message Model. I suggest the following values for the Header "from" and "to":

from : <hyperty-runtimeURL>/graph-connector

hyperty-runtimeURL is the runtime address that I guess should be passed in the constructor

to: URL.UserGuidURL

This should be compliant with User GUID URL eg user-guid://XXXXMYGUIDXXX

I guess the GReg exposes a Rest API and its mapping to the Message Model should be straightforward. I suggest to put

for bootstrapping, the Global Registry Record must be sent

This sounds like a Create operation to be sent by a Create Message with the record in the Message Body

when calling someone, the GUID needs to be resolved

This should be carried by a Read Message and no body message should be needed.

Can you point me to an existing implementation of a Protostub?

Pls have a look on the vertx protostub and contact @shumy for any support you may need.

[pchainho]

I consider getting the contact data an integral part of bootstrapping. If we go with a backup Hyperty, this Hyperty would be required to be installed in order to finish bootstrapping. I am unsure of the implications it has opting to implement this as a Hyperty. To me, it sounds like we create a dependency between the Hyperty Runtime / GraphConnector and one specific Hyperty (except for the case someone does not want to load her contacts). What do you think?

Not sure I understood but my assumption was that the Graph Connector is always the one creating the contact data. Perhaps, it could be created by importing data from existing services but that could be an Hyperty / App itself or a special runtime UI that would use the Graph Connector API.

I guess, the Graph Connector data backup service would be something optional but the Graph Connector would not depend on that ie the Graph Connector can exist in each user device runtime without backup service. The dependency would be on the other direction ie the Backup service would depend on the Graph Connector.

On the other hand it would be nice to keep the Graph Connector synchronised among the different devices used by the same GUID.

Does this make sense or am I misunderstanding something?

[fbeierle]

To understand the ProtoStub concept better, is this how it is?: A ProtoStub is JavaScript code that is downloaded to the runtime and executed in the browser of the user, sending messages to the MsgNode. In this case, the MsgNode would then call the REST interface of the Global Registry and return the response via the message system towards the runtime.

The runtime only has to call the methods (via a Message?) of the ProtoStub, the MsgNode has to expose an interface to receive the calls of the ProtoStub.

If this is the way it works, each MsgNode must implement this functionality.

The communication would then be: Runtime --Messages-- MsgNode --REST-- Global Registry

For the implementation, this would mean, we need something in the Runtime (finding the ProtoStub, downloading it, and calling its methods), the ProtoStub itself, and something in the MsgNode (retrieving and handling the messages from the deployed ProtoStub). Who constructs the messages? The ProtoStub? Or does the Graph Connector have to do it and pass the message to the ProtoStub?

[rjflp]

So far I had assumed that the use of the MsgNode was only required to bridge two peers that may be behind a NAT box. Last meeting, I heard that in a NATless, IPv6 future, we could do without the messaging node. If this is indeed so, we could communicate directly with the GlobalRegistry without going through the MsgNode. I can certainly bypass the MsgNode to communicate with a server. In the case of the Domain Registry this is a non issue as both the MsgNode and the Domain Registry are operated by the same entity.

An alternative would be to use: Runtime (runtime core or hyperty?) - protostub - REST - Global Registry

The advantages would be: -Not having to deal with the particularities of the several messaging node being used. -Having the ability to update the hyperty as we change the security consideration of the global registry (e.g. using several DHT nodes to perform a query and using consensus to prevent an eclipse attack). -Not having to go back and forth with messages, as we would not share the private key with the messaging node. -In general it would be easier to update an hyperty than the several messaging nodes.

[pchainho]

I would say, both are possible and can be supported with a protostub, but for a first version, having a Global Registry protostub is better.

From the graph connector perspective, it would just have to send the messages I've mentioned in my previous comment.

[pchainho]

@fbeierle recently you have been updating the Graph Connector data model. To let you know that the data model has moved to dev-service-framework according to https://github.com/reTHINK-project/core-framework/issues/157

This means new updates of the Graph Connector data model must be done here

[fbeierle]

Ok, thanks.

I have two questions:

• Is there a developer guide for ProtoStubs?
• What is the proper way for a runtime component to send messages? From what I could gather from the existing code: Provide a method to set a MessageBus and create JSON objects as messages. Is this correct?

[acheambe]

I have no answer but a suggestion on creating messages:

Provide a method to set a MessageBus and create JSON objects as messages. Is this correct?

Would be good to use the message factory from the dev-service-framework to create messages

[pchainho]

Is there a developer guide for ProtoStubs? Not yet, but is planned for D3.2. As mentioned above, Pls have a look on the vertx protostub and contact @shumy for any support you may need.

What is the proper way for a runtime component to send messages? From what I could gather from the existing code: Provide a method to set a MessageBus and create JSON objects as messages. Is this correct?

In general this is correct.

[pchainho]

what is the status of the Graph Connector?

[fbeierle]

I'm waiting to commit the MsgNode -> GlobalRegistry Connector, see https://github.com/reTHINK-project/dev-msg-node-vertx/issues/12 I'm still working on the messages the Graph Connector has to sent and writing tests for it.

Furthermore, I think, the Graph Connector has to be instantiated in RuntimeUA.js, right?

[fbeierle]

I finished the code for the messages the Graph Connector sends for the Global Registry.

As for the instantiation of the Graph Connector for the RuntimeUA, does it just need to be set as a property in the constructor of RuntimeUA: https://github.com/reTHINK-project/dev-runtime-core/blob/dev-0.3/src/runtime/RuntimeUA.js#L33?

[fbeierle]

@vitormsilva or @pchainho, are you able to help me here?

[vitormsilva]

Hi @fbeierle,

I think so, there you have all dependeces you need;

[vitormsilva]

We realize the runtimeUA distribution file, at this moment, have 1Mb of size.

This is a huge size, but when we try to discovery from where these size appear, we notice without the GraphConnector the distribution file reduce it size to 200Kb.

I don't know what the problem was, but all the third party modules are necessary? Since graph connector is not used yet, we would like to remove it instantiation from the master branch.

[fbeierle]

In the Graph Connector, we only added those 3rd party modules that are necessary.

Since graph connector is not used yet

I am unsure what you mean by this. For WP5, we definitely need the Graph Connector. It is a vital part of the reTHINK bootstrapping process, as it creates the GUIDs and communicates with the Global Registry.

I am not very familiar with the process that creates the runtimeUA file. Maybe this process can be optimized to reduce the resulting file size?

[vitormsilva]

Hi @fbeierle,

Regarding the runtimeUA file, the file was uglified and all spaces removed, to increase that optimization process. I removed all dependencies from runtimeUA source code, and add one by one, and when I added the GraphConnector the size of the file increase from +/- 285Kb, to +/- 958Kb.. you can test in your runtimeUA file. You can look at the gulp task, and can help me to optimize the process if something is not right we can change.

The runtimeUA file can be optimized, not inside the runtime-core itself, but, the most size of the file come from dev-service-framework, in there, probably, all of us need to optimize something.

[fbeierle]

Thank you for the detailed explanation. I think it is important to note that the size is not a bug but simply a result of using several external libraries whose functions are copied to the runtimeUA. Maybe it could be an option to generate two files. One that contains new implementations from the reTHINK project and one that contains the needed code from third party libraries. The third party code would most likely not changed very often and the browser could use a cached version.

[vitormsilva]

Yes it's a good hint, but i don't know if it works, because the runtimeUA should be distributed without external dependencies. The runtime will be downloaded like an hyperty.

[fbeierle]

Currently, we are working on an example showing the inter-working of GraphConnector and Global Registry. For this, I need to send a message to the (Vertx) Msg Node, which will handle the message via the GlobalRegistryConnector. So far, as an addressing scheme, global://registry/ is used (see https://github.com/reTHINK-project/dev-runtime-core/blob/dev-0.4/src/graphconnector/GraphConnector.js#L224). This causes the RuntimeUA to look for a ProtoStub matching the domain "registry" and such a ProtoStub is not found. How can I send a message to the Msg Node?

[pchainho]

The RuntimeRegistry.resolve() was updated to support global://registry. pls have a try

[fbeierle]

Thanks, it works now.

[pchainho]

What is the status of this one?

[fbeierle]

In runtime-core, all the necessary functions are included and tested. The branch is here: https://github.com/reTHINK-project/dev-runtime-core/tree/develop-graphConnector, I merged the latest changes of the develop branch.

As talked about in today's WP4 call, here, we provide a visualization of the message flow for the integration of the Graph Connector with the Global Registry: https://github.com/reTHINK-project/dev-addressbook-app/wiki

@reTHINK-project/wp3 @reTHINK-project/wp4

I will update this issue when the runtime-browser integration is finished.

[pchainho]

The Runtime Core Components like the Graph Connector shouldn't be reached by the App

I would recommend you to have this GUI as an admin GUI similar to Policy Engine and Id Module admin GUI. Then the Admin GUI are inside the core sandbox and have access to the core components API.

This issue was created for that

[fbeierle]

This is the way we have been implementing this since quite a while, see, e.g., the Pull Request from June: https://github.com/reTHINK-project/dev-runtime-browser/pull/52. The address book app is not just a simple Admin GUI, but an app we develop for WP5.

Furthermore, it seems like the right way to use the runtime-core to send messages to the MsgNode in order to query the Registries, see here: https://github.com/reTHINK-project/dev-discovery/issues/2#issuecomment-243621828.

[pchainho]

I was not aware of your design before. The runtime was designed a long time ago (pls see D3.1) fullfilling strong security requirements. Different types of components, notably core runtime components and protostubs, are executed in isolated sandboxes. Core Runtime components and protostubs, are only reached through the Message bus and message bus endpoints are only provided to components deployed and managed by the runtime itself. That's why Apps are not able to send messages to be routed through the MN.

The admin GUI was conceived early this year to let the user safely manage its runtime core functionalities including the graph connector. I don't like the idea to let Apps write my graph and to be able to generate (delete?) my guid.

One option, would be to have one lib to be imported by Hyperties to read the Graph Connector, then Hyperties could expose some API to Apps. This is something similar to the current discovery lib. We are also considering having these libs fully encapsulated by (Support) Hyperty eg a Disovery Hyperty.

But I would still keep writing access into the Runtime Admin GUI.

I think the effort is very limited and we would give you all the support needed

[fbeierle]

Thanks for your replay. I have a few questions:

What is the technical difference between an Admin GUI and an App? Aren't both HTML/JS that anyone can write? Or am I missing something?

What is a lib in the reTHINK context and how do they work? How would the lib be able to access the Runtime Core/Graph Connector?

[pchainho]

What is the technical difference between an Admin GUI and an App? Aren't both HTML/JS that anyone can write? Or am I missing something?

The Admin GUI is inside the Core Runtime Sandbox (rethink iFrame) and have access to all core runtime components, while the App is outside and does not have access to any core runtime components. This means the Admin GUI is bundled with the Browser Runtime dist and only the Runtime Provider is able to designe/ implement it. Something similar to a Browser settings.

What is a lib in the reTHINK context and how do they work? How would the lib be able to access the Runtime Core/Graph Connector?

Libs used by Hyperties are just normal javascript libs that can be imported by Hyperties. The only possible difference is that may need access to the msg bus (sandbox minibus) endpoint to send / receive messages. For example, the discovery lib has the minibus passed in the constructor.

[pchainho]

What is the status of this one?

[fbeierle]

Everything used to work fine, but some changes somewhere (service framework? global registry?) led to another issue that we're currently working on: https://github.com/reTHINK-project/dev-runtime-browser/issues/62

@ishantiw @SenanSharhan @sgoendoer

[pchainho]

as clarified here this issue should be solved having an Admin GUI for the Graph Connector and using the last version of the Toolkit for the development.

Could you update us on this pls?

[fbeierle]

The first version of the Address Book Admin GUI is available. To run it with the dev-hyperty-toolkit:

• Checkout dev-hyperty-toolkit, runtimes branch
• Change the version of runtime-brwoser in package.json in dev-hyperty-toolkit to: "runtime-browser": "github:rethink-project/dev-runtime-browser#develop-GraphConnector-Test",
• run "npm update"
• run "npm run start:browser"

There are some bugs introduced by changing from App to Admin GUI, that's why I didn't create new pull requests for runtime-core and runtime-browser yet.

[fbeierle]

Update: easier way to run it: checkout branch "addressbook-app" from dev-hyperty-toolkit and run "npm run start:browser"

[fbeierle]

Update: We integrate the storage manager for the Graph Connector and also integrated the discovery service in the Address Book.

Pull requests for master branches in runtime-core and runtime-browser: https://github.com/reTHINK-project/dev-runtime-core/pull/157 https://github.com/reTHINK-project/dev-runtime-browser/pull/72

To run (same as before): checkout branch "addressbook-app" from dev-hyperty-toolkit and run "npm run start:browser"

[pchainho]

since I think this already merged I'm closing