joshco commented 4 years ago

FYI The wot library has a lot of depth to it, it was neat to see the event subscription using the plain http binding just work out of the box. It's really more than a reference implementation. kudos.

I wrote an example exposed thing that is hosted in a NodeJS application, using express for the browser UI, and wot for the exposed thing, which is controlled by browserified node-wot. When deploying it in the cloud, I ran in to issues with HTTP PORT mapping and the BASE URL for the TD URLs, which required changes to the http-binding to get it working.

If there's consensus on how to address these, I'm happy to create a PR.

HTTP PORT

With the thing embedded in NodsJS, there's two http listeners i a single process. This all worked fine on my development system, 3000 (express) and 8081 (wot)

In the cloud, I use Dokku, which is a docker based "heroku in a box" that you can run on your own ec2 instance. It is heroku compatible and uses the heroku build-packs. https://github.com/dokku/dokku

My first deploy got me this:

    > lightbar@0.0.0 start /app
       > node ./bin/www
       HttpServer starting on port 8081
       Port 5000 is already in use
       npm ERR! code ELIFECYCLE
       npm ERR! errno 1

(The full heroku buildpack output is here: https://gist.github.com/joshco/e5fbf2b7167504fbc66b6ffc83b447cc)

Port 5000 is the port the build-pack sets in an environment variable that the application will use. (this is the same as the rails and other build-packs) When I did a curl to port 5000, i got back the wot servient response, where I expected the Node Express app to be.

8081 is the port I specified in the WOT http config, and where I expected the WOT to be. When I did a curl to 8081, it wouldn't connect.

After troubleshooting, it turns out that WOT is fooling us. In the start method of http-server, it's logging that it is going to use the port specified in the http config (8081), but later in the code, in line 121 it gives preference to the PORT env variable if it is present. So it's actually bound to 5000, wining the race, leaving express to get the error.

To make my project work, I changed the code from

(+process.env.PORT || this.port, this.address); To +process.env.WOT_PORT || +process.env.PORT || _this.port;

In http-server.ts https://github.com/eclipse/thingweb.node-wot/blob/b9d0a3d85448d2d0652dcd3cfe5980ad5c12fd39/packages/binding-http/src/http-server.ts#L100-L123

I added a WOT_PORT so the preference is WOT_PORT (wot specific) | PORT (heroku style generic) | _this.port (wot http config)

Base URL

The other challenge, which I think many will run into when using docker or other heroku type systems, is making the servient provide the right URLs in the TD et al.

`"forms":[{"href":"http://192.168.1.111:8081/WOT/properties/color","contentType":"application/json."`

The servient in the container can only work with the network interfaces on the container. So that means a domain name or IP that maps to one of the local interfaces, which is not the internet facing interface, or domain name. Attempting to use an external domain name in the server.address field or STATIC env var of the CLI gets EADDRNOTAVAIL

To fix this, I modified the expose method in http-server.ts where it defines the base variable it uses to create URL strings:

From

let base: string = this.scheme + "://" + address + ":" + this.getPort() + "/" + encodeURIComponent(title);
let href = base + "/" + this.ALL_DIR + "/" +  encodeURIComponent( this.ALL_PROPERTIES);

To

var domain_url = (process.env.WOT_URL_BASE|| (this.scheme + "://" + address + ":" + this.getPort()));
var base = domain_url + "/" + encodeURIComponent(title);

https://github.com/eclipse/thingweb.node-wot/blob/b9d0a3d85448d2d0652dcd3cfe5980ad5c12fd39/packages/binding-http/src/http-server.ts#L190-L193

PS: If you want to see the thing: http://thingpatrol.dev.joshco.org

relu91 commented 4 years ago

I am ok with changing the env name to WOT_PORT is more specific and should avoid issues like yours.

Although, about the base URL there more "complex" strategies that are been evaluated, see @sebastiankb comment over here. Also, have you tried to specify the base property in your thing description? I am not sure if this mechanism is implemented inside node-wot.

Furthermore, in our docker setup, we needed to ADD a form instead of modifying the existing generated URLs. This is because we still need to exploit intra-container communication but in addition, we want to access the servient from the outside.

PS: If you want to see the thing: http://thingpatrol.dev.joshco.org

PS: nice UI 👍

joshco commented 4 years ago

I'll check out the base property. Re WOT_PORT, changing it makes sense. Another possibility is to leave PORT, but add WOT_PORT and give it preference. That lets PORT work if node-wot is the only server in what they are deploying and work with the standard heroku build packs. either way...

joshco commented 4 years ago

It does not appear to be implemented, see below. Looking at possible implementations, the existing code enumerates all IP interfaces, and one could argue that a single baseURI should not apply to all interfaces. My own solution has the same problem. as related to #803

My result of using base in stock wot-node: I added a base URI, which is defined as a string:

   return servient.start().then((WoT) => {
        WoT.produce({
            "@context": "https://www.w3.org/2019/wot/td/v1",
            title: "WOT",
            base: "http://wot2.loopback.site",
            properties: {

When I view the TD raw, in the browser, I see:

{
  "@context": "https://www.w3.org/2019/wot/td/v1",
  "title": "WOT",
  "base": "http://wot2.loopback.site",
  "properties": {
    "color": {
      "type": "string",
      "format": "color",
      "readOnly": false,
      "writeOnly": false,
      "observable": false,
      "forms": [
        {
          "href": "http://192.168.1.111:8081/WOT/properties/color",
          "contentType": "application/json",
          "op": [
            "readproperty",
            "writeproperty"
          ]
        },
        {
          "href": "http://192.168.50.177:8081/WOT/properties/color",
          "contentType": "application/json",
          "op": [
            "readproperty",
            "writeproperty"
          ]
        },
        {
          "href": "http://10.0.75.1:8081/WOT/properties/color",
          "contentType": "application/json",
          "op": [
            "readproperty",
            "writeproperty"
          ]
        }
      ]
    },

When I use the browserified wot in my client, and look at chrome dev tools, it does not seem to be paying attention to the base attribute.

In the stock wot library http-server code, I don't see any other definition of base, except: this.scheme + "://" + address + ":" + this.getPort()

danielpeintner commented 4 years ago

I did not dig into the code but I believe the strategy in node-wot was (and still is) that once a TD is processed the base field is populated in all forms which makes processing a lot easier. The full uri is there where you expect it to be.

Maybe it makes sense to rethink this strategy and leave base as is and pass around besides the form also a possible base.

egekorkan commented 4 years ago

So you can use the static address in the servient configuration to set your custom URL. Also, the base is given per TD whereas the static address is given for a servient that can host multiple Things with TDs that each have multiple forms with different base URIs. There can be different cases such as:

base is given in all TDs of the servient and no static address is given for the servient. This is rather easy
base is given in some TDs of the servient and no static address is given for the servient. This then needs a decision: Should the TDs with no base be generated with automatic URIs or should they somehow get the base given in one of the TDs? For the latter, what happens when the base values are different?
no base is given but static address is there. This is the current case and all the TDs of the servient will have the static address
base and static address is given. Currently, the base is ignored and it should probably be?

joshco commented 4 years ago

Address

When I tried to use address as a custom URL, eg 'mysuperthing.com", it failed with EADDRNOTAVAIL. eg, it tried to resolve the hostname to an IP address, expecting the IP to be a local interface that it could bind a listen socket to.

Canonical URI

One thought I had is if perhaps a "CanonicalURIBase" property would be clearer. If all URLs had that prefix because it was "the official" consumer facing location, that would make sense to be present in all listeners in a servient.

No matter which interface you came in and got the TD on, the official location is CanonicalURIBase + path. For container-container access, it would work, but be inefficient since it would loop around the external interface.

Virtual URI

Another thought is to use something like this is as an additional URL that shows up in the TD , properties and forms, but isn't actually it's own listener. it's assumed to be somewhere else like on an internet facing gateway, or heroku/docker/nginx/proxy/magical deployment platform. Let's call it virtual.

Check me if I'm wrong, but the consumer seems to pick the right addresses/urls to connect. Eg my development server has 4 IPs and http listeners, but the consumer which is actually on a different machine picks an interface it can actually get to, vs an inaccessible interface on the dev server. Is this coincidence or is it really doing that?

So having an additional virtual URL in the TD definitions would allow the public facing canonical URL scenario (as i ran into) to work, since the consumer can tell its current URL and recognize what it should use in the TD.

It could also allow direct container to container access without the inefficiency, since those consumers would just ignore the canonical URLs since it doesn't know anything about them. The consumer has presumably been configured with an initial endpoint to consume from, and it can use that to select it's next link, which would be the container-container path.

Layers

Finally, it seems like there is a coupling in the http binding between the TCP/IP layer and the HTTP layer which is different than what one would find in web servers like nginx/apache and their virtual hosts. The web server will listen on any IP/hostname that it is told to via bind address, but that's independent from the URLs it will map, serve, or proxy to other backends, making use of the HTTP Host header.

joshco commented 4 years ago

@danielpeintner @egekorkan @relu91 I built new app, a PowerPoint remote controller. This time, the URLs in the forms are in a different order. The node-wot consumer kept picking the first one, which is unreachable. Strangely, the original app has the right one first. Still looking into this.

The controller runs on your PC with PowerPoint, and builds an ngrok tunnel for you. It will let you use a browser node-wot on your phone as a presentation clicker.

It's pretty slick, you just

Start PowerPoint with your presentation loaded
run npm install ; npm start
The console output will have a QR code to scan with your phone. that will take you to the browser client, talking to your pc via auto-built ngrok tunnel
Once the browser is loaded, click 'consume' on the browser and you'll have control buttons and status.

See UX here: https://github.com/joshco/wot_powerpoint

Please try it out. Let me know if there are issues. Maybe next time someone is giving a presentation on WoT, give it a try.

Mac users aren't left out either. The underlying node module 'slideshow' says it can support macOS with PowerPoint and Keynote. If anyone with a mac can give feedback, that would be helpful.

danielpeintner commented 4 years ago

Question: pointing to local repo https://github.com/joshco/wot_powerpoint/blob/master/package.json#L11 is because of the issue you are encountering?

joshco commented 4 years ago

@danielpeintner Yes, your assumption is correct. Also, while the BASE changes worked, the PORT changes that I made in SNAPSHOT 6 didnt work in SNAPSHOT 7. I'm getting an unhandled promise rejection. So i couldn't import the tip of the branch. I think i understand why now, but the wot_powerpoint will be upgraded once that's settled. Have you tried it?

eclipse-thingweb / node-wot

HTTP PORT and BASE_URL deployment issues #214

HTTP PORT

Base URL

Address

Canonical URI

Virtual URI

Layers