Build/package management issues on Arch/UBOS

jernst commented 6 years ago

I'm attempting to build the gateway on Arch Linux x86_64, with the eventual goal of creating a @uboslinux package for it on x86_64, armv6, armv7 and aarch64. I'm following the instructions in https://github.com/mozilla-iot/gateway/blob/master/README.md and here are the issues/questions I have encoutered so far:

When running yarn, there's: warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json. Our build can delete this file first, but is there a better way?
For a clean build, it would be good to override node-gyp with node-gpy --devdir <someplacelocal>. Is there a good place where to add this flag? Same for .yarnrc and its local files.
It appears the gateway stores its info in $HOME/.mozilla-iot. How can I get it to use a system service-style directory, like /var/lib/mozilla-iot?
What are the valuable files that need to be backed up and restored?
It appears the gateway uses SqLite. How would we use a different database such as MySQL/mariadb or PostgreSQL? Or is that not advisable?
How does data migration work from one version of the gateway to the next? Some apps have command-line scripts for that, e.g. mediawiki does.
What's nanomsg used for? Is that intended for production?
Is there a way of running the gateway at a context path that's not the root, e.g. example.com/gateway?
Is there a way to configure the gateway from a script, rather than through the web?
How would we disable, or preferably not include, the code that does:
1. Letsencrypt
2. Pagekite
3. Enables the user to turn on/off ssh?
It would be useful if the systemd.services that are apparently generated in https://github.com/mozilla-iot/gateway/blob/287d1ed912ebbb0d0e34e877143647146911c130/image/prepare-base.sh would be separate files. Generation like that tends to (and in the cases I can think of, like our's) conflict with distros' package managers's desire to clean up nicely for upgrades or uninstalls.
The gateway appears to include a lot of "drivers" for different communications protocols. How would we build the gateway so that only a named set of them is included in the build? (e.g. are there flags perhaps like --with-XXX or --without-XXX) Or preferably, so that all of them are separate modules/plugins/accessories? (This would allow a user to configure the gateway minus those drivers that they don't have devices for, and don't have intentions to have, so they can't be attacked that way, for example.)

Thanks for indulging me :-)

mrstegeman commented 6 years ago

Thank you for your thoughts! I've added some comments below.

When running yarn, there's: warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json. Our build can delete this file first, but is there a better way?

Internally, we use yarn during development. However, we recently added back the package-lock.json file to support users/developers who don't necessarily want to install yarn. We try to keep the two lock files in sync, so there should (theoretically) no difference between the two.

For a clean build, it would be good to override node-gyp with node-gpy --devdir <someplacelocal>. Is there a good place where to add this flag? Same for .yarnrc and its local files.

Not sure...

It appears the gateway stores its info in $HOME/.mozilla-iot. How can I get it to use a system service-style directory, like /var/lib/mozilla-iot?

This location can be edited in the config/default.js file.

What are the valuable files that need to be backed up and restored?

Only the things in the above profile directory.

It appears the gateway uses SqLite. How would we use a different database such as MySQL/mariadb or PostgreSQL? Or is that not advisable?

The database layer could be abstracted to use another database, but there won't be much gain. We don't store a ton of data in the database right now.

How does data migration work from one version of the gateway to the next? Some apps have command-line scripts for that, e.g. mediawiki does.

All of the version migration work is done in src/user-profile.js

What's nanomsg used for? Is that intended for production?

nanomsg is the IPC layer for our add-on system. All add-ons run in their own process, so they communicate back with the gateway via nanomsg.

Is there a way of running the gateway at a context path that's not the root, e.g. example.com/gateway?

Not currently, but that would be a nice addition. I've filed #1509 for that.

Is there a way to configure the gateway from a script, rather than through the web?

We don't have a script for doing so, but all configuration is done in config/default.js, and after #1502 is merged, in ~/.mozilla-iot/config/local.json.

How would we disable, or preferably not include, the code that does:

Letsencrypt

Pagekite

Enables the user to turn on/off ssh?

LE is only used when using Mozilla's tunneling service. If the user chooses not to set up a .mozilla-iot.org subdomain during first time setup, LE is never used.
Same as LE.
The SSH toggle is currently only implemented and enabled for the Raspberry Pi.

It would be useful if the systemd.services that are apparently generated in https://github.com/mozilla-iot/gateway/blob/287d1ed912ebbb0d0e34e877143647146911c130/image/prepare-base.sh would be separate files. Generation like that tends to (and in the cases I can think of, like our's) conflict with distros' package managers's desire to clean up nicely for upgrades or uninstalls.

I agree. I've filed #1510.

The gateway appears to include a lot of "drivers" for different communications protocols. How would we build the gateway so that only a named set of them is included in the build? (e.g. are there flags perhaps like --with-XXX or --without-XXX) Or preferably, so that all of them are separate modules/plugins/accessories? (This would allow a user to configure the gateway minus those drivers that they don't have devices for, and don't have intentions to have, so they can't be attacked that way, for example.)

What you're talking about is add-ons. We only install default add-ons for the prebuilt Raspberry Pi image. Outside of that, the gateway does not come with any add-ons by default. They are installed by the user.

jernst commented 6 years ago

Thanks @mrstegeman, this is very useful! Some follow-up:

Is there some documentation how the add-on's interact with the main app? I'd like to create a package for the core app, and a separate package for each add-on. If I say yarn presumably it builds all? Where do I slice and dice to figure out what belongs to what?

dhylands commented 6 years ago

This page documents the IPC messages between the adapters and the gateway. https://github.com/mozilla-iot/wiki/wiki/Adapter-IPC

We already bundle the addons into packages. This file has the URLs of the packages, and these pacakages are what the gateway installs. https://github.com/mozilla-iot/addon-list/blob/master/list.json

We have an addon-builder repository that builds the addon packages. You can find that repository here: https://github.com/mozilla-iot/addon-builder This builds addons which need native support of some type and aren't pure JS. The addon-builder runns the package.sh script which is included with each addon.

WebThingsIO / gateway

Build/package management issues on Arch/UBOS #1507