Only partially working out of the box on my ubuntu laptop

goatchurchprime commented 4 years ago

Cloned the repo, disabled any services attaching to the ports 1880, 1883 and 3000, then:

Do this with conda activated

pip install docker-compose docker-compose up -d big install docker-compose logs -f

MQTT works (with these commands in different windows):

mosquitto_sub -h localhost -v -t "#" mosquitto_pub -h localhost -t "hi" -m "there"

Node-RED now appears to work:

http://localhost:1880 You need to use "mosquitto" as the broker, not localhost.

The influxDB node is missing, so I can't make database data.

Jupyter also didn't come live for a while on http://localhost:8888/lab with password mingstack. Then does after a few minutes. It starts out with no syntax highlighting and nothing will run. Now it does.

Grafana comes in on http://localhost:3000 default user/pass=admin/admin and then you are forced to change it.

I was hoping to run the following to connect+download from the grafana database using the following python code:

import influxdb client = influxdb.DataFrameClient("192.168.43.1", 8086, "root", "root", "balena-sense") k = client.query("select * from fridgepower order by time desc limit 10")

Some needs

Where is the /data file connected to externally that the jupyter lab could connect/load/save from?
install influxdb into python
Inclusion of node-red serial
Inclusion of pyserial in jupyter
Inclusion of jupyter-micropython-kernel
Provision the system with an initial flow (eg doing random numbers) saving to an initial database with an initial dashboard.

MatthewCroughan commented 4 years ago

This was due to Julian being on an older version of docker (docker 17.03), not much could really be done. It might be worth putting some prerequisite dependencies in the Readme, such as what docker version should be in use and a link to how to install Docker.

ajlennon commented 4 years ago

@goatchurchprime can you retest this with whatever Docker version @MatthewCroughan says you need and revise whatever outstanding issues there are?

MatthewCroughan commented 4 years ago

@ajlennon we already went through this in telegram, I had him uninstall all old versions of Docker then run https://get.docker.com/.

Should mark this issue as resolved once we document this in the README.

ajlennon commented 4 years ago

There are a bunch of other issues up there about things that need to be installed by default (like the InfluxDB nodes)

goatchurchprime commented 4 years ago

Things are getting close for an amazing package that knocks people out. I'm looking for an instant installation of the full stack pre-loaded with all the tools on the go. There may be multiple installs of the MING stack on personal computers, that can push data around via mosquitto brokers, until it's understood that one should do it with one lab install (properly backed up) that everyone connects to.

Having been around to some researchers, we need things the serial-ports to work, because that's how lab-equipment is often operated. Also, there are some systems that only have windows interfaces (through the same serial-COM ports) that dump their data into a file. Node-red should be able to tail those files and adjust its control behavour. The jupyter-influxDB integration makes it possible to pick up and analyse the data from the experiments when they are run.

Chris says that wsl2 means we should be able to get docker in linux on windows almost natively.

The notes were made after we'd sorted out the docker version:

Docker version 19.03.5, build 633a0ea838

Docker-compose only runs when I have conda activated, then:

docker-compose version 1.25.4, build unknown

Overcoming missing components (that should be loaded by default)

Node-red: Installing node-red-contrib-influxdb using manage pallette

This should have a couple of example flows to kick it off, like a repeating inject node that passes the following javascript payload into the influxdb database:

msg.payload = [ { "xxx":Math.random()*2+1}, {"yyytagtype":"zzztag"} ];

I am using database=aaadatabase, and measurement=bbbmeasurement to make it clear that those names are user-defined and not mistaken for keywords.

Influxdb: This should boot up with one default database in it. The http://localhost:8086 link, which is supposed to be a dashboard for it, gets 404 page not found. I don't have the command line login to work it. See Jupyter section for how I added a database.

Jupyter notebook:

The influxdb connection

[install the module]
import subprocess
print(subprocess.check_output(["pip3",  "install", "influxdb"]).decode())
[now restart kernel]

import influxdb
client = influxdb.DataFrameClient("influxdb")
client.create_database("aaadatabase")

[wait for the nodered to add a bunch of random values]
c = influxdb.DataFrameClient("influxdb", database="aaadatabase")
x = c.query("select * from bbbmeasurement order by time")
z = x["bbbmeasurement"]
display(z)

from matplotlib import pyplot as plt
plt.plot(z.xxx)

The serial port connection:

import subprocess
print(subprocess.check_output(["pip3",  "install", "pyserial"]).decode())

import serial, serial.tools.list_ports
lp = list(serial.tools.list_ports.grep(""))
lp.sort(key=lambda X: (X.hwid == "n/a", X.device))  # n/a could be good evidence that the port is non-existent
ports = [x.device  for x in lp]
print(ports)

Micropython

If we can get serial connection, we might as well install the micropython kernel. There might be some other kernels to add in, eg Julia, which is a very popular language among academics.

Other node-RED components

Install node-red-node-serialport (this won't work till we solve it for the Jupyter container).

Install node-red-dashboard (hooray! this works).

We should have a default node-flow showing off the dashboard with a couple of dials on it because getting it going

ajlennon commented 4 years ago

Yeah agreed. There's a bunch of stuff that should be installed out of the box on node-red

MatthewCroughan commented 4 years ago

@ajlennon I disagree, it should be an optional or build argument, it should be a feature that can be activated at build time, but why would we install things by default for the user rather than letting them import their own stuff?

MatthewCroughan commented 4 years ago

We could also make an additional project that uses this project as a submodule, MING then becomes a base layer that is pulled in, MING with absolutely no addons. We then build features on top.

MatthewCroughan commented 4 years ago

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.

The instructions you have provided are more than enough to get on with this. Thanks.

MatthewCroughan commented 4 years ago

Influxdb: This should boot up with one default database in it. The http://localhost:8086 link, which is supposed to be a dashboard for it, gets 404 page not found. I don't have the command line login to work it. See Jupyter section for how I added a database.

The default database name is ming_default which is specified in the docker-compose file: https://github.com/DynamicDevices/ming/blob/4c77867e654a5bb35ee6d450e5d0c6d5ca3fa8b8/docker-compose.yml#L19

ajlennon commented 4 years ago

@ajlennon I disagree, it should be an optional or build argument, it should be a feature that can be activated at build time, but why would we install things by default for the user rather than letting them import their own stuff?

The value of Docker / MING is that users can get going immediately without faffing about or having to understand what to configure or turn on or which buttons to push.

I see no downside in activities that people are likely to want to undertake (e.g. using serial ports) having support installed and ready to go...

ajlennon commented 4 years ago

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean.

The instructions you have provided are more than enough to get on with this. Thanks.

Not at all keen on forking MING. There's no downside to having all this installed and ready to go in the standard build

goatchurchprime commented 4 years ago

In Node-RED set the Influx node to: influxdb:8086, database: ming_default

In Grafana we need a standardized password. And then to be provisioned with a link to the ming_default database. Then it needs a panel view into the

Use as a function to seed the database with: Math.sin(Math.PI2(((new Date()).getTime()%60000)/60000) + 1 + Math.random()*0.2;

May seed with two functions to show how values can interact in the same table.

Where are these /data directories? Can they get shared with the outer file system?

Jupyter-lab has a Terminal window, so we can install anything we need there, though it doesn't do the control-Up history commands.

Also can't run apt-get install, because I want Git to see if I can get micropython.

Version of JupyterLab may be out of date. My personal one says this on startup:

Build Recommended JupyterLab build is suggested:

jupyter-threejs needs to be included in build jupyterlab-datawidgets needs to be included in build

This could mean that the ipyvolume examples might work: https://ipyvolume.readthedocs.io/en/latest/

I wonder what happens if I do this docker-compose up a second time:
It just says it's all up to date.

May need the cheatsheet for closing down and opening up and updating the system.

Also, I'll need scripts to help backup the different components, etc. The kind of guidance and workflows.

other keep safe commands:

Clone https://github.com/DynamicDevices/ming

In the directory, do:

docker-compose up -d

This shows what's up and running:

docker ps

To list the images:

docker images

To log what's going on, do:

docker-compose logs -f

To kill all docker containers and images:

echo "Nuking docker containers, images, and volumes." && docker stop $(docker ps -aq) || true && docker rm $(docker ps -aq) || true && docker rmi -f $(docker images -aq) || true && docker volume rm $(docker volume ls -q) || true

ajlennon commented 4 years ago

Also can't run apt-get install, because I want Git to see if I can get micropython.

Why not? Have you tried apt-get update first?

ajlennon commented 4 years ago

@goatchurchprime the hostname to use for servers is the name of the container (as you say above) rather than localhost. This is correct behaviour and should probably be written down somewhere

goatchurchprime commented 4 years ago

When we get some of these issues sorted out, I'll start a habit of building it up from scratch and tearing it down on different computers to prove it works.

To make these tests easier, we need to provision it with some exemplar node-red flows and dashboards to make these tests quicker. Just try it out like this way for a while with the whole kitchen sink.

Then if it doesn't feel right we can fork off into derivations and all that complexity. But not while the dividing line is unclear (eg should the ming_default database provision be in the clean upstream copy, or the enabled downstream copy?) It just causes hassle of managing two repos and potentially fixing up both of them when stuff is in flux.

MatthewCroughan commented 4 years ago

@goatchurchprime What you're looking for is not to modify the base image of MING so that everyone in the world ends up with a bunch of lab software in their MING install, but to fork MING and use it as a base to make your own embedded package that can be installed for your own needs, with your own extra software, which is what this is all about. I'll help you out with this in person tonight and show you what I mean. The instructions you have provided are more than enough to get on with this. Thanks.

Not at all keen on forking MING. There's no downside to having all this installed and ready to go in the standard build

I am thinking now that the addition of obvious things like the influxdb node in Node-Red since we provide an InfluxDB container so it should be there by default.

But there are obvious things that should not be included by default because they have not much to do with the stack as a whole and just add a whole new level of complexity to maintaining MING.

For example, when working on https://gitlab.com/windfactory/hullbuoy I needed to add a bunch of nodes to node-red. So I simply appended this to the dockerfile:

RUN npm install node-red-contrib-influxdb \
    node-red-node-email \
    node-red-contrib-osc \
    node-red-node-random \
    node-red-contrib-loop-processing \
    node-red-contrib-time-range-switch \
    node-red-node-pi-gpiod

At no point would I say this should be in the MING default, but someone might think it does. At this point you're supposed to make a new layer, or simply add it yourself for your own needs rather than include it in the base layer.

Installing the node-red-influxdb-contrib makes sense, because we provide Influx.

Installing node-red-node-serialport and node-red-dashboard is sensible if it's functionality that a lot of people use and want out of the box.

But let's say Julian wanted to install node-red-node-random, principally he's supposed to take MING and add that to his own fork, which is his own embedded device/package which uses that extra thing because there are an infinite amount of nodes and possibilities for things that he and his project might want or need.

At some point it would get ridiculous and should not be included in the base, you should go off and copy MING and add those things for your own purposes/project.

That said, I think is right to include all of the things Julian has mentioned since they are small and useful additions that are highly geared towards the use case of everyone using MING. I will however add a build arg to not include them. That ARG would be barebones or something of that nature, so that when building people can use docker build --build-args=barebones and remove the un-necessary componentry.

I'll submit a PR later tonight or tomorrow with these changes for review.

MatthewCroughan commented 4 years ago

Also can't run apt-get install, because I want Git to see if I can get micropython.

The only container in use that is based on Debian is Jupyter. And as mentioned by Alex, you do need to run apt update the first time the container is ran before you can use apt.

MatthewCroughan commented 4 years ago

Where are these /data directories?

They're volumes. Volumes have Docker documentation, https://docs.docker.com/storage/volumes/.

They're in /var/lib/docker/volumes with their relevant names as specified in the docker-compose.yml. https://github.com/DynamicDevices/ming/blob/4c77867e654a5bb35ee6d450e5d0c6d5ca3fa8b8/docker-compose.yml#L2-L6

ajlennon commented 4 years ago

At no point would I say this should be in the MING default, but someone might think it does. At this point you're supposed to make a new layer, or simply add it yourself for your own needs rather than include it in the base layer.

Agreed. It's finding the line.

Installing the node-red-influxdb-contrib makes sense, because we provide Influx.

Agreed.

Installing node-red-node-serialport and node-red-dashboard is sensible if it's functionality that a lot of people use and want out of the box.

Yes. Agreed.

But let's say Julian wanted to install node-red-node-random, principally he's supposed to take MING and add that to his own fork, which is his own embedded device/package which uses that extra thing because there are an infinite amount of nodes and possibilities for things that he and his project might want or need.

At some point it would get ridiculous and should not be included in the base, you should go off and copy MING and add those things for your own purposes/project.

Agreed. We need to consider the downside of installing extra nodes. In general I would say there is very little downside to adding functionality to node-red

That said, I think is right to include all of the things Julian has mentioned since they are small and useful additions that are highly geared towards the use case of everyone using MING. I will however add a build arg to not include them. That ARG would be barebones or something of that nature, so that when building people can use docker build --build-args=barebones and remove the un-necessary componentry.

Yes that would be excellent

I'll submit a PR later tonight or tomorrow with these changes for review.

Ta!

goatchurchprime commented 4 years ago

I can see the file systems. It's fairly obscure and su protected, here:

sudo ls /var/lib/docker/volumes/ming_jupyterlab-data/_data

To get into the jupyter image to add new stuff when you don't have root access:

docker exec -u 0 -it ming_jupyterlab_1 bash You can then set the su password with "passwd"

We might want both "jupyter notebook" and "jupyter lab" running.

The latter is the up to date version, but the former is the more hacky because it is allowed to run arbitrary unsanitized javascript, and therefore has better interactivity and features.

eg this works in it: https://ipyvolume.readthedocs.io/en/latest/

The interactive sliders and stuff don't work in lab. It can be re-enabled here: https://stackoverflow.com/questions/49542417/how-to-get-ipywidgets-working-in-jupyter-lab

I ran this before setting it up:

jupyter labextension install @jupyter-widgets/jupyterlab-manager

Then did a docker container restart ming_jupyter_1.

And now the simplest widget example works:

from ipywidgets import widgets
def f(x):
    print(x)
widgets.interact(f, x=10)

This is very good.

Now for the 3D capabilities as outlined here: https://pythreejs.readthedocs.io/en/stable/ or at https://github.com/jupyter-widgets/pythreejs

jupyter labextension install jupyter-threejs

I got the pythreejs to work, but it's way too low level and complicated to drive, in comparison to https://ipyvolume.readthedocs.io/en/latest/ There is no way scientists could ever use it.

Here's some docs: https://github.com/maartenbreddels/ipyvolume

conda install -c conda-forge nodejs # or some other way to have a recent node jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter labextension install ipyvolume jupyter labextension install jupyter-threejs

ajlennon commented 4 years ago

I'm fixing meta-mono at the moment but intending to sort out sudo access for you in the Jupyter container

ajlennon commented 4 years ago

@goatchurchprime What are you doing to run MING locally on your laptop? (command lines please?)

MatthewCroughan commented 4 years ago

@ajlennon Why is sudo necessary inside the container? Keep in mind that we're supporting both Balena and Docker here. The password for Jupyter is set by an environment variable in the entrypoint start.sh

You do not set it inside the container as that would not be ephemeral/stateless.

However, I think the password is set as a hash in the persistent /data directory, this needs further discussion, as it may be possible to do this in multiple ways by editing the start.sh to check for the existence of a changed password first before it goes ahead and sets a default one.

https://github.com/DynamicDevices/ming/blob/4c77867e654a5bb35ee6d450e5d0c6d5ca3fa8b8/jupyterlab/start.sh#L1-L14

MatthewCroughan commented 4 years ago

Also if we're going to make changes like this, it makes sense at this point to have a development branch, which now that I have commit access is where I'll be putting my changes.

ajlennon commented 4 years ago

@goatchurchprime wants to be able to add things into the container

ajlennon commented 4 years ago

I've added sudo support to the Jupyter container @goatchurchprime in #26

ajlennon commented 4 years ago

OK you've got the node-red influxdb and serial nodes now by default #27

ajlennon commented 4 years ago

And the dashboard in #28

ajlennon commented 4 years ago

@goatchurchprime Probably need to look at giving node-red access to the underlying serial devices as we discussed over lunch.

MatthewCroughan commented 4 years ago

@ajlennon You should just be able to make the container privileged and it'll have hardware access. Alternatively you can specify devices in the docker-compose file like so https://docs.docker.com/compose/compose-file/#devices

ajlennon commented 4 years ago

@ajlennon You should just be able to make the container privileged and it'll have hardware access. Alternatively you can specify devices in the docker-compose file like so https://docs.docker.com/compose/compose-file/#devices

We really need to try hard to avoid granting privileged access. The problem we have is with hot-pluggable devices.

MatthewCroughan commented 4 years ago

@ajlennon There is no way to do this easily otherwise. Is it really that hard to add it to the file specifically when needed? That's why I'm saying we cannot make something that is one-size fits all, there needs to be a bit of manual intervention here.

E.g make JupyterLabs privileged and accessible locally only on the pc it's on (127.0.0.1) when needed.

MatthewCroughan commented 4 years ago

The way Balena handles hot-plugged devices relates strongly to the base image, where they run udevadm and communicate between the host's udevd. This might be something we can add and run to each image, I have done this in OctoBalena. https://github.com/MatthewCroughan/octobalena/commit/f603b671478eeacc9010e8869159eb6d7083a944

See this for more details: https://www.balena.io/docs/reference/base-images/base-images/#how-the-images-work-at-runtime

ajlennon commented 4 years ago

E.g make JupyterLabs privileged and accessible locally only on the pc it's on (127.0.0.1) when needed.

We may have to but I'd like to make sure there's absolutely no other alternative if we did as it's a security concern.

MatthewCroughan commented 4 years ago

It is not a security concern if it's ran on the localhost only. How could it possibly be? If you're the only one who can access it? Windows Explorer can do more damage..?

MatthewCroughan commented 4 years ago

If you want dynamic hardware access you need udev, and that requires privilege. We can only do static hardware passthrough without privilege, as far as I have read in all the available documentation.

ajlennon commented 4 years ago

It's a security concern.

MatthewCroughan commented 4 years ago

@ajlennon Then running Jupyter Lab on its own without docker is a security concern? Security is already improved by running it in a container at all. We're not moving backwards by giving it privilege.

Security is not the focus of the jupyterlab container at least when hardware access is required. We can't move forward and do interesting things if there is a focus on security for it. Hardware access requires privilege, and by focusing too much on it we're only going to stall.

MatthewCroughan commented 4 years ago

We could provide multiple compose files. One that gives hardware access and one that does not. Giving hardware access by its nature is insecure. We can't change that nature, so we shouldn't try to.

ajlennon commented 4 years ago

We're not moving backwards by giving it privilege.

Yes, we would be.

Security is not the focus of the jupyterlab container at least when hardware access is required. We can't move forward and do interesting things if there is a focus on security for it. Hardware access requires privilege, and by focusing too much on it we're only going to stall.

There needs to be a focus on security. Hardware access doesn't require privilege.

MatthewCroughan commented 4 years ago

https://gitlab.com/windfactory/hullbuoy/commit/67bec7cdeef6c1bde68fa9efea1134726ae57f8d

Check this commit out for how I got around an issue like this by running a pigpiod container that does have privilege. Then I could access GPIO on the pi without having to give a node-red container any privilege, that is definitely insecure.

But how is it insecure to host the privileged pigpiod container?

Balena base images use udevd, that's how they get dynamic hardware allocation. Maybe we can work that out somehow on all platforms.

What you're asking for will work fine with Balena base images on BalenaOS, but I don't know how well it will work in all Linux environments or WSL, so I'll check it out and report back on it in a separate issue.

MatthewCroughan commented 4 years ago

And like I said, it's possible to statically map devices with no privilege out of the box with Docker, but dynamic discovery of devices in /dev/ requires the aforementioned feature that is in all Balena base images.

MatthewCroughan commented 4 years ago

Seeing as Alex has blasted through a few of the to-do's, I'll make a list here of what's left to do:

[x] Add sudo to the jupyterlab container #26
[x] Add nodered nodes for serial support and influxdb #27
[x] Add nodered dashboard support #28
[ ] Figure out dynamic device discovery so that the jupyterlab can speak to devices
[ ] Add micropython kernel to jupyterlab
[ ] Add Julia kernel to jupyterlab
[ ] Add jupyterlab-manager to jupyterlab
[ ] Add pythreejs to jupyterlab
[ ] Add influxdb and pyserial to jupyterlab
[ ] Add a recent nodejs to jupyterlab
[ ] Provide some example flows in nodered
[ ] Put all of these features in build args so they can be optionally removed at build time

ajlennon commented 4 years ago

We might want to use udev to kick off nc or socat to listen on a port when a USB device is inserted, then connect that port to the container

socat TCP-LISTEN:4161,fork,reuseaddr FILE:/dev/ttyUSB0,b57600,raw

e.g. https://superuser.com/questions/614494/redirect-serial-com-to-tcp-port

MatthewCroughan commented 4 years ago

Im currently working on the majority of the jupyterlab issues now, in future will make separate issues so we can use the kanban board to represent what we are working on so we don't work on the same thing, though doubt that's a risk now.

MatthewCroughan commented 4 years ago

None of the requested Jupyter extensions build on armv7 properly:

# Install useful JupyterLab extensions as discussed in
# https://github.com/DynamicDevices/ming/issues/25

RUN jupyter labextension install \
    @jupyter-widgets/jupyterlab-manager \
    jupyter-threejs \
    ipyvolume \
    && cat /tmp/jupyterlab-debug-*.log

The following error is returned:

[main] An error occured. [main] RuntimeError: JupyterLab failed to build [main] See the log file for details: /tmp/jupyterlab-debug-sc3jthqy.log [main] Removing intermediate container fbb8f9637bf8

Yes.. That's right. It generates a random filename in /tmp/ and doesn't give you any log information to stdout which is really annoying to debug. I can't find a CLI option to make it log to stdout and the secret hash is not given until the end. Works fine on x86.

MatthewCroughan commented 4 years ago

Interestingly trying to install that manually does something weird and kills a basic jupyterlab container on a pi before I can get a log output. I'm thinking it's running out of ram or something.

DynamicDevices / ming-original

Only partially working out of the box on my ubuntu laptop #25