Experimenting with "dynamically loaded services in Substrate"

[yoshikiohshima]

I am trying to follow description here:

https://github.com/ajbouh/substrate/pull/91/files

1. I pulled the main branch
2. I created a directory called images/miniDocker
3. I created index.js in it that looks like:

   const express = require('express')
   const app = express()
   app.use(express.urlencoded({ extended: true }))
   app.use(express.json())  
   app.get('/', (req, res) => res.send('Hello, Substrate!'))
   app.listen(5000, () => console.log(`⚡️[bootup]: Server is running at port: 5000`))

4. I created package.json that looks like:

   {
   "name": "minidocker",
   "version": "1.0.0",
   "description": "",
   "main": "index.js",
   "scripts": {
   "test": "echo \"Error: no test specified\" && exit 1"
   },
   "keywords": [],
   "author": "",
   "license": "ISC",
   "dependencies": {
   "express": "^4.18.2"
   }
   }

5. I created Dockerfile:

   
   # Fetching the minified node image on apline linux
   FROM node:slim

Declaring env

ENV NODE_ENV development

Setting up the work directory

WORKDIR /express-docker

Copying all the files in our project

COPY images/foo/. /work/.

Installing dependencies

RUN npm install

Starting our application

CMD [ "node", "index.js" ]

Exposing server port

EXPOSE 5000

(all from this page, basically https://www.geeksforgeeks.org/how-to-dockerize-an-expressjs-app/)

6. I created `defs/miniDocker.cue` that looks like:

package defs enable: "miniDocker": true imagespecs: "miniDocker": {} lenses: "miniDocker": { spawn: { environment: { SOME_VARIABLE: "some_value" } } }

7. I ran `./remote ./dev.sh systemd-reload`, which did some work but I don't see "miniDocker" mentioned in the resulting log.
8. So I guessed that maybe `./remote ./dev.sh reload` was needed and tried it but I cannot tell where is actually `/work` and where those files are supposed to show up.
9.  but I now know that there are indeed examples in /images and /defs so I'll look at some of them to figure things out.

[ajbouh]

Some modifications for your thing to work:

1. make sure you git add ... your files before the ./remote ... command
2. listen to process.env.PORT in your nodejs code (instead of 5000)
3. your COPY line should be COPY images/miniDocker/. /work/.
4. can delete SOME_VARIABLE from the defs file
5. can delete EXPOSE line in Dockerfile
6. after ... systemd-reload (which is just an alias to reload), you can visit https://substrate.home.arpa/gw/miniDocker/

[yoshikiohshima]

git add ... my files did make it move forward. (and fixing the COPY line). it tries to invoke npm and fails. Is it something you'd install globally, or I'd run the node installer in this container?

[ajbouh]

Can you run a git diff --cached | pbcopy (will overwrite your clipboard) and paste it here in a comment in a ``` codeblock? I can try to reproduce and debug

[yoshikiohshima]

diff --git a/defs/miniDocker.cue b/defs/miniDocker.cue
new file mode 100644
index 0000000..ea8404d
--- /dev/null
+++ b/defs/miniDocker.cue
@@ -0,0 +1,10 @@
+package defs
+enable: "miniDocker": true
+imagespecs: "miniDocker": {}
+lenses: "miniDocker": {
+    spawn: {
+        environment: {
+           SOME_VARIABLE: "some_value"
+        }
+    }
+}
diff --git a/images/miniDocker/Dockerfile b/images/miniDocker/Dockerfile
new file mode 100644
index 0000000..14a146f
--- /dev/null
+++ b/images/miniDocker/Dockerfile
@@ -0,0 +1,20 @@
+# Fetching the minified node image on apline linux
+FROM node:slim
+
+# Declaring env
+ENV NODE_ENV development
+
+# Setting up the work directory
+WORKDIR /express-docker
+
+# Copying all the files in our project
+COPY images/miniDocker/. /work/.
+
+# Installing dependencies
+RUN npm install
+
+# Starting our application
+CMD [ "node", "index.js" ]
+
+# Exposing server port
+EXPOSE 5000
\ No newline at end of file
diff --git a/images/miniDocker/index.js b/images/miniDocker/index.js
new file mode 100644
index 0000000..192e9cc
--- /dev/null
+++ b/images/miniDocker/index.js
@@ -0,0 +1,10 @@
+const express = require('express')
+const app = express()
+  
+app.use(express.urlencoded({ extended: true }))
+app.use(express.json())
+  
+app.get('/', (req, res) => res.send('Hello, Substrate!'))
+  
+app.listen(5000, () => console.log(`⚡️[bootup]: Server is running at port: 5000`))
+
diff --git a/images/miniDocker/package.json b/images/miniDocker/package.json
new file mode 100644
index 0000000..a5f6bc3
--- /dev/null
+++ b/images/miniDocker/package.json
@@ -0,0 +1,15 @@
+{
+  "name": "minidocker",
+  "version": "1.0.0",
+  "description": "",
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "express": "^4.18.2"
+  }
+}

[ajbouh]

Here are the changes I needed (atop your patch) to make it work:

diff --git a/images/miniDocker/Dockerfile b/images/miniDocker/Dockerfile
index 14a146f6..ac0e5918 100644
--- a/images/miniDocker/Dockerfile
+++ b/images/miniDocker/Dockerfile
@@ -8,7 +8,7 @@ ENV NODE_ENV development
 WORKDIR /express-docker

 # Copying all the files in our project
-COPY images/miniDocker/. /work/.
+COPY images/miniDocker/. .

 # Installing dependencies
 RUN npm install
diff --git a/images/miniDocker/index.js b/images/miniDocker/index.js
index 192e9cc9..c5b9902c 100644
--- a/images/miniDocker/index.js
+++ b/images/miniDocker/index.js
@@ -6,5 +6,5 @@ app.use(express.json())

 app.get('/', (req, res) => res.send('Hello, Substrate!'))

-app.listen(5000, () => console.log(`⚡️[bootup]: Server is running at port: 5000`))
+app.listen(process.env.PORT, () => console.log(`⚡️[bootup]: Server is running at port: ${process.env.PORT}`))

[yoshikiohshima]

running ./remote .dev.sh reload.. maybe I should've run systemd-reload? In any case, is there a way to just send one "image" to the machine?

[yoshikiohshima]

it works now!

[ajbouh]

running ./remote .dev.sh reload.. maybe I should've run systemd-reload? In any case, is there a way to just send one "image" to the machine?

systemd-reload and reload are aliases (see https://github.com/ajbouh/substrate/blob/main/dev.sh#L482-L486)

It's possible to put the short name of the service after the reload and it won't do quite as much work as normal. It still sends all the files to the remote machine (when wrapped with ./remote ...), but it should build fewer things on the remote machine.

So the thing to try would be:

./remote ./dev.sh systemd-reload miniDocker

[yoshikiohshima]

ok. And then I now want to learn about "live editing". I sort of on purpose chose a language that I don't have to compile or such. I'll check it out.

[ajbouh]

I've merged support for live editing defs (https://github.com/ajbouh/substrate/blob/main/defs/substrate.cue) and the substrate ui (https://github.com/ajbouh/substrate/blob/main/defs/ui.cue). I've also sketched out live editing for the bridge2 UI in #102.

The basic idea is to mount the source code to the proper path within a docker container.

You can run code in the container that watches the filesystem for changes, or you can always re-read the latest value (for example when the user refreshes a page in their browser and the server re-reads it from disk).

[ajbouh]

@yoshikiohshima is this issue good to close? or is there more remaining on this exploration?

[yoshikiohshima]

I pushed this: https://github.com/ajbouh/substrate/tree/miniService/experiments/miniService This doc is good to be our "primer" directory somewhere. and this issue can be closed, before or after that doc gets a place, yes.