shankari
commented
1 year ago So it seems like we should be able to create a new Dockerfile that builds on build-kit-alpine, compiles to create a new pre-built image, and then launches a basic SIL
Open shankari opened 1 year ago
shankari
commented
1 year ago So it seems like we should be able to create a new Dockerfile that builds on build-kit-alpine, compiles to create a new pre-built image, and then launches a basic SIL
shankari
commented
1 year ago First check:
shankari
commented
1 year ago (I guess we can ask them what happened, for now, we can potentially use that PR as a template of how to build) Is that more recent or is the buildkit image build more recent? (or did this morph into the buildkit image?)
shankari
commented
1 year ago Bingo! They created the new file on May 13th and updated on May 14th. Last update to the PR was May 14th So basically they decided to move it over to a separate repo. The PR is obsolete. The new github action is the correct one and I can move forward with it.
shankari
commented
1 year ago Ok, so now that this has percolated into my brain, I think I know what I want to do. I want to make a repo called everest-demo (currently running in my personal org) It has the Dockerfile that I write and the docker-compose and the PWD
Depending on our final decision, we either submit this to EVerest as is, or do the GitHub commands to merge it in without commit history (easier) or with commit history (I've done it before).
shankari
commented
1 year ago (this is that repo). All updates before this were copied from a file I was maintaining. Moving to this issue now.
shankari
commented
1 year ago The build-kit action is only manually triggered (on workflow_dispatch). Was successfully built 5 days ago
In everest-core, however, things are more murky.
There is a single build_and_test workflow defined in the repo, whose stability seems to depend on the particular PR
It doesn't look like they have any scheduled tasks runs defined to assess long-term stability
However, they do have a couple of other actions that are apparently run on pull requests as well https://github.com/EVerest/everest-core/actions/workflows/build_ci_testing_base_image.yml https://github.com/EVerest/everest-core/actions/workflows/run_ci_testing.yml
However, neither of them have run for 3 months so it looks like they used to have other workflows but removed them? Although I don't see that in the history for the file, unless they deleted the directory and recreated.
shankari
commented
1 year ago Final question before starting this. How/when do we rebuild the docker image for the demo and where do we store it? Options:
That makes it fairly clear - start with manual, particularly for a demo!! Can automate based on workflow run later. But no nasty demo surprises due to automated update!
Fortunately, we can push images to the GitHub docker registry with a manual token https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
Fortunately, docker-compose also supports pushing the generated images https://docs.docker.com/compose/compose-file/build/#publishing-built-images
Usage: docker compose [OPTIONS] COMMAND
Commands:
push Push service imagesI do think that we need to be careful about the name in the docker-compose - it might need to start with ghcr.io/NAMESPACE and we need to configure docker-compose accordingly.
docker push ghcr.io/NAMESPACE/IMAGE_NAME:latest
Enough procrastination/investigation - time to start with the code and open a PR Detailed discussion will now move to the PR
shankari
commented
1 year ago Another issue is that the current workflow runs clang-format through a GitHub action (run-clang-format) which in turn, runs the action as a docker command
runs:
using: 'docker'
image: 'docker://ghcr.io/everest/everest-clang-format:latest'
args:
- /github/workspace/${{ inputs.source-dir }}
- --extensions
- ${{ inputs.extensions }}
- -e
- /github/workspace/${{ inputs.source-dir }}/${{ inputs.exclude }}
- --color
- ${{ inputs.color }}
- -rI don't think I can run a docker command while building a docker image.
Options are: (1) Try and find where that image is being built from and copy its Dockerfile (2) Look at the manual instructions and copy them over
I would really like to use the same process that the CI does, to increase the chances that this will work, so I don't want to use (2) unless I have to.
I wonder if we can use docker-compose dependencies to work around this - but I think that they only work for launch, not for build.
shankari
commented
1 year ago Hm, maybe we can use docker-compose order to get around this
If I create a docker-compose based on .ci/e2e/docker-compose.yaml
services:
mqtt-server:
build: mosquitto
e2e-test-server:
build: demo
depends_on:
- mqtt-serverBut if I switch the order in docker-compose, they are still built in the same order
shankari
commented
1 year ago I also found the Dockerfile for the clang container https://github.com/EVerest/everest-utils/blob/main/docker/everest-clang-format/Dockerfile
And now I understand the reason for the different containers - the buildkit container is based on alpine while the clang one is based on debian.
But there is a debian dockerfile as well https://github.com/EVerest/everest-ci/tree/main/docker/images/build-kit
But they don't build and release debian images
So is this just clang not working on debian? Let's create a container and find out
shankari
commented
1 year ago Trying to follow the clang Dockerfile instructions, curl https://github.com/EVerest/run-clang-format/releases/download/clang-format-16/clang-format doesn't seem to return anything.
But downloading it from https://github.com/EVerest/run-clang-format/releases/download/clang-format-16/ does seem to work
I wonder if this is restricted to logged in users, which is why we need the ssh keys? But this is a docker command, so I don't see why it should matter.
Verified that it is downloadable using a private window Or if I put it into a docker container
Moving on to trying to run it.
/ext # /usr/bin/run-clang-format
usage: run-clang-format [-h] [--clang-format-executable EXECUTABLE] [--extensions EXTENSIONS] [-r] [-d] [-i]
[-q] [-j N] [--color {auto,always,never}] [-e PATTERN] [--style STYLE]
file [file ...]I have cloned the core repo and moved it to the ext directory to be consistent with the buildkit
docker run \
--volume "$(pwd):/ext" \
--name compile-container \
build-kit run-script compileBut the action is called with the arguments
source-dir: source
extensions: hpp,cpp
exclude: cacheAnd I don't find a source dir in the core.
Doh! it is reconfigured in the setup actions
- name: Checkout everest-core
uses: actions/checkout@v3
with:
path: sourceSo the "source" directory is the one where the .git repo is, so ext for us
shankari
commented
1 year ago as expected, the debian clang does not work on alpine.
I can see it cat /usr/bin/clang-format but cannot run it
# /usr/bin/clang-format
/bin/sh: /usr/bin/clang-format: not foundI really don't want to get sucked in to an alpine/debian crossport at this point. So what can we do?
First order of business once we start real work is to fix their build process...
But for now, it seems like the best option is #3 - use their debian container
The debian Dockerfile seems to be maintained in sync with the alpine Dockerfile, but it is not built automatically.
Since we are doing all this manually, it seems like we should be able to build from it (potentially as part of docker-compose build using the github URL so we don't have too many copies sitting around), and then add clang stuff to in our dockerfile and build everything.
shankari
commented
1 year ago Added an image that builds from the Dockerfile on GitHub and ran into
#1 [internal] load git source https://github.com/EVerest/everest-ci.git#master:docker/images/build-kit
#1 ERROR: subdir not supported yet
------
> [internal] load git source https://github.com/EVerest/everest-ci.git#master:docker/images/build-kit:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to load cache key: subdir not supported yetThis can apparently be fixed by using buildx. They use buildx for their Dockerfile, but I think that is primarily to support debian and alpine at the same time. Let's copy over their Dockerfile and see if we can at least get it to work
shankari
commented
1 year ago That built successfully after copying over the entryfile and the maven settings. Trying to run the scripts now...
shankari
commented
1 year ago Can't start because of
$ docker run -it everest-demo-e2e-test-server:latest /bin/sh
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/entrypoint.sh": permission denied: unknown.$ docker run -it everest-demo-build-kit-debian:latest /bin/sh
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/entrypoint.sh": permission denied: unknown.
ERRO[0000] error waiting for container: context canceledOverriding with --entrypoint worked
shankari
commented
1 year ago clang now seems to run successfully - at least it doesn't give any errors except highlighting this entry that needs to be reformatted
root@2bad2ccf3881:/ext# /usr/bin/run-clang-format . --extensions "hpp,cpp" -e ./cache -r
--- ./modules/OCPP201/OCPP201.cpp (original)
+++ ./modules/OCPP201/OCPP201.cpp (reformatted)
@@ -438,8 +438,8 @@
evse_id, 1, session_id, timestamp,
ocpp::v201::TriggerReasonEnum::RemoteStart, // FIXME(piet): Use proper reason here
meter_value, id_token, std::nullopt, reservation_id, remote_start_id,
- ocpp::v201::ChargingStateEnum::Charging); // FIXME(piet): add proper groupIdToken +
- // ChargingStateEnum
+ ocpp::v201::ChargingStateEnum::Charging); // FIXME(piet): add proper groupIdToken +
+ // ChargingStateEnum
break;
}
case types::evse_manager::SessionEventEnum::TransactionFinished: {After fiddling around a bit with the paths (the code needs to be in /ext/sources), I can get it to start compiliing by running the compile script manually (sources/.ci/compile_script). Will try to copy over the entrypoint correctly, but unclear if we need to use it.
Definitely cannot run the build every time - it is taking many many minutes and is not even done yet.
shankari
commented
1 year ago Given this https://github.com/shankari/everest-demo/issues/1#issuecomment-1685418895, which appears to only use the format functionality in clang-format, we probably don't need to run clang for the demo, which will simplify this a lot 😄
But let's first get it working with all of their modules and then simplify
shankari
commented
1 year ago Built successfully!
[422/576] Generating ld-ev for module Example
Creating file Example/ld-ev.hpp
Creating file Example/ld-ev.cpp
[576/576] Linking CXX executable modules/examples/Example/ExampleLet's first run the tests successfully before trying to run the SIL
Successfully built everestpy
Installing collected packages: everestpy
Successfully installed everestpy-0.0.4e2e tests are failing because they are launched in yet another docker container, and while trying to run
cd tests
pytest --everest-prefix ../dist core_tests/startup_tests.pythere is no ../dist
But there is a /workspace/dist, which is the only directory there. Seems to work better in the action, hopefully will be fixed once we actually get the endpoint set up correctly
[edm]: Scanning "/ext/source" for dependencies.
[edm]: Parsing dependencies file: /ext/source/dependencies.yaml
[edm]: Using parent directory as workspace path: /ext
[edm]: Using workspace directory "/ext".Running pytest --everest-prefix /workspace/dist core_tests/startup_tests.py instead passes everything other than the MQTT tests, which will not work unless we run it in the docker-compose environment. Let's try to at least find the SIL scripts although we can't run them without the MQTT
nb test_and_install deletes the build. Why?!`
shankari
commented
1 year ago In parallel, let's get started on creating the script to launch the SIL and export the port. The exported port should be openable from the PWD UI https://stackoverflow.com/questions/61104342/play-with-docker-access-ip-address-on-webbrowser
shankari
commented
1 year ago Ok so the scripts are in the build
everest-core/build/run-scripts/run-sil.sh seems to be fairly simple, and fails because of lack of MQTT.everest-core/build/run-scripts/nodered-sil.sh seems to run docker?Yup
cd /workspace/dist/share/everest/docker
docker build -t everest-nodered .
docker run --rm --network host --name everest_nodered --mount type=bind,source=/ext/source/config/nodered/config-sil-flow.json,target=/data/flows.json everest-noderedIt builds a new image and runs it. So in a way it is simple - the currrent service can just run-sil.sh
and we can have another service for nodered. But it is a bit of a problem that the nodered image is built from the dist directory.
run-sil.shAnd the only file in
# ls -al /workspace/dist/share/everest/docker
total 12
drwxr-xr-x 2 root root 4096 Aug 20 23:36 .
drwxr-xr-x 10 root root 4096 Aug 20 23:36 ..
-rw-r--r-- 1 root root 188 Aug 20 22:16 Dockerfile
# cat Dockerfile
FROM nodered/node-red:2.2.3
RUN npm install node-red-dashboard
RUN npm install node-red-contrib-ui-actions
RUN npm install node-red-node-ui-table
RUN npm install node-red-contrib-ui-levelWhich is also a file here https://github.com/EVerest/everest-core/blob/bf2de55b40eb44f84072a7e2daf07f891945d185/cmake/assets/docker/Dockerfile#L3
shankari
commented
1 year ago There seems to be a little bit of confusion around which port to use. Per https://github.com/EVerest/everest.github.io/blob/c8ce5904f620c39fb1880430dac033720fc087b2/docs/contacts-snapshot/_sources/tutorials/run_sil/index.rst.txt#L32 the nodered dashboard and the UI are both at port 1880.
But per https://github.com/EVerest/everest.github.io/blob/c8ce5904f620c39fb1880430dac033720fc087b2/docs/latest/_sources/general/03_quick_start_guide.rst.txt#L6, updated more recently overall, the admin dashboard (which appears to be the nodered dashboard) is at 8849
shankari
commented
1 year ago The nodered dashboard is essentially independent and doesn't really need the everest build as long as it has all the configs. Let's try to run it independently and see which ports it exposes.
Launch succeeds, but it is not able to run MQTT stuff.
> node-red-docker@2.2.3 start /usr/src/node-red
> node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
21 Aug 00:42:52 - [info]
Welcome to Node-RED
===================
21 Aug 00:42:52 - [info] Node-RED version: v2.2.3
21 Aug 00:42:52 - [info] Node.js version: v14.18.2
21 Aug 00:42:52 - [info] Linux 5.15.49-linuxkit x64 LE
21 Aug 00:42:52 - [info] Loading palette nodes
21 Aug 00:42:52 - [info] Dashboard version 3.5.0 started at /ui
21 Aug 00:42:52 - [info] Settings file : /data/settings.js
21 Aug 00:42:52 - [info] Context store : 'default' [module=memory]
21 Aug 00:42:52 - [info] User directory : /data
21 Aug 00:42:52 - [warn] Projects disabled : editorTheme.projects.enabled=false
21 Aug 00:42:52 - [info] Flows file : /data/flows.json
21 Aug 00:42:52 - [info] Server now running at http://127.0.0.1:1880/
21 Aug 00:42:52 - [warn]
21 Aug 00:42:53 - [info] [mqtt-broker:fc8686af.48d178] Connection failed to broker: mqtt://localhost:1883
21 Aug 00:43:08 - [info] [mqtt-broker:fc8686af.48d178] Connection failed to broker: mqtt://localhost:1883
21 Aug 00:43:23 - [info] [mqtt-broker:fc8686af.48d178] Connection failed to broker: mqtt://localhost:1883
21 Aug 00:43:38 - [info] [mqtt-broker:fc8686af.48d178] Connection failed to broker: mqtt://localhost:1883Cannot also access the port, but that is likely due to the port not being mapped. Trying to re-run with the port mapping... Still not working, likely due to the MQTT. Presumably we don't need the port mapping due to the --host?
shankari
commented
1 year ago This works:
docker run --rm -it -p 1880:1880 -v node_red_data:/data --name mynodered nodered/node-red:2.2.3
this doesn't work
docker run --rm -it --network host -v node_red_data:/data --name mynodered nodered/node-red:2.2.3
It seems to start up fine, but the page is not accessible.
21 Aug 00:58:40 - [info] Node-RED version: v2.2.3
21 Aug 00:58:40 - [info] Node.js version: v14.18.2
21 Aug 00:58:40 - [info] Linux 5.15.49-linuxkit x64 LE
21 Aug 00:58:41 - [info] Loading palette nodes
21 Aug 00:58:41 - [info] Settings file : /data/settings.js
21 Aug 00:58:41 - [info] Context store : 'default' [module=memory]
21 Aug 00:58:41 - [info] User directory : /data
21 Aug 00:58:41 - [warn] Projects disabled : editorTheme.projects.enabled=false
21 Aug 00:58:41 - [info] Flows file : /data/flows.json
21 Aug 00:58:41 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
21 Aug 00:58:41 - [warn] Encrypted credentials not found
21 Aug 00:58:41 - [info] Server now running at http://127.0.0.1:1880/
21 Aug 00:58:41 - [info] Starting flows
21 Aug 00:58:41 - [info] Started flowsChecked the container
"ExposedPorts": {
"1880/tcp": {}
},
"Ports": {},Ok... so I don't know how the docker command was ever supposed to work. Now running the working command with the mounted flow...
docker run --rm -it -p 1880:1880 -v node_red_data:/data --mount type=bind,source=/.../everest-core/config/nodered/config-sil-flow.json,target=/data/flows.json --name mynodered nodered/node-red:2.2.3works but has errors with the load
shankari
commented
1 year ago Building from the everest-nodered image works, and removes the errors, but a lot of the external commands are having trouble connecting, likely due to the MQTT issue
$ docker run --rm -it -p 1880:1880 -v node_red_data:/data --mount type=bind,source=/.../everest-core/config/nodered/config-sil-flow.json,target=/data/flows.json --name mynodered everest-noderedAhah! https://cookbook.nodered.org/mqtt/connect-to-broker
No environment variable
it is hardcoded in the flow file - e.g.
"type": "mqtt-broker",
"name": "",
"broker": "localhost",
"port": "1883",
"clientid": "",
"usetls": false,Can confirm that this is the right way to do it "Linking Containers". Not sure why they don't just use docker-compose
On production, we will copy the configs into the container instead of mounting them, so can modify the hostname. On dev, there's no point in doing that then we will change the mounted values, and we might as well change them ahead of time if needed in that case.
So let's copy over the flow files to our demo directory and manually rename them.
There are other components:
shankari
commented
1 year ago Note also that it looks like admin-panel is used in the EVerest framework https://github.com/EVerest/everest-framework/blob/df044f9a4d723a3e7f9ff9fe966c645b9b33c2c3/src/controller/CMakeLists.txt#L47
But neither of them seem to be in the source
/ext/source# grep -r \*everestjs\* .
/ext/source#
/ext/source# grep -r \*EvModuleList\* .
/ext/source#So we might want to deal an expanded dockerfile that follows all the instructions below: https://everest.github.io/nightly/general/03_quick_start_guide.html
shankari
commented
1 year ago We cannot use the clang in the Dockerfile - if there is any super small error, it fails
#13 [ 8/13] RUN /usr/bin/run-clang-format . --extensions "hpp,cpp" -e ./cache -r
#13 0.236 --- ./run-clang-format/src/third_party/qux.cpp (original)
#13 0.236 +++ ./run-clang-format/src/third_party/qux.cpp (reformatted)
#13 0.236 @@ -1,8 +1,8 @@
#13 0.236 // This code is ignored by the .clang-format-ignore file.
#13 0.236
#13 0.236 -int qux (bool cond) {
#13 0.236 - if(cond){
#13 0.236 - return - 1;
#13 0.236 +int qux(bool cond) {
#13 0.236 + if (cond) {
#13 0.236 + return -1;
#13 0.236 }
#13 0.236
#13 0.236 return 0;
#13 0.236 --- ./run-clang-format/src/foo.cpp (original)
#13 0.236 +++ ./run-clang-format/src/foo.cpp (reformatted)
#13 0.236 @@ -2,6 +2,6 @@
#13 0.236
#13 0.236 std::string foo(const std::string &a, const std::string &b) {
#13 0.236 std::string sum;
#13 0.236 - sum = a + b;
#13 0.236 + sum = a + b;
#13 0.236 return sum;
#13 0.236 }
#13 ERROR: executor failed running [/bin/sh -c /usr/bin/run-clang-format . --extensions "hpp,cpp" -e ./cache -r]: exit code: 1
------
> [ 8/13] RUN /usr/bin/run-clang-format . --extensions "hpp,cpp" -e ./cache -r:
#13 0.236 --- ./run-clang-format/src/foo.cpp (original)
#13 0.236 +++ ./run-clang-format/src/foo.cpp (reformatted)
#13 0.236 @@ -2,6 +2,6 @@
#13 0.236
#13 0.236 std::string foo(const std::string &a, const std::string &b) {
#13 0.236 std::string sum;
#13 0.236 - sum = a + b;
#13 0.236 + sum = a + b;
#13 0.236 return sum;
#13 0.236 }
------
failed to solve: executor failed running [/bin/sh -c /usr/bin/run-clang-format . --extensions "hpp,cpp" -e ./cache -r]: exit code: 1Compile is also failing, running it manually first and then putting it into the Dockerfile
#16 [11/12] RUN run-script compile
#16 0.249 /bin/sh: 1: run-script: not found
#16 ERROR: executor failed running [/bin/sh -c run-script compile]: exit code: 127
------
> [11/12] RUN run-script compile:
#16 0.249 /bin/sh: 1: run-script: not found
------
failed to solve: executor failed running [/bin/sh -c run-script compile]: exit code: 127Entrypoint is still broken
$ docker run -it everest-demo-manager /bin/bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/entrypoint.sh": permission denied: unknown.
ERRO[0000] error waiting for container: context canceledIt was just the permissions - bash /entrypoint.sh run-script compile works
Given that we can't run clang anyway, I am very tempted to just go back to the alpine build, which will make our life much easier overall
shankari
commented
1 year ago After switching to alpine, the image was built successfully
#12 exporting to image
#12 exporting layers
#12 exporting layers 14.0s done
#12 writing image sha256:9bf3c9bb58a2774b678dac40ef2e91f42a114f03490bdff5e1d1d223b22bc105 done
#12 naming to docker.io/library/everest-demo-manager 0.0s done
#12 DONE 14.0sNow I will exec into the manager and try to run
shankari
commented
1 year ago Entrypoint in the dockerfile is not working?! but is in the docker-compose The nodered container is now connected to MQTT and shows as "connected"
everest-demo-mqtt-server-1 | 1692590573: New connection from 192.168.2.165:42672 on port 1883.
everest-demo-mqtt-server-1 | 1692590573: New client connected from 192.168.2.165:42672 as nodered_ad70af80b46de7b1 (p2, c1, k60).
everest-demo-node-red-1 | 21 Aug 04:02:53 - [info] [mqtt-broker:fc8686af.48d178] Connected to broker: mqtt://mqtt-server:1883Start up the manager as well, and that seems to work...
2023-08-21 04:05:58.882189 [INFO] manager :: EVerest manager starting using /ext/source/config/config-sil.yaml
2023-08-21 04:05:58.882255 [INFO] manager :: EVerest using MQTT broker mqtt-server:1883
2023-08-21 04:05:58.882280 [INFO] manager :: EVerest telemetry enabled
2023-08-21 04:05:59.355271 [INFO] everest_ctrl :: everest controller process started ...
2023-08-21 04:05:59.355498 [INFO] everest_ctrl :: Launching controller service on port 8849
2023-08-21 04:06:00.996474 [INFO] api:API :: Module api initialized.
2023-08-21 04:06:01.147806 [INFO] connector_1:Evs :: Module connector_1 initialized.
2023-08-21 04:06:01.151094 [INFO] energy_manager: :: Module energy_manager initialized.
2023-08-21 04:06:01.281156 [INFO] token_validator :: Module token_validator initialized.
2023-08-21 04:06:01.285056 [INFO] grid_connection :: Module grid_connection_point initialized.
2023-08-21 04:06:01.322436 [INFO] auth:Auth :: Module auth initialized.
2023-08-21 04:06:01.389648 [INFO] iso15118_charge :: TCP server on eth0 is listening on port [fe80::42:c0ff:fea8:2a4%275]:61341
2023-08-21 04:06:01.389757 [INFO] iso15118_charge :: TLS server on eth0 is listening on port [fe80::42:c0ff:fea8:2a4%275]:64109
2023-08-21 04:06:01.389800 [INFO] iso15118_charge :: SDP socket setup succeeded
2023-08-21 04:06:01.389941 [INFO] iso15118_charge :: Module iso15118_charger initialized.
2023-08-21 04:06:01.435038 [INFO] car_simulator:J :: Module car_simulator initialized.
2023-08-21 04:06:01.530832 [INFO] persistent_stor :: Module persistent_store initialized.
2023-08-21 04:06:01.531557 [INFO] setup:Setup :: Module setup initialized.
2023-08-21 04:06:01.582444 [INFO] connector_1_pow :: Module connector_1_powerpath initialized.
2023-08-21 04:06:01.596913 [INFO] slac:JsSlacSimu :: Module slac initialized.
2023-08-21 04:06:01.729395 [INFO] token_provider: :: Module token_provider initialized.
2023-08-21 04:06:02.206653 [INFO] iso15118_car :: Module iso15118_car initialized.
2023-08-21 04:06:02.227318 [INFO] manager :: >>> All modules are initialized. EVerest up and running <<<
2023-08-21 04:06:03.673677 [INFO] connector_1:Evs :: Max AC hardware capabilities: 32A/3ph
2023-08-21 04:06:03.899127 [INFO] connector_1:Evs :: AC HLC mode enabled.
2023-08-21 04:06:03.899773 [INFO] connector_1:Evs :: 🌀🌀🌀 Ready to start charging 🌀🌀🌀| UI works | charging works | Error is detected |
|---|---|---|
shankari
commented
1 year ago ok so the goal for tonight is to clean this up and put it into PWD.
Slight change of plan:
shankari
commented
1 year ago I think that these are their official docker instructions: https://github.com/EVerest/everest-utils/tree/main/docker
It includes a debian based "everest-playground" https://github.com/EVerest/everest-utils/blob/main/docker/everest-docker-image/Dockerfile
And it doesn't do clang 😄 but it is not tested by the CI so I don't trust it as much They also want to seem to create a dfiferent image for each configuration, which is cool but should definitely be automated.
But more importantly, it also includes StEVe instructions so I can get OCPP to work.
shankari
commented
1 year ago Using StEVe is a giant pain as well. There is no pre-built image - you have to check out the code and build from scratch.
And they are using dockerize instead of docker-compose to wait for the DB instance to start up
And it's not clear how to pass in the URL to the mariadb, or is it assumed to be localhost?
shankari
commented
1 year ago High level conclusion:
For example:
In other words, like a typical underfunded open-source project. I happen to be very familiar with at least one of those.
I think that they have a strong technical foundation to build on and we can quickly make the engineering and tooling improvements needed, by staffing up with strong software engineers, even if they don't have specific domain expertise. That can free them (and other industry stakeholders) up to do more of the domain specific work.
shankari
commented
1 year ago Note that packages are published as private by default. It looks also that we can't change this for user repos, but can do so at an org level. Should make sure that this is turned on once we contribute this to the org repo
shankari
commented
1 year ago Quick notes about steve.
shankari
commented
1 year ago Looks like steve doesn't build?
Downloaded from central: https://repo1.maven.org/maven2/org/apache/apache/23/apache-23.pom (18 kB at 91 kB/s)
everest-demo-steve-1 | [INFO]
everest-demo-steve-1 | [INFO] -------------------------< de.rwth.idsg:steve >-------------------------
everest-demo-steve-1 | [INFO] Building steve 3.4.5
everest-demo-steve-1 | [INFO] --------------------------------[ jar ]---------------------------------
everest-demo-steve-1 | Downloading from central: https://repo.maven.apache.org/maven2/com/mycila/license-maven-plugin/4.1/license-maven-plugin-4.1.pom
everest-demo-steve-1 | [INFO] ------------------------------------------------------------------------
everest-demo-steve-1 | [INFO] BUILD FAILURE
everest-demo-steve-1 | [INFO] ------------------------------------------------------------------------
everest-demo-steve-1 | [INFO] Total time: 3.698 s
everest-demo-steve-1 | [INFO] Finished at: 2023-08-23T02:32:21Z
everest-demo-steve-1 | [INFO] ------------------------------------------------------------------------
everest-demo-steve-1 | [ERROR] Plugin com.mycila:license-maven-plugin:4.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.mycila:license-maven-plugin:jar:4.1: Could not transfer artifact com.mycila:license-maven-plugin:pom:4.1 from/to central (https://repo.maven.apache.org/maven2): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]ok so there's something kind of weird here. the steve dockerfile (https://github.com/EVerest/everest-utils/tree/main/docker/steve) builds off maven:3.6.1-jdk-11. When I tried to run the base image directly, I didn't specify the shell, so I got the errror below. Doh!
Status: Downloaded newer image for maven:3.6.1-jdk-11
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.084 s
[INFO] Finished at: 2023-08-23T02:40:05Z
[INFO] ------------------------------------------------------------------------
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]On using the shell, and mvn package I got the same error as above. Looking more carefully, it is the stupid ssl error...
After resolving that, it builds, but then errors out while trying to connect to mariadb. So pretty close.
BUT the build process doesn't actually build. It only downloads the data.
$ docker-compose -f docker-compose.run-sil-ocpp.build.yml build --no-cache --pull --progress=plainThe build happens when you launch, because I guess the mvn image installs the pom.xml in the workdir.
shankari
commented
1 year ago On launch, the build succeeded, we connected to the database, and then appeared to start a bunch of migrations
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.6.8 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.6.9 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.7.0 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.7.1 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.7.2 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.7.3 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.7.6 - update"
everest-demo-steve-1 | [INFO] Migrating schema `ocpp-db` to version "0.7.7 - update"
everest-demo-steve-1 | [ERROR] Migration of schema `ocpp-db` to version "0.7.7 - update" failed! Please restore backups and roll back database and code!everest-demo-steve-1 | [INFO] ------------------------------------------------------------------------
everest-demo-steve-1 | [INFO] BUILD FAILURE
everest-demo-steve-1 | [INFO] ------------------------------------------------------------------------
everest-demo-steve-1 | [INFO] Total time: 35.684 s
everest-demo-steve-1 | [INFO] Finished at: 2023-08-23T03:39:14Z
everest-demo-steve-1 | [INFO] ------------------------------------------------------------------------
everest-demo-steve-1 | [ERROR] Failed to execute goal org.flywaydb:flyway-maven-plugin:7.9.1:migrate (default) on project steve: org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException:
everest-demo-steve-1 | [ERROR] Migration V0_7_7__update.sql failed
everest-demo-steve-1 | [ERROR] -----------------------------------
everest-demo-steve-1 | [ERROR] SQL State : HY000
everest-demo-steve-1 | [ERROR] Error Code : 1833
everest-demo-steve-1 | [ERROR] Message : Cannot change column 'chargeBoxId': used in a foreign key constraint 'ocpp@002ddb/FK_chargeBoxId_c' of table 'ocpp@002ddb/connector'
everest-demo-steve-1 | [ERROR] Location : /steve/src/main/resources/db/migration/V0_7_7__update.sql (/steve/src/main/resources/db/migration/V0_7_7__update.sql)
everest-demo-steve-1 | [ERROR] Line : 1We are setting this up from scratch - do we really need all these migrations?
shankari
commented
1 year ago Known issue https://github.com/steve-community/steve/pull/1213/files, applying recommended fix
shankari
commented
1 year ago That starts it up
everest-demo-steve-1 | . Done!
everest-demo-steve-1 | Hint: You can stop the application by pressing CTRL+C
everest-demo-steve-1 |
everest-demo-steve-1 | Access the web interface using
everest-demo-steve-1 | - http://192.168.2.197:8180/steve/manager/home
everest-demo-steve-1 | - https://192.168.2.197:8443/steve/manager/home
everest-demo-steve-1 | SOAP endpoint for OCPP
everest-demo-steve-1 | - http://192.168.2.197:8180/steve/services/CentralSystemService
everest-demo-steve-1 | - https://192.168.2.197:8443/steve/services/CentralSystemService
everest-demo-steve-1 | WebSocket/JSON endpoint for OCPP
everest-demo-steve-1 | - ws://192.168.2.197:8180/steve/websocket/CentralSystemService/(chargeBoxId)
everest-demo-steve-1 | - wss://192.168.2.197:8443/steve/websocket/CentralSystemService/(chargeBoxId)The manager is still not able to connect to it, but we just need to override the config, I think
shankari
commented
1 year ago We can spin this as "resilience to failure" etc but would like to see if we don't have to take quite as long.
shankari
commented
1 year ago Apparently the "needs DB to compile" issue is known https://github.com/steve-community/steve/issues/420 and is being considered for improvement, but has not yet been improved and might not be improved in the near future. I certainly don't have time to fix it now and the maintainers are pretty opposed to containerization and the tone is pretty unpleasant
shankari
commented
1 year ago so now the connection starts
2023-08-23 04:49:14.567827 [INFO] ocpp:OCPP :: Connecting to plain websocket at uri: ws://steve:8180/steve/websocket/CentralSystemService/cp001 with profile: 0and is received but fails because there is no chargepoint ID
everest-demo-steve-1 | [WARN ] 2023-08-23 04:49:26,819 org.eclipse.jetty.server.HttpChannel (qtp247162961-16) - /steve/websocket/CentralSystemService/cp001
everest-demo-steve-1 | org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.socket.server.HandshakeFailureException: ChargeBoxId 'cp001' is not recognized.
everest-demo-steve-1 | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-5.3.7.jar:5.3.7]let's try to open the admin dashboard and see if we can create the point
shankari
commented
1 year ago After manually adding a chargeBox called cp001 using the blue "Add New" button, the connection worked!
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,452 de.rwth.idsg.steve.ocpp.ws.WebSocketLogger (qtp247162961-29) - [chargeBoxId=cp001, sessionId=3df7f3bc-e076-d275-090f-f2a6eb9d02ac] Connection is established
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,467 de.rwth.idsg.steve.ocpp.ws.WebSocketLogger (qtp247162961-27) - [chargeBoxId=cp001, sessionId=3df7f3bc-e076-d275-090f-f2a6eb9d02ac] Received: [2,"cc7ba7d6-66a9-47e3-a3be-91a5fae49cdf","BootNotification",{"chargeBoxSerialNumber":"cp001","chargePointModel":"Yeti","chargePointVendor":"Pionix","firmwareVersion":"0.1"}]
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,489 de.rwth.idsg.steve.service.CentralSystemService16_Service (qtp247162961-27) - The boot of the chargebox 'cp001' with registration status 'Optional[ACCEPTED]' is acknowledged.
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,585 de.rwth.idsg.steve.ocpp.ws.WebSocketLogger (qtp247162961-27) - [chargeBoxId=cp001, sessionId=3df7f3bc-e076-d275-090f-f2a6eb9d02ac] Sending: [3,"cc7ba7d6-66a9-47e3-a3be-91a5fae49cdf",{"status":"Accepted","currentTime":"2023-08-23T05:14:05.489Z","interval":14400}]
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,593 de.rwth.idsg.steve.ocpp.ws.WebSocketLogger (qtp247162961-43) - [chargeBoxId=cp001, sessionId=3df7f3bc-e076-d275-090f-f2a6eb9d02ac] Received: [2,"938ca1d8-f39a-4184-85b4-6f3029b026b9","StatusNotification",{"connectorId":0,"errorCode":"NoError","status":"Available"}]
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,620 de.rwth.idsg.steve.repository.impl.OcppServerRepositoryImpl (qtp247162961-43) - The connector cp001/0 is NEW, and inserted into DB.
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,628 de.rwth.idsg.steve.ocpp.ws.WebSocketLogger (qtp247162961-43) - [chargeBoxId=cp001, sessionId=3df7f3bc-e076-d275-090f-f2a6eb9d02ac] Sending: [3,"938ca1d8-f39a-4184-85b4-6f3029b026b9",{}]
everest-demo-steve-1 | [INFO ] 2023-08-23 05:14:05,632 de.rwth.idsg.steve.ocpp.ws.WebSocketLogger (qtp247162961-27) - [chargeBoxId=cp001, sessionId=3df7f3bc-e076-d275-090f-f2a6eb9d02ac] Received: [2,"fee091f9-da9c-4733-b08c-71dff605dd18","StatusNotification",{"connectorId":1,"errorCode":"NoError","status":"Available"}]and
2023-08-23 05:14:05.390127 [INFO] ocpp:OCPP :: Reconnecting to plain websocket at uri: ws://steve:8180/steve/websocket/CentralSystemService/cp001 with profile: 0
2023-08-23 05:14:05.446145 [INFO] ocpp:OCPP :: OCPP client successfully connected to plain websocket server
2023-08-23 05:15:05.595665 [INFO] ocpp:OCPP :: Checking if OCSP cache should be updated
2023-08-23 05:15:05.596462 [INFO] ocpp:OCPP :: Requesting OCSP response.
2023-08-23 05:15:05.600287 [INFO] ocpp:OCPP :: Checking if V2GCertificate has expired
2023-08-23 05:15:05.600821 [INFO] ocpp:OCPP :: V2GCertificate is invalid in 0 days. Requesting new certificate with certificate signing request
High level goals are in the README
They already have an
automated-ci-testing-base-imageandThere is a
.cidirectory which hascompile.shandtest_and_install.shThere is a
.github/workflowswhicheverest/everest-ci/github-actions/run-clang-format@v1.0.0ghcr.io/everest/build-kit-alpine:latest