Simulator improvements

[shankari]

• roll forward to master
• ensure that we can see the energy fluctuations
• show ISO + OCPP messages in the UI (either on the side or in the debug screen)

[the-bay-kay]

Noticed a regression in the Node-RED behavior when updating to the latest version of the flows:

• When choosing a simulation method, there are a set of commands used to start, stop, pause, and resume the commands (see this thread for some basic context...)

• Within the current version of EVerest's ISO15118-2+OCPP2.0.1 Demo, we only have commands to start and stop the simulation under the "Plug & Charge" method: if we attempt to pause, we receive the following error, and crash:

The command sequence has changed enough between versions that we cannot just copy-and-paste the old sequence, so let's splice in our modified dt/eamount cmd and add back in the pause / resume functionality...

EDIT: Well, splicing in the commands was easy enough! However, the pause/resume functionality is not the same as the non-P&C. We pause the session OK, but when attempting to resume, run into this certificate error...

In the interest of time (we want the demo rolled forward ASAP), I'll leave this a known regression and we can inspect after the dust has settled.

[the-bay-kay]

Let's make a checklist of areas I need to update for the roll-forward:

• Nodered:
  • [x] Update flows to include changes up to Renegotiation *
  • [ ] *Fix the regression described above
  • [x] Update Dockerfile to be at parity with renegotiation demo
• Manager:
  • Compiled Modules
  • [ ] Create a fork of 2024.9.0-rc2 to add module changes to
  • [ ] Temporarily update manager/Dockerfile to build off using fork
    • I think this is the best approach -- since I'm unsure which release the main of everest-demo is on, we want to ensure we're building off of the September Release. Please correct me if there is a better way to live update everest-core while working on the demo! (e.g., should we just patch after pulling?)
  • [ ] Update EvseManager.cpp, OCPP201.cpp
    • [ ] Double check there are no other files to be changed on the C++ side of thing (e.g., do we adjust iso_server.cpp?)
  • Simulator:
  • [ ] Update PyJosEv's /evcc/ files
  • [ ] Add powercurve.py for powercurve generation

[shankari]

I think this is the best approach -- since I'm unsure which release the main of everest-demo is on, we want to ensure we're building off of the September Release. Please correct me if there is a better way to live update everest-core while working on the demo! (e.g., should we just patch after pulling?)

@the-bay-kay I have been patching after pulling - e.g. https://github.com/US-JOET/everest-demo/blob/879548aafdf9ff2366de45f1b77cd25d5667ebb2/demo-iso15118-2-ac-plus-ocpp.sh#L354

I would start with that, and switch to having manager/Dockerfile build off using fork if the number of patches gets too large.

[the-bay-kay]

Switching to 2024.9.0-rc2, we encounter the following config error...

It seems Shankari ran into a similar issue on this previous issue when building on the uMWC - presumably, I'll need to copy over the config files from the last known working demo (e.g., 2024.3.0) -- I need to run for some evening appointments, but will investigate further after...

[shankari]

presumably, I'll need to copy over the config files from the last known working demo (e.g., 2024.3.0)

That is almost certainly not the correct approach to take. I am not sure where you are getting the config from (you haven't indicated what you did to accomplish "switching to 2024.9.0-rc2"). However, if there is a mismatch in modules, it is almost certainly due to an old config, referring to an old module, being copied over, and the module being renamed in the current release.

In that case, the old config is the problem and copying it over won't fix anything

[the-bay-kay]

...you haven't indicated what you did accomplish "switching to 2024.9.0-rc2"

Apologies for the lack of clarity -- I've switched the version in manager/Dockerfile like I described in the checklist above (so we pull & build 2024.9.0-rc2). The config conflict makes sense. Let me read more closely through the dockerfile and corresponding config.jsons, to see if I missed any versions there. Likewise, since the error message explicitly mentions the manifest.yaml of JsCarSimulator is missing, I'll see what has changed in that module between 2024.3.0 and 2024.9.0-rc2

[the-bay-kay]

Let's trace back and see where this fails:

• We build using demo-iso15118-2-ac-plus-ocpp.sh, modified to point to manager/Dockerfile: this builds without error
• Within Docker Desktop's exec, we run sh /ext/source/build/run-scripts/run-sil-ocpp201-pnc.sh, which then fails with the error described above
• Looking at the contents of run-sil-ocpp201-pnc.sh...

  LD_LIBRARY_PATH=/ext/source/build/dist/lib:$LD_LIBRARY_PATH \
  PATH=/ext/source/build/dist/bin:$PATH \
  manager \
  --prefix /ext/source/build/dist \
  -- conf /ext/source/config/config-sil-ocpp201-pnc.yaml \
  \
  $@

  We launch using the config /ext/source/config/config-sil-ocpp201-pnc.yaml. So, looking at that...

Aha. If these release notes are to be believed, it seems that JsCarSimulator was replaced with JsEvManager. We're copying over an old config from everest-demo, that does not match the one in core linked above. So, let's copy over the updated configs...

[the-bay-kay]

Success -- updating the config got us past the manifest loading. It seems we have to update the OCPP database file as well -- we reach the following failstate:

Fail to Boot: Database out of date

```bash 2024-10-22 16:53:27.114732 [INFO] ocpp:OCPP201 :: Established connection to database: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db" 2024-10-22 16:53:27.115098 [ERRO] ocpp:OCPP201 void ocpp::v201::InitDeviceModelDb::execute_init_sql(bool) :: Database does not support migrations yet, please update the database. terminate called after throwing an instance of 'boost::wrapexcept >' what(): Database does not support migrations yet, please update the database. ```

We create device_model_storage.db when we run...

COPY device_model_storage_maeve_sp1.db ./dist/share/everest/modules/OCPP201/device_model_storage.db

... So let's look for an updated database file and slot that in. At a glance, the only file with the same name as device_model_storage_maeve_sp1.db I can find in the EVerest organization is the one described above in the demo repository -- so, let's widen our search to (i) the MaEVe repo, and (ii) other .db files.

[the-bay-kay]

With no obvious lead as for a replacement db file, let's trace back:

• This error was initially added in this commit, for PR 681.
• Looking at the EVerest Zulip, it seems I'm not the first person to be confused by the new OCPP database system. That thread points me...
• Here, to the documentation for device initialization via the new configurations.

If I understand the documentation correctly, the device_model_storage_maeve_sp1.db is a custom database file that is not up to date. So, two questions:

• Is there an updated database file that I should be using that I have not found yet?
• If not, how should I go about creating a new version of this file?

@shankari , if you have any insight on working with the updated database schema, that would be greatly appreciated : )

I'll continue to read through the docs to see if I missed anything, and get a better understanding of the initialization process.

EDIT: Under the "to-do" for OCPP 2.0.1 integration, it says to use the provided database file... but I haven't been able to find this...

[shankari]

@the-bay-kay https://github.com/EVerest/everest-demo/issues?q=is%3Aissue+sqlite+is%3Aclosed

[the-bay-kay]

My database experience has been limited almost entirely to MongoDB and NoSQL systems, so I am approaching this with a beginner's eye. With that said, let's learn along the way:

• Our issue is that the database does not support migration. Database migration was added relatively recently to EVerest, and allows users to update their DB's without destroying existing data.

• Question: How do we enable migration?

  • Well, reading through the docs linked above, they say...

    Old databases need to be removed so a new database can be created using the migrations. This is to make sure that there is exact control over the schema of the database and no remains are present.

  Let's put a pin in that for now.

• The error is thrown on this line, when get_user_version() == 0. We can confirm this is the case for our database file by running...

  sqlite> PRAGMA user_version;
  0

• Going back to the docs, we can learn more about the purpose of the user version:

  ...The version numbers of the files will be used together with the database's user_version field to determine which migrations files to run to get to the target version.

Because our database was created before the new initialization process, it will not follow the same "verion history" described within the design considerations. It seems the only way to move forward is to (as mentioned above) create a brand new database, using the new schema and internal database data (e.g., user_version >= 1). Is there really no way we can retrofit an old database?? I'll look into creating a new database with the init file now...

---

EDIT: Let's read through this... This seems to detail the process of updating components using a custom .json file. This seems similar to what we are doing now, but not identical (e.g., component config should be in component_config/custom, which we do not touch). I think this is good to know about, but not the "update" process we need.

[shankari]

Please see the way in which I created this database earlier

[the-bay-kay]

Please see the way in which I created this database earlier

To confirm, I do need to re-create the MaEVe database? Do you have a specific thread I can reference? Looking at the commit history of everest-demo, the method for creating OCPP 201 Device Model Databases has changed since the maeve files were updated 7 months ago (we now use a build target instead of the python script). I wasn't able to find a specific build process in the issues you linked above, but I may have missed something -- I'll read through again. Looking at this PR, I can see that the citrine demos were added <4 months ago... Let me know if I'm missing something, otherwise I'll try re-creating the databases to support migration.

[shankari]

This is not the MaEVe database - it is the EVerest database. MaEVe is the CSMS layer and should not affect the running of the station software. I put in the link to find the issues where I used sqlite to edit the database file.

You should follow those instructions to copy and edit the file properly.

[the-bay-kay]

This is not the MaEVe database - it is the EVerest database.

Right -- but the file we copy over is called device_model_storage_maeve_sp1.db. Referring to it as "maeve database" may have been a confusing shorthand on my part.

I put in the link to find the issues where I used sqlite to edit the database file.

If you're referring to this thread, resetting the host as described did not fix the issue. Nor is it an issue of an incorrect url. Around release 2024.7.0, the database was substantially changed. Relevant to our work is the adoption of a migration stratedgy, which relies on table parameters our database does not have (hence the "user_version()" error we receive: the solution isn't as simple as bumping this value up to 1).

All of the demo-repository's databases were created roughly in release 2024.3.0 (e.g., the version the demo is based off of in manager/Dockerfile): any database created before the refactor is not compatible with the current schema. Please see this Zulip thread where I got more clarification on this from the Everest Cloud Communication group. The community suggests re-creating the database to ensure that it is compatible with the latest schema.

[shankari]

@the-bay-kay you are working on upgrading the SIL. So the steps to take the SIL and run it on the uMWC are not relevant to you. That's why I suggested that you search for sqlite in general. You could also see when the lines to add the custom DB were added to the Dockerfile.

It was added in https://github.com/EVerest/everest-demo/pull/19 which referred to https://github.com/EVerest/everest-demo/issues/25 which shows the edits to the sqlite database, including copying it back and forth

The community suggests re-creating the database to ensure that it is compatible with the latest schema.

I don't disagree. The point I am trying to make is that the database checked into the codebase was created by editing the database that was created by EVerest automatically at startup.

After the correctly formatted database is created automatically at startup

You should follow those instructions to copy and edit the file properly.

[the-bay-kay]

So, in order to re-create the database, we need to build EVerest without inserting a custom database. So, let's comment out the copies:

File Changes

`manager/Dockerfile` ```Dockerfile # Copy over the custom config *after* compilation and installation # COPY config-docker.json ./dist/share/everest/modules/OCPP/config-docker.json # COPY config.json ./dist/share/everest/modules/OCPP201/config.json # COPY device_model_storage_maeve_sp1.db ./dist/share/everest/modules/OCPP201/device_model_storage.db COPY run-test.sh /ext/source/tests/run-test.sh ``` `demo-iso15118-2-ac-plus-ocpp.sh` ```bash elif [[ "$DEMO_VERSION" =~ sp3 ]]; then echo "Copying device DB, configured to SecurityProfile: 3" # docker cp manager/device_model_storage_maeve_sp3.db \ # everest-ac-demo-manager-1:/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db fi ````

When we run like this, we're successfully generating the database!

...but, as expected, fail to connect.

Let's compare the old file and the new template, and get this one up to speed...