Open the-bay-kay opened 1 month ago
@the-bay-kay I believe that the plan is for the "optimized curve" to be computed on the car.
ChargeParameterDiscoveryRes
lowest_viable_power
and maximum_supported_power
PowerDeliveryReq
Note that the important part, per DJ and Jacob, is (2) since the car should have control of the charging schedule for the departure time. We are only implementing (1) because it lets us showcase multiple SASchedules without fully supporting tariffs (and because I want to 😄 ) So you might want to tackle (2) first, since even if we don't finish (1), we will have a clean demo.
I believe that the plan is for the "optimized curve" to be computed on the car.... So you might want to tackle (2) first, since even if we don't finish (1), we will have a clean demo.
Aaah, right! Cool cool -- sounds like a good plan!
Updated Timeline and Issue name to reflect the project's immediate goals.
note that node-red has a charts module: https://stevesnoderedguide.com/using-the-node-red-chart-node https://flows.nodered.org/node/node-red-contrib-graphs
... including (i) plotting coefficients, and (ii) Sending an image
Let's start by investigating the first option. As currently configured, our Node-RED does have a chart function, though it seems to be geared to presentation-style charts (Bar Charts, Pie Charts, etc.). I don't believe we would be able to plot the anticipated equations.
Node-RED does have a curve plotting module we could add, that seems geared to plotting functions specifically! (link)
note that node-red has a charts module: ...
Likewise, this looks like a good option -- may be more flexible than the curve module, will compare the two options and go forward from there!
Assessing our options:
contrib-graph
module is that it does not plot to the ui (http://127.0.0.1:1880/ui): Rather, it plots to (http://127.0.0.1:1880/dashboard/). While this isn't a dealbreaker, setting two windows side-by-side isn't ideal (especially when we already have a modular UI).Let's take a shot at (2b) first, before exploring alternatives. My hope is that we can insert the dt
and eamount
flow nodes into the function as "captured" variables, and use the function input for our "K" variable.
can't you also just plot a line graph and have DJ give you a set of coordinates?
can't you also just plot a line graph and have DJ give you a set of coordinates?
I was searching for an option to plot via coefficients like we discussed a few days ago, but that is an option. I'll go ahead with that approach!
I was searching for an option to plot via coefficients like we discussed a few days ago, but that is an option. I'll go ahead with that approach!
Right, I think that plotting a standard quadratic curve would be more efficient, but if that is complex, we can fall back to the option that is known to work, at least for now
Just wasted a solid hour and a half trying to fix this bug... turns out the old curve module I was experimenting with bricked the UI... Anyway, we've got a basic chart running: let's write the function and generate some datapoints...
When attempting to plot an array of data, we do so for the points, but do not receive a line as we would expect. My intuition says this is either (i) an issue with how we're formatting the inbound data, or (ii) an issue with how the chart is configured to accept the data.
Another minor issue with the line-chart method of plotting is that I have not figured out how to use non-timestamps within the X axis label. We can theoretically define custom labels (setting to "automatic" attempts to format as a time stamp), but it's uncertain if we the custom labels need be tied to a timestamp format like the defaults (HH:MM, for example)
The x axis does represent time - the curve represents power drawn during the charge session, right?
... we do not receive a line as we would expect. My intuition says this is either (i) an issue with how we're formatting the inbound data, or....
Yup, this was my problem. Changing the way the arrays were nested, we get an OK result. Let's add some templates for the powercurve data next
We're successfully plotting multiple curves -- let's hook up the MQTT messages now...
Before making the changes to everest-core/modules/EvseManager/EvseManager.cpp to enable MQTT broadcasting, let's better understand a few things, including
Effectively, there's two modes for our graph: "Preview", and "Actual". Since we've got the rough preview working, let's see how we'll graph the "Actual" curve, once the optimizer is run on the SASchedule.
To re-render the graph, it seems like we receive ChargeParameterDiscoveryResponse (and likewise, the SASchedule) in evcc/states/din_spec_states.py, specifically in ChargeParameterDiscovery(). My first thought is that we will want to utilize EVEREST_CTX's publish()
method, as we are already doing with AC_EVPowerReady. That being said, the definition of this publication seems vague... Let's look for where this publication is received, or try to find the way we publish via MQTT.
Regardless, it's good to find this chunk of code where we find the SASchedule -- once we add DJ's optimizer, it should slot in roughly here.
While tracing these calls, I've been sketching out a rough sequence diagram to highlight the important calls within this process (attached below the cut).
An aside -- here is a sample of the powercurve graph preview, changing dynamically with the scale and departureTime -- this is with a dummy linear function for now, but switching to a different (quadratic) function should be as straight forward as swapping the function.
In testing, I into some limitations -- Node-RED freezes when attempting to chart a large number of points with a large range. If necessary, we can limit the "sample rate" of the visualization, to reduce the number of points being charted (e.g., plotting the value every minute instead of every second may increase the speed of rendering more complex functions)
... Let's look for where this publication is received, or try to find the way we publish via MQTT.
We know the answer to the first half, at least. We subscribe to the context publications within JsEvManager's index.js. Let's start by close reading this file to look examples of MQTT publication -- after all, JsEvManager receives the initial commands through an MQTT subscription...
While we subscribe to MQTT broadcasts in EVManager's car_simulatorImpl.cpp, it appears we do not publish to MQTT as we do in EVSEManager's evse_managerImpl.cpp. Since PowerCurve selection should occur within the EV (context), we'll likely need to add publication to this file. So, follow up question: how will JSEvManager communicate with car_simulatorImpl? Let's investigate...
there will be an additional step between receiving an SASchedule and initializing charging.
This is the PowerDeliveryReq
that the EVCC sends to the EVSE. It is already sent as part of the ISO message pair
Out of the myriad of locations where we find PowerDeliveryReq, it seems that we want to add the optimizer to the async process_message function within the PreCharge Class in ext-switchev-iso15118/evcc/states/din_spec_states.py
, confirmed when comparing it to the PowerDeliveryReq() logs (screenshot below). Let's trace from this to see where we can receive this information in Node-RED
DIN != ISO, it is a precursor to ISO. I would look to see where I had to edit the departure time in my previous set of changes. Maybe it is in the din_states
file, but I would expect to find it in the iso_states
or similar file
In din_states
, it seems we'll want to modify iso15118_2_states.py's process_message(), specifically around line 807. Knowing where we want to change this, there are a few unanswered questions:
control
and numpy
libraries that the script utilizes, but this seems like re-inventing the wheel. This is within possibility, but will bulk up the Node-RED code substantially. (The Math.js library has some, but not all of our equivalent functions -- matrix(), etc.). index.js
. The question then is how. Do we follow the example of JsYetiSimulator's index.js? Does the mod
object perform the same? Or, is this publication in JsEvManager
what we should use as a template? This seems like gilding the lily -- out of all of these options, the first seems to be the best.
After some discussion with the team, we've opted to go for "B" : Running the python script from Node-RED for the preview. This will leave us with less overhead in the preview (we won't need to wait for the MQTT messages to bounce around), and saves the hassle of re-inventing the wheel.
Currently, we've got basic files running on Node-RED -- an example script can be launched from the UI, and an output can be read. The kicker is installing additional Python packages. We need both Numpy and Control, and ideally want to upgrade to Python 3.x (Our Node-RED image is currently on 2.7). From what I understand, we'll need to re-build the docker image, editing the dockerfile to install the correct pip packages (relevant thread here). So, we need to (i) find the dockerfile (E.x., the manager dockerfile is here), and (ii) re-build. Ideally, this change should be similar to the recent changes to Everest-Manager here.
The relevant Dockerfile is here: so, let's make the changes...
Changes have been made to nodered/Dockerfile
, now we are (temporarily) building locally. Now that we are doing so, we can use apk
to install PIP, and then run pip to install the necessary dependencies. My initial tests failed, but I believe this is because I was installing / running pip incorrectly (we install python-devel OK, but I believe need to call the correct version of pip installed with the devel package...)
Is there no existing python install on that container? Did you try docker exec
and then run python? Most unix distros come with python pre-installed now
... Most unix distros come with python pre-installed now
Correct! Our docker comes pre-packaged with Python 2.7.18: the issue is, it does not come with pip (we need this to install control
and numpy
for the optimizer). See:
why not use conda like we use elsewhere? You can download miniconda like we do in e-mission, it can set up python3 and comes with pip
why not use conda like we use elsewhere?
Not a bad idea -- that seems cleaner than doing the python package management within the Dockerfile (e.g., I've successfully updated Python, still having issues with pip...). Let me try adding miniconda to the container now!
Narrowing in on the installation issue -- it seems both 2.7 and 3.9 come preinstalled (neither with pip)., the issue is that we are not correctly executing apk add
(likely, because we are not correctly switching to root). We should be able to install pip3
or conda
once we fix this.
When running the installation script for miniconda link, we run into the following issue during execution: After accepting the license and choosing the default location, the installation terminates with the following:
Currently looking into what may cause this issue...
I tried creating the container using docker-compose.yml
and ran into a different issue, which is related to installing on Alpine Linux https://superuser.com/questions/1621329/conda-install-in-alpine-image
~ $ curl -o miniconda.sh -L https://repo.anaconda.com/miniconda/Miniconda3-py39_23.5.2-0-Linux-x86_64.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 89.0M 100 89.0M 0 0 8796k 0 0:00:10 0:00:10 --:--:-- 10.2M
~ $ bash miniconda.sh -b -p $HOME
ERROR: File or directory already exists: '/usr/src/node-red'
If you want to update an existing installation, use the -u option.
~ $ bash miniconda.sh -b -p $HOME/miniconda3
PREFIX=/usr/src/node-red/miniconda3
Unpacking payload ...
miniconda.sh: line 353: /usr/src/node-red/miniconda3/conda.exe: No such file or directory
I can retry with the full multi-tier docker compose but let me see if I can figure it out with this first.
Idea from: https://stackoverflow.com/a/62555259, ensurepip docs: https://docs.python.org/3/library/ensurepip.html
Woot! That was the main roadblock -- we're running the script now 🥳 Let me hook up the UI inputs, and I'll push my changes and move on to the simulator portion.
Had to futz with some of the inputs, but we now can use the UI to change these values! There seems to be little change when the ks
value is changed -- focusing on getting this working, we can investigate later...
Currently editing the python script to "return" a JSON object, so that the chart can read these values in the correct format!
We're finally plotting the curve!! The output is a bit wonky -- there are some issues plotting the YC
component, but it is being plotted. In the interest of time, I plan to push what I have, and move on to adding the powercurve function to the simulator. Below is a recording of the dynamic curve plotting. Once changes are pushed to my branch, I will update with steps to reproduce.
To Reproduce:
bash demo-iso15118-2-ac-plus-ocpp.sh -r $(pwd) -b test-demo -3
node-red
docker container, and navigate to the exec
tab (alternatively, perform these commands with docker exec
)'python3 -m ensurepip
/usr/src/node-red/.local/bin/pip3 install numpy control
TODO:
Adding a simple logger.info('Hello World!')
to the place where I thought we needed to add the powercurve to PowerDeliveryReq (link), we don't see the log within a charging session (plug-in, charging, unplug).... Perhaps I was off the mark? Let's look for where else we could add this
EDIT: It should be noted that this may not necessarily be an issue with iso15118_2_states.py
, there are other locations where the PowerDeliveryReq
is used. Let's explore those first before weeding through the other 22 files
Perhaps we should cut out the middle man, and edit the shared message directly? Let's try that... Let's trythe PowerDelivery class specifically....
It seems the existing build_current_demand_data is only for DC, so if we want to modify the AC charging event, we may need to modify this... First, let's explore the EVCC's simulator.py, specifically get_charge_params_v2
To get a hack-y demo working, our current progress has included:
evcc/simulator.py
's process_sa_schedules_v2 Using the PMax and Duration_Time from secc_schedule.p_max_schedule.schedule_entries
.
EAmount
from these values (e.g., use PMax and DT to re-calculate the Watt-Hours )-- at the moment, the scalar factor is hardcoded (which we have discussed), and we are hard-coding EAmount. We'll need to shift some things around to get these values s.t. the curve can be also passed to PowerDeliveryReq
paho-mqtt
back to ext-switchev-iso15118
, in order to broadcast this updated curve to Node-RED. (I say "back", because this repository used to use paho-mqtt once upon a time!) We've successfully connected this to Node-RED, all that is left is to adjust Node-RED for the visual.
Some unexpected behavior: When running a paho-mqtt
from a preview script, it behaves ok...
But when adding the exact same call to evcc/simulator.py
...
curve_vals = LQRchargecurve(departure_time, eamount, ks)
publish.single("everest_external/nodered/{}/evcc/powercurve", "!dlrow olleH", hostname="mqtt-server")
We fail to start charging, with the following error (We know this is the issue, as removing the publication allows us to charge as expected -- the LQR runs OK)
While we aren't communicating via MQTT in this module, perhaps we're crowding the outgoing messages, and have some overlap elsewhere? Let's explore this further...
EDIT: Publishing within iso15118_2_states.py
causes no issues - let's see if we can get that method to work instead
Finally got the curve data sent to MQTT!! Had to return the curve data with process_sa_schedule_v2
, and then send it within iso15118_2_states.py
(had to convert to a string) -- just need to finish the Node-RED plotting log, and then we should be good to go!
We've got a working demo! 🥳
Below is a recording of the demo -- highlights include:
There are definitely some kinks that we need to iron out, but exciting to see it working. Once I get the relevant code committed, I'll go ahead and post steps to re-produce this demo.
https://github.com/user-attachments/assets/3f12d719-2fe6-479f-9d84-12cb9f45bc25
With the Node-RED visualiations working, and the powercurve correctly added, the next step is to adjust the simulation process in order to reflect the EVCC's powercurve.
I believe the best way to do so is by generating a ChargingProfile
for the PowerDeliveryReq
, sent directly after the ChargeParameterDiscoveryRes()
. I believe this approach:
One drawback to this approach is the limitation of the ChargingProfile
type. According to Table 71 of the spec, we are allocated a maximum of 24 ChargingProfileEnteries. As such, we'd only be able to "sample" 24 points of the curve -- E.g., if a charging session occured over 12 hours, we'd adjust the power draw every half hour.
Below is a sequence diagram roughly outlining the flow of PowerCurve Generation & Adoption:
@the-bay-kay what is the timing of these steps? Is there actually potential to wait for the user to select a power curve? I assume not, but I wonder if the spec only requires strict timing after the PowerDeliveryReq and whether there is room for limited user interaction that is compatible with the spec
@the-bay-kay what is the timing of these steps? Is there actually potential to wait for the user to select a power curve? I assume not, but I wonder if the spec only requires strict timing after the PowerDeliveryReq and whether there is room for limited user interaction that is compatible with the spec
Good question -- as written, we immediately shoot off the PowerDeliveryReq after receiving the ChargeParameterDsicoveryRequest. Looking at the spec...
Other than these 2 clauses (additionally, V2G2-848 for the "stop" PowerDeliveryReq), it seems there are no restrictions on timings within the spec. I see no reason why we couldn't pause for user input at this point, when we want to add algorithm selection down the road!
EDIT: this diagram suggests there should be a 250 second requirement after the ChargeParameterDiscoveryRequest, but I could not verify this in the spec. Correct me if I'm missing something! Likewise, we'd be making the changes after the Resolution, so even if this were the case, I'm not entirely sure it'd apply
I've official got the new schedule generating, as described above, but am running into a timeout when attempting to send the entire schedule. Below is a copy of the error, and the two messages being sent (the latter causing the error). Currently investigating what may cause, this, will update accordingly... My first thought is to try a shorter schedule, perhaps this is too much text for a single message?
EDIT: Yup, that's looking like the issue. Paring down the new schedule to 4 items as a test, we start charging O.K.... So what gives? I guess Table 71 of ISO 15118-2 says we can have a maximum of 24, but maybe EVerest doesn't support that yet... Let's see exactly how many we can fit in before we reach the timeout
EDIT 2: Well, when we pass a schedule of length 23, it works fine!! Weird... We can investigate that discrepancy later -- for now, this seems to be working!! I'll include a video and some future steps after this.
Good news -- @shankari , like we expected, we're able to interrupt the charging initialization during process_sa_schedule_v2
. Below is a video of this working -- just need to send the charging curve data before the interception, and the "resume" function will return the curve that we use!
https://github.com/user-attachments/assets/d0f2f817-ec36-491c-97f4-ab3e548de733
Looking for our options to dynamically display a choice to the user, this appears to be non-trivial. At first I thought nodered-dashboard's ui-notificaiton would be our answer, but this does not allow us to display selections. I was hopeful that Node-RED's API RED.notify would be the final answer, but we receive the error below when attempting to call this.... We're using Node-RED 2.2.3, so our version should be compatible with this API. Am I misunderstanding the context which it's used? I briefly explored using an implementation similar to this, but was unsuccessful (I'm assuming it was a conflict with the dashboard package we use?) Will keep de-bugging the RED.notify method, and update with results.
EDIT: Yup, I was misunderstanding -- it appears RED.notify is used for the implementation of Node-RED modules. If we were designing our own modules, this may be helpful. Unfortunately, we need something a little more "out of the box"...
Much to my chagrin, I think the best option (to get this working, at least) is to have a dropdown
component that is dynamically populated via the MQTT in. A notification will then prompt the user to make a selection. Upon doing so, the selection will be sent back, and we will clear the dropdown. This is far from elegant (We'll have any empty dropdown hanging around the UI), but I think it will at least get us to a working state.
I think the best option ...a drop down dynamically populated via the MQTT in. A notification will then prompt the user to make a selection... and we will clear the dropdown.
Done! I need to fix the curve preview again, but that should be straight forward. Good news is, the mid-charge selection process is now working! Below is a video of this in action -- for the sake of our code and my sanity, I'll wrap up tomorrow after some sleep 😀
https://github.com/user-attachments/assets/5d47c828-c8a1-4f81-8f7a-612d443c98e0
We've got a functioning demo! Below is a video of the demo being run: the user may select from one of two "preset curves" (a stand in for our algorithm selection), and then the charge session will occur accordingly. Users are able to preview curves during the idle
phase, and user inputs are used when calculating the curve presets.
While exciting, this isn't necessarily a "camera-ready" demo -- I still want to fix the notification process, for example. After showing this off to the team tomorrow, I'll create a series of clean-up tasks for this project.
https://github.com/user-attachments/assets/2dc64598-fec1-49a3-a5a7-4fe12fb6d37a
Context
This PR builds off @shankari and I's work in Issue #64 . This work will be done on the functional DepartureTime demos.
High Level Goal:
... to present the user with multiple charging profiles, which they then can select and use for their current charging session
Subsequent Goals:
Edit: Updated timeline to reflect 2-week sprint goals. Multiple SASchedules is a stretch goal, implementation of power curve selection is higher priority.
EDIT 2: Changed goals to reflect implementation adjustments.