search

dingo35 / SmartEVSE-3.5

Smart Electric Vehicle Charging Station (EVSE)
MIT License
38 stars 13 forks source link

OCPP support #82

Open matth-x opened 1 month ago

matth-x commented 1 month ago

Integrate OCPP 1.6 into the SmartEVSE firmware.

Current TODO list:

dingo35 commented 1 month ago

O boy, this might be a major one, I never noticed that part of the docs.

Thanks for pointing this out, I'm going to look into it tomorrow; I'd like to take these kinds of non-OCPP related stuff separate and commit them as atomic as possible, if thats ok with you.

Thanks!

Matthias Akstaller @.***> schreef op 1 juni 2024 11:07:53 CEST:

@matth-x commented on this pull request.

@@ -2999,15 +3021,6 @@ void Timer1S(void * parameter) { } // while(1) }

-void Mongoose_Poll(void * parameter) {

  • while (true) {
  • if (WiFi.isConnected())
  • mg_mgr_poll(&mgr, 1000);

Each call into the mongoose module needs to happen on the same RTOS thread, as the documentation says: https://mongoose.ws/documentation/tutorials/core/multi-threaded/#overview

I believe that the root cause of the crash you mention is somewhere else and unfortunately, such callback-driven frameworks are notoriously hard to debug. If I find possible causes, I will look for alternative solutions.

In general, preemptive multitasking like FreeRTOS threading is dangerous if the software components are not explicitly protected against it (and most high-level libraries on microcontrollers aren't). I would suggest moving all networking-related stuff on the main task as well as non-time-critical code, i.e. mongoose (including MQTT and the web server), OCPP and Wi-Fi management. Otherwise you always need to bear in mind which task is executing what which makes the code more complex to follow + adds the need for extra synchronization mechanisms.

-- Reply to this email directly or view it on GitHub: https://github.com/dingo35/SmartEVSE-3.5/pull/82#discussion_r1623185455 You are receiving this because you commented.

Message ID: @.***>

matth-x commented 4 weeks ago

Hi @dingo35,

No problem to move the tasks change to another PR. In the end, I could imagine to take all OCPP-related code changes of this PR and apply them in a clean way on a fresh new branch from master. Then it makes no difference if we take out the non-OCPP changes from here.

Anyway, the most recent updates contain:

For the next steps I have two questions:

dingo35 commented 3 weeks ago

Hi Matthias,

Great work! I added some comments, stopped adding the "missing the ENABLE_OCPP here" at some time, you probably get the message :-) .

About  the mgr_init, mgr_poll and mgr_free: For us, mgr_init should be in the wifi event "connected" since we need it to stop/start when the captive portal is started.... Same for mgr_free in wifi event "disconnect" . I don't think this is any problem for you? I will move mgr_poll to loop(), how about I do that in a separate commit in my master, and insulate it from OCPP ?

As to MQTT announce, every topic that MQTT publishes needs to be announce so MQTT clients (like HomeAssistant) can auto-discover it. In your case, the topics do not have a unit of measurement, so place them both after: announce("RFIDLastRead", "sensor"); ... and your topics will appear automagically in HomeAssistant...

As for the web dashboard, we have primitively used a normal text editor but I would love to have a great WYSIWYG editor to maintain it. Of course it should be open source so the community can use it freely. Perhaps we can get rid of that awfull .css too , I don't dare to touch it....

Best regards, Hans.

"Matthias Akstaller" @.***> schreef op 4 juni 2024 om 22:34:

Hi @dingo35 https://github.com/dingo35 ,

No problem to move the tasks change to another PR. In the end, I could imagine to take all OCPP-related code changes of this PR and apply them in a clean way on a fresh new branch from master. Then it makes no difference if we take out the non-OCPP changes from here.

Anyway, the most recent updates contain:

  • RFID authorization: as a first step, I think OCPP should monopolize the RFID reader. If OCPP is enabled and RFID is in EnableOne or EnableAll mode, then the UUID is forwarded to the OCPP lib

  • The HTTP API allows to set the server URL, enable / disable OCPP and view the connection status

  • Two MQTT topics

  • LittleFS enabled

For the next steps I have two questions:

  • What is "announce" in the MQTT area? Do you think it's necessary for OCPP?

  • How to develop the web dashboard? Do you use a WYSIWYG editor or do you develop the code in a text editor?

—

Reply to this email directly, view it on GitHub https://github.com/dingo35/SmartEVSE-3.5/pull/82#issuecomment-2148371317 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOSIX7QJNR3RBGTFLOW22DZFYQF3AVCNFSM6AAAAABITFPLJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBYGM3TCMZRG4 .

You are receiving this because you were mentioned.

matth-x commented 3 weeks ago

Hi Hans,

Thanks for the code review! I changed the menu position as suggested and added the missing ENABLE_OCPP statements (it compiles now when ENABLE_OCPP=0).

The Mongoose lifecycle is actually a critical topic, because my WebSocket wrapper depends on it and so does MicroOcpp, transitively. In the Mongoose docs and examples I never found that initializing the manager in the very beginning of the boot procedure would be a somehow discouraged practice. In fact, I haven't seen any problems if Mongoose is initialized before the Wi-Fi connection succeeds and remains alive throughout network reconfigurations.

OCPP is also active without network connection. If the charger boots into an offline state, most operators want that it still authorizes RFID cards according to local whitelists and captures charging sessions for a later transmission to the server. So it would be important to be able to have OCPP always on.

How hard would it be extend the lifecycle of Mongoose with the current captive portal implementation?

Okay, I will implement the MQTT announce routine by duplicating the RFID code then.

For the web dashboard, it's no problem to work with a text editor to add the few OCPP settings.

dingo35 commented 3 weeks ago

In the current master branch mgr_init is called at WifiSetup, and mgr_free is never caĺled in the normal lifecycle.

Only when the portal is called, mgr_free is called, and at the end, mgr_init again. So usually the portal is only called on first time installation, and perhaps a few other times during the lifetime of the SmartEVSE, when wifi credentials change.

Would that be a problem for OCPP? I can look for a way to only shut the listeners down? In what way is mongoose critical to OCPP? Is there a tcp client, http server, or what? What ports are you using?

Also: do you need certificates on the device? Are they specific to the device or can they be generic?

Keep up the great work!

Hans.

matth-x commented 3 weeks ago

From the user perspective it's no problem to disable OCPP while the Wi-Fi portal is up and running. The only thing necessary is to shut down OCPP before MG is deinitialized and then to start it up again. I looked for possible locations on the recent master branch, but doing so I still found that the MG lifecycle model seems quite complex. The reason to shut down MG is to free the ports 80 and 443 again for WiFiManager? I'm wondering why not unbind the listeners on these ports and keep MG initialized. The other background tasks of MG (i.e. MQTT and OCPP trying to reconnect) shouldn't be a problem.

This would also avoid MG functions being called from another thread here:

https://github.com/dingo35/SmartEVSE-3.5/blob/f60523249f87dc30cc3f5c5017b187abc8072719/SmartEVSE-3/src/evse.cpp#L4769

https://github.com/dingo35/SmartEVSE-3.5/blob/f60523249f87dc30cc3f5c5017b187abc8072719/SmartEVSE-3/src/evse.cpp#L4777

What do you think? I can also make a quick write up in another draft PR so we can discuss about it there.

Also: do you need certificates on the device? Are they specific to the device or can they be generic?

The device needs only one root CA cert for the OCPP server. But between OCPP servers, the root certs can differ of course. How do you plan to do certificate management for MQTT? I think the easiest is to just use the same approach for both protocols.

In OCPP, there is also the possibility to have the certificates managed by the server. I will add that as an extra feature in future, but I'm not sure how widely it is used in practice at the moment.

Keep up the great work!

Thanks, I also appreciate how well this project is made!

dingo35 commented 3 weeks ago

I tried removing those mg_mgr_free and mg_mgr_init lines out of the code, because the listeners ARE already closed when in portal mode:

but the portal becomes unresponsive if I remove those two lines; it might be the closing isn't done completely or in time, or it has to do with the fact that the closing is done on receipt of the first traffic in portal mode (WIFImode == 2) ; probably the latter one. I need the mg_connection *c for the closing, I would hate to have to make a global out of that, also, the portal is started from the menu, and also as a separate task, so not in the mongoose thread.....

How about we close down OCPP completely when in / before going into portal mode, and either restart OCPP or, if that is not possible, reboot?

We don't have any root certificate for MQTT currently, I'm currently integrating an automatic OTA update facility that will use a CA certificate. I guess we would prefer to have that on LittleFS partition to keep the code small, perhaps take care of that on the initial flash? Although if that is programmed, the certificate would be in the code anyways ?

Hans.

"Matthias Akstaller" @.***> schreef op 6 juni 2024 om 20:35:

From the user perspective it's no problem to disable OCPP while the Wi-Fi portal is up and running. The only thing necessary is to shut down OCPP before MG is deinitialized and then to start it up again. I looked for possible locations on the recent master branch, but doing so I still found that the MG lifecycle model seems quite complex. The reason to shut down MG is to free the ports 80 and 443 again for WiFiManager? I'm wondering why not unbind the listeners on these ports and keep MG initialized. The other background tasks of MG (i.e. MQTT and OCPP trying to reconnect) shouldn't be a problem.

This would also avoid MG functions being called from another thread here:

https://github.com/dingo35/SmartEVSE-3.5/blob/f60523249f87dc30cc3f5c5017b187abc8072719/SmartEVSE-3/src/evse.cpp#L4769

https://github.com/dingo35/SmartEVSE-3.5/blob/f60523249f87dc30cc3f5c5017b187abc8072719/SmartEVSE-3/src/evse.cpp#L4777

What do you think? I can also make a quick write up in another draft PR so we can discuss about it there.

Also: do you need certificates on the device? Are they specific to the device or can they be generic?

The device needs only one root CA cert for the OCPP server. But between OCPP servers, the root certs can differ of course. How do you plan to do certificate management for MQTT? I think the easiest is to just use the same approach for both protocols.

In OCPP, there is also the possibility to have the certificates managed by the server. I will add that as an extra feature in future, but I'm not sure how widely it is used in practice at the moment.

Keep up the great work!

Thanks, I also appreciate how well this project is made!

—

Reply to this email directly, view it on GitHub https://github.com/dingo35/SmartEVSE-3.5/pull/82#issuecomment-2153158813 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOSIXZZBQ7SD2YVWEHYHUDZGCTXLAVCNFSM6AAAAABITFPLJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGE2TQOBRGM .

You are receiving this because you were mentioned.

matth-x commented 3 weeks ago

I made a little test before suggesting this. The resulting setup is:

The alternative of shutting down OCPP would mean roughly the same setup, because ocppDeinit() and mg_mgr_free() also need to be called from the main thread. Instead of defining the port listeners as globals we would need other global variables which allow to shut down MG and OCPP from another thread. Not bad, but it adds to the complexity of the system if the MG lifecycle is variable.

We don't have any root certificate for MQTT currently, I'm currently integrating an automatic OTA update facility that will use a CA certificate. I guess we would prefer to have that on LittleFS partition to keep the code small, perhaps take care of that on the initial flash? Although if that is programmed, the certificate would be in the code anyways ?

Actually I think that compiling the certificate into the firmware has some advantages, if done automatically. The built-in OTA procedure of espressif is well tested and comes with fail-safety and integrity check measures. We would effectively reuse all of that. On the other hand, for some users that's no option, because they want a pre-programmed device where they only need to upload the cert file over the web dashboard.

For OCPP (or MG in general) it's no problem to work with built-in certs or certs from the data partition, both manually uploaded certs or stored as factory default.

dingo35 commented 3 weeks ago

Inline!

"Matthias Akstaller" @.***> schreef op 7 juni 2024 om 12:56:

I made a little test before suggesting this. The resulting setup is:

  • Port listeners 80 and 443 become globals

  • Before starting WiFiManager, unbind the ports by setting the flag port_listener_80->is_closing = 1; (same for 443)

  • Wait for one loop() iteration on the main thread to be executed (the next mg_mgr_poll() closes the connection). However I'm not sure if that's actually needed, didn't do it in my test and the WiFiManager web page still showed up

  • Start WiFiManager

Ok lets do it this way. Since you tested it, include it in your commits, no use for me doing it all over again.

The alternative of shutting down OCPP would mean roughly the same setup, because ocppDeinit() and mg_mgr_free() also need to be called from the main thread. Instead of defining the port listeners as globals we would need other global variables which allow to shut down MG and OCPP from another thread. Not bad, but it adds to the complexity of the system if the MG lifecycle is variable.

We don't have any root certificate for MQTT currently, I'm currently integrating an automatic OTA update facility that will use a CA certificate. I guess we would prefer to have that on LittleFS partition to keep the code small, perhaps take care of that on the initial flash? Although if that is programmed, the certificate would be in the code anyways ?

Actually I think that compiling the certificate into the firmware has some advantages, if done automatically. The built-in OTA procedure of espressif is well tested and comes with fail-safety and integrity check measures. We would effectively reuse all of that. On the other hand, for some users that's no option, because they want a pre-programmed device where they only need to upload the cert file over the web dashboard.

For OCPP (or MG in general) it's no problem to work with built-in certs or certs from the data partition, both manually uploaded certs or stored as factory default.

—

Ok lets go for the builtin stuff as much as possible!

Reply to this email directly, view it on GitHub https://github.com/dingo35/SmartEVSE-3.5/pull/82#issuecomment-2154596210 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOSIX2GD7TNERM7MEBCCDTZGGGV7AVCNFSM6AAAAABITFPLJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJUGU4TMMRRGA .

You are receiving this because you were mentioned.

matth-x commented 2 weeks ago

The initial integration is ready now!

To comment on the latest changes: in 3ee0fb19e856e3b6e67696f2ada85bc6f1861335 I implemented the Mongoose HTTP server lifecycle as described in https://github.com/dingo35/SmartEVSE-3.5/pull/82#issuecomment-2154596210. In the next commit 71e84128517a342b2e8b51c83c2190ac35273711 I recreated the MQTT timer lifecycle that it stops while the WiFi connection is lost. I think that it wouldn't be harmful if that timer continued running (maybe at a lower frequency), but it's no hassle to stop it.

For Smart Charging, the connector lock (and RFID authorization) I implemented "OCPP-only" modes, i.e. OCPP takes over full control over the connector lock and RFID (if in mode EnableOne or EnableAll) and the load balancer if the load balancer is disabled.

For firmware updates and diagnostics uploads I finally made the built-in solution of MicroOcpp ready (which will also be tested in other installations). We could, however, customize the OTA process and add more hardware diagnostics, if we have use cases which require that.

fluppie commented 2 weeks ago

Nice, tried both the fork of @matth-x and the merged one here. With the fork of Matthias I can connect to E-Flux OCPP_1

With the merged one here I doesn't connect. Pretty weird?

First test with remotestart and remotestop seems to work. Only to be shut off after 1-2s since the station and tag is not authorized. But I hear the contactor clicking.

afbeelding

afbeelding

Tagging @dingo35 @mstegen so they know I'm already fiddling around.

They only thing that I cannot get to work is the RFID reader. Looks like the local rfid.txt list is disabled/ignored (what I've also read here in the conv.) but the RFIDLastRead is not getting updated when holding tags in front of the reader. So looks like it's really shut off?

matth-x commented 2 weeks ago

Hi @fluppie, thanks for taking a look at the OCPP integration!

The existing RFID whitelisting is fully bypassed in this first integration. OCPP catches the RFID events and doesn't forward them to SmartEVSE again. So RFIDLastRead remains blank. But you should see Authorize messages arriving at the server.

To use RFID-based authorization with OCPP, the RFID settings must be "EnableOne" or "EnableAll". To use OCPP Smart Charging, the SmartEVSE load balancer must be turned off so that it's in mode "0". Not too user friendly, but should do the job in the beginning.

fluppie commented 2 weeks ago

I can't get the RFID stuff to work. Tried it on a real one connected to a vehicle and then on a EVSE next to my computer with only an RFID reader connected. is there a way to see if the EVSE is actually reading a tag? Maybe we should flash the red and green LED's on the reader at the same time when a tag is read? (that's more towards @dingo35 and @mstegen ) So we get visual feedback that something is happening. For example on the Alfen Eve charging points there's also a buzzer that beeps when you hold a tag in front of the reader. Some "succes" beep and a "rejected" beepbeep.

BTW did you also try to build the merged version here? when I compile from Dingo35's repo it will not connect to the backoffice. Compiling from your repo works.

mstegen commented 2 weeks ago

Just noticed the RFID readers we use send 'only' the 6 LSB of the UID, while Mifare cards can have a 4, 7 or 10 byte UID. With the SteVe simulator this works fine, but it might not with a real backend. Will ask the manufacturer of the RFID readers is there is a way to extend the number of bytes that can be read.

Yes good idea. The LED should flash if the card is accepted or not.