Closed shankari closed 1 month ago
shankari
commented
3 months ago /mnt/user_data/opt/everest != /opt/everest. You are running the pre-installed manager from /opt/everest It is also not clear where you are running this from ./ just indicates "current directory"
faizanmir21
commented
3 months ago I was in /mnt/user_data directory but like you said its not the same.
Starting the manager again
sudo mnt/user_data/opt/everest/bin/manager --conf /mnt/user_data/opt/everest/etc/everest/config-sil-ocpp201-pnc.yamlRan into this error
2024-08-16 06:18:49.065844 [INFO] ocpp:OCPP201 :: Successfully retrieved Device Model from DeviceModelStorage
2024-08-16 06:18:49.066833 [ERRO] ocpp:OCPP201 void ocpp::v201::DeviceModel::check_integrity(const std::map<int, int>&) :: Integrity check in Device Model storage failed:Number of EVSE configured in device model is incompatible with number of configured EVSEs of the ChargePoint: 2: 1
terminate called after throwing an instance of 'ocpp::v201::DeviceModelStorageError'
what(): Number of EVSE configured in device model is incompatible with number of configured EVSEs of the ChargePoint: 2: 1
2024-08-16 06:18:49.455190 [INFO] evse_manager_1: :: Ignoring BSP Event, BSP is not enabled yet.Has 2 EVSE
shankari
commented
3 months ago @faizanmir21 check the prior comments here and in related demo issues. you need to edit the device_model.db or copy over one from the repo. Again, everything that I did to get things to work is written down.
You also need to be careful about relative versus absolute paths
mnt/user_data/opt/everest/bin/manager is a relative path
/mnt/user_data/opt/everest/etc/everest/config-sil-ocpp201-pnc.yaml is an absolute path
mixing and matching them is a recipe for disaster.
faizanmir21
commented
3 months ago Got it ! Removed the second evse from the database:
sqlite3 mnt/user_data/opt/everest/share/everest/modules/OCPP201/device_model_storage.db
Now I am seeing this, it says ERROR but it looks like a log indicating the max AC current was set to 32amps, correct me if I am wrong here.
2024-08-19 20:07:17.114787 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: In ISO15118 charger impl, after updating AC max current to 0.000000, we get 0.000000
2024-08-19 20:07:17.117362 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: In ISO15118 charger impl, after updating AC max current to 32.000000, we get 32.000000Another error I am seeing is, it is try to connect to a local OCPP server.
2024-08-19 20:07:20.448225 [ERRO] ocpp:OCPP201 int ocpp::WebsocketTlsTPM::process_callback(void*, int, void*, void*, size_t) :: CLIENT_CONNECTION_ERROR: conn fail: 113
2024-08-19 20:07:20.448566 [ERRO] ocpp:OCPP201 void ocpp::WebsocketTlsTPM::on_conn_fail() :: OCPP client connection to server failed
2024-08-19 20:07:20.448977 [INFO] ocpp:OCPP201 :: Reconnecting in: 3000ms, attempt: 1I see you used this to reset the host with CSMS server at "ocppCsmsUrl": "ws://169.254.223.121/ws/cp001"
sqlite> update variable_attribute set "value"='[{"configurationSlot": 1, "connectionData": {"messageTimeout": 30, "ocppCsmsUrl": "ws://169.254.223.121/ws/cp001", "ocppInterface": "Wired0", "ocppTransport": "JSON", "ocppVersion": "OCPP20", "securityProfile": 1}}]' where id=161I am seeing this to setup an CSMS server on GitHub repo.
π¨ Basic and ISO 15118-2 AC Charging with OCPP 2.0.1 CSMS (MaEVe Security Profile 1) β‘: curl https://raw.githubusercontent.com/everest/everest-demo/main/demo-iso15118-2-ac-plus-ocpp.sh | bash -s -- -1
π¨ Basic and ISO 15118-2 AC Charging with OCPP 2.0.1 CSMS (MaEVe Security Profile 2) β‘: curl https://raw.githubusercontent.com/everest/everest-demo/main/demo-iso15118-2-ac-plus-ocpp.sh | bash -s -- -2
π¨ Basic and ISO 15118-2 AC Charging with OCPP 2.0.1 CSMS (MaEVe Security Profile 3) β‘: curl https://raw.githubusercontent.com/everest/everest-demo/main/demo-iso15118-2-ac-plus-ocpp.sh | bash -s -- -3Setup CSMS from the charin repo using:
bash demo-iso15118-2-ac-plus-ocpp.sh -r $(pwd) -3
Updated the device_model_storage.db and changed the CsmsUrl to 192.168.65.1/ws/cp001
sqlite> update variable_attribute set "value"='[{"configurationSlot": 1, "connectionData": {"messageTimeout": 30, "ocppCsmsUrl": "ws://169.254.223.121/ws/cp001", "ocppInterface": "Wired0", "ocppTransport": "JSON", "ocppVersion": "OCPP20", "securityProfile": 1}}]' where id=161 Getting this output
2024-08-21 07:25:46.017771 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: In ISO15118 charger impl, after updating AC max current to 32.000000, we get 32.000000
2024-08-21 07:25:46.711014 [INFO] ocpp:OCPP201 :: Using network iface:
2024-08-21 07:25:46.712208 [INFO] ocpp:OCPP201 :: LWS connect with info port: [80] address: [192.168.65.1] path: [/ws/cp001] protocol: [ocpp2.0.1]
2024-08-21 07:25:47.122021 [INFO] energy_manager: :: NO TRADE
2024-08-21 07:25:47.123399 [INFO] energy_manager: :: Sending enfored limits (import) to :evse_manager_1 {
"ac_max_current_A": 32.0
}
2024-08-21 07:25:47.125302 [INFO] energy_manager: :: returning optimized values vector of length 1
2024-08-21 07:25:47.126106 [INFO] energy_manager: :: evse_manager_1 Enforce limits 32A -9999W Seems to be connected to the CSMS server. Error output showing the max AC current was set to 32A and we are getting 32A.
shankari
commented
3 months ago @faizanmir21 the logs are marked as ERROR but only so that I could see the logs without changing the log level. Those logs should actually be at the DEBUG/INFO level.
I am pretty sure I put that into the issue tracking the software changes: https://github.com/EVerest/everest-demo/issues/44 but I am not 100% sure that I did
faizanmir21
commented
3 months ago Updated the config to enable uMWC BSP
bsp:
- module_id: yeti_driver_1
implementation_id: board_supportDefined yeti_driver_1 as
yeti_driver:
module: MicroMegaWattBSP
config_module:
baud_rate: 115200
reset_gpio: 27
serial_port: /dev/serial0configured slac to also be real and not a simulator and removed the car simulator
slac:
module: EvseSlac
config_implementation:
main:
device: eth1Loading the updated config file
2024-08-29 08:07:06.896778 [INFO] ocpp:OCPP201 :: Module ocpp initialized [3836ms]
2024-08-29 08:07:07.038824 [INFO] evse_manager_1: :: Module evse_manager_1 initialized [4178ms]
2024-08-29 08:07:08.408035 [INFO] manager :: πππ All modules are initialized. EVerest up and running [6759ms] πππ
2024-08-29 08:07:08.418580 [WARN] ocpp:OCPP201 void module::OCPP201::ready() :: Launching timer for calling set_external_limits function
sh: 1: echo: echo: I/O error
terminate called after throwing an instance of 'std::out_of_range'
what(): No known string conversion for provided enum of type Connector_type
2024-08-29 08:07:08.436836 [INFO] energy_manager: :: returning optimized values vector of length 0
2024-08-29 08:07:08.453225 [INFO] slac:EvseSlac :: Starting the SLAC state machine
2024-08-29 08:07:08.454844 [INFO] slac:EvseSlac :: Qualcomm PLC Device Attributes:
HW Platform: QCA7000
SW Platform: MAC
Firmware:
Build date: 1.2.5-0
ZC signal: Missing
Line frequency: 50Hz@faizanmir21 it is not clear what is happening here. Is the service crashing or has it started up successfully?
faizanmir21
commented
3 months ago No, it crashes as I load the config.
2024-09-04 20:24:08.069931 [INFO] manager :: ________ __ _
2024-09-04 20:24:08.070850 [INFO] manager :: | ____\ \ / / | |
2024-09-04 20:24:08.071034 [INFO] manager :: | |__ \ \ / /__ _ __ ___ ___| |_
2024-09-04 20:24:08.071173 [INFO] manager :: | __| \ \/ / _ \ '__/ _ \/ __| __|
2024-09-04 20:24:08.071308 [INFO] manager :: | |____ \ / __/ | | __/\__ \ |_
2024-09-04 20:24:08.071436 [INFO] manager :: |______| \/ \___|_| \___||___/\__|
2024-09-04 20:24:08.071563 [INFO] manager ::
2024-09-04 20:24:08.071681 [INFO] manager :: Using MQTT broker localhost:1883
2024-09-04 20:24:08.100978 [INFO] everest_ctrl :: Launching controller service on port 8849
2024-09-04 20:24:08.165953 [INFO] manager :: Loading config file at: /mnt/user_data/opt/everest/etc/everest/user-config/config-sil-ocpp201-pnc.yaml
2024-09-04 20:24:09.096893 [INFO] manager :: Config loading completed in 1019ms
2024-09-04 20:24:12.905283 [INFO] system:System :: Module system initialized [3555ms]
2024-09-04 20:24:12.921525 [INFO] energy_manager: :: Module energy_manager initialized [3733ms]
2024-09-04 20:24:12.985645 [INFO] grid_connection :: Module grid_connection_point initialized [3684ms]
2024-09-04 20:24:13.075770 [INFO] slac:EvseSlac :: Module slac initialized [3721ms]
2024-09-04 20:24:13.103779 [INFO] api:API :: Module api initialized [3932ms]
2024-09-04 20:24:13.114732 [INFO] token_provider_ :: Module token_provider_1 initialized [3746ms]
2024-09-04 20:24:13.125286 [INFO] auth:Auth :: Module auth initialized [3957ms]
2024-09-04 20:24:13.197236 [INFO] ocpp:OCPP201 :: Module ocpp initialized [3695ms]
2024-09-04 20:24:13.215439 [INFO] evse_security:E :: Module evse_security initialized [3874ms]
2024-09-04 20:24:13.232503 [INFO] yeti_driver_1:M :: Module yeti_driver_1 initialized [3792ms]
2024-09-04 20:24:13.252601 [INFO] iso15118_charge :: TCP server on wlan0 is listening on port [fe80::f657:d2a1:20d:b660%3]:61341
2024-09-04 20:24:13.253187 [INFO] iso15118_charge :: TLS server on wlan0 is listening on port [fe80::f657:d2a1:20d:b660%3]:64109
2024-09-04 20:24:13.253385 [INFO] iso15118_charge :: SDP socket setup succeeded
2024-09-04 20:24:13.254546 [INFO] iso15118_charge :: Module iso15118_charger initialized [3910ms]
2024-09-04 20:24:13.471189 [INFO] evse_manager_1: :: Module evse_manager_1 initialized [4200ms]
2024-09-04 20:24:14.913976 [INFO] manager :: πππ All modules are initialized. EVerest up and running [6868ms] πππ
2024-09-04 20:24:14.918959 [WARN] ocpp:OCPP201 void module::OCPP201::ready() :: Launching timer for calling set_external_limits function
sh: 1: echo: echo: I/O error
terminate called after throwing an instance of 'std::out_of_range'
what(): No known string conversion for provided enum of type Connector_type
2024-09-04 20:24:14.950648 [INFO] energy_manager: :: returning optimized values vector of length 0
2024-09-04 20:24:14.990884 [INFO] slac:EvseSlac :: Starting the SLAC state machine
2024-09-04 20:24:14.992738 [INFO] slac:EvseSlac :: Qualcomm PLC Device Attributes:
HW Platform: QCA7000
SW Platform: MAC
Firmware:
Build date: 1.2.5-0
ZC signal: Missing
Line frequency: 50Hz
2024-09-04 20:24:14.995363 [INFO] ocpp:OCPP201 :: Established connection to database: "/mnt/user_data/opt/everest/share/everest/modules/OCPP201/device_model_storage.db"
2024-09-04 20:24:14.995811 [INFO] ocpp:OCPP201 :: Established connection to device model database successfully: "/mnt/user_data/opt/everest/share/everest/modules/OCPP201/device_model_storage.db"
2024-09-04 20:24:15.057240 [INFO] ocpp:OCPP201 :: Successfully retrieved Device Model from DeviceModelStorage
2024-09-04 20:24:15.058554 [INFO] ocpp:OCPP201 :: Established connection to database: "/tmp/ocpp201/cp.db"
2024-09-04 20:24:15.059189 [INFO] ocpp:OCPP201 :: Target version: 2, current version: 2
2024-09-04 20:24:15.059504 [INFO] ocpp:OCPP201 :: No migrations to apply since versions match
2024-09-04 20:24:15.059804 [INFO] ocpp:OCPP201 :: Successfully closed database: "/tmp/ocpp201/cp.db"
2024-09-04 20:24:15.060318 [INFO] ocpp:OCPP201 :: Established connection to database: "/tmp/ocpp201/cp.db"
2024-09-04 20:24:15.066478 [INFO] ocpp:OCPP201 :: Logging OCPP messages to log file: /tmp/everest_ocpp_logs/2024-09-04T19:24:15.066Z.log
2024-09-04 20:24:15.066979 [INFO] ocpp:OCPP201 :: Logging OCPP messages to html file: /tmp/everest_ocpp_logs/2024-09-04T19:24:15.066Z.html
2024-09-04 20:24:15.067368 [INFO] ocpp:OCPP201 :: Logging SecurityEvents to file
2024-09-04 20:24:15.093313 [INFO] slac:EvseSlac :: Entered Reset state
2024-09-04 20:24:15.093701 [INFO] slac:EvseSlac :: New NMK key: 39:41:5A:35:47:4E:31:43:48:54:4E:37:44:4A:31:4F
2024-09-04 20:24:15.094409 [INFO] slac:EvseSlac :: Received CM_SET_KEY_CNF
2024-09-04 20:24:15.094907 [INFO] slac:EvseSlac :: Entered Idle state
2024-09-04 20:24:15.182387 [INFO] ocpp:OCPP201 :: TxStartPoints from device model: PowerPathClosed
2024-09-04 20:24:15.183356 [INFO] ocpp:OCPP201 :: TxStopPoints from device model: EVConnected,Authorized
2024-09-04 20:24:15.810714 [CRIT] manager int boot(const boost::program_options::variables_map&) :: Module yeti_driver_1 (pid: 2609) exited with status: 134. Terminating all modules.
2024-09-04 20:24:15.812087 [INFO] manager :: SIGTERM of child: api (pid: 2597) succeeded.
2024-09-04 20:24:15.812280 [INFO] manager :: SIGTERM of child: auth (pid: 2598) succeeded.
2024-09-04 20:24:15.812390 [INFO] manager :: SIGTERM of child: energy_manager (pid: 2599) succeeded.
2024-09-04 20:24:15.812609 [INFO] manager :: SIGTERM of child: evse_manager_1 (pid: 2600) succeeded.
2024-09-04 20:24:15.812743 [INFO] manager :: SIGTERM of child: evse_security (pid: 2601) succeeded.
2024-09-04 20:24:15.812856 [INFO] manager :: SIGTERM of child: grid_connection_point (pid: 2602) succeeded.
2024-09-04 20:24:15.812961 [INFO] manager :: SIGTERM of child: iso15118_car (pid: 2603) succeeded.
2024-09-04 20:24:15.813063 [INFO] manager :: SIGTERM of child: iso15118_charger (pid: 2604) succeeded.
2024-09-04 20:24:15.813198 [INFO] manager :: SIGTERM of child: ocpp (pid: 2605) succeeded.
2024-09-04 20:24:15.813300 [INFO] manager :: SIGTERM of child: slac (pid: 2606) succeeded.
2024-09-04 20:24:15.821406 [INFO] manager :: SIGTERM of child: system (pid: 2607) succeeded.
2024-09-04 20:24:15.821779 [INFO] manager :: SIGTERM of child: token_provider_1 (pid: 2608) succeeded.
2024-09-04 20:24:15.822012 [CRIT] manager int boot(const boost::program_options::variables_map&) :: Exiting manager.
seems to be calling calling set_external_limitsfunction
faizanmir21
commented
3 months ago Also noticed this message on CSMS
2024-09-04 19:27:49.486710 [ERRO] ocpp:OCPP201 void ocpp::v201::ChargePoint::handle_message(const ocpp::EnhancedMessage<ocpp::v201::MessageType>&) :: json_message called: [3,"7be3e28a-7247-4e47-9fca-7c4faa43c486",{"currentTime":"2024-09-04T19:27:49Z"}]
The timestamp seems to be off between the two
faizanmir21
commented
2 months ago The yaml file for Evse Manager module does not call set_external_limits function. However the the yaml file in interface dir has a set_external_limits function.
Located at: mnt/user_data/opt/everest/share/everest/interfaces/evse_manager.yaml
set_external_limits:
description: Set additional external energy flow limits at this node.
arguments:
value:
description: UUID of node that this limit applies to
type: object
$ref: /energy#/ExternalLimitsIn the config file, the grid_connection_point uses EnergyNode module as:
grid_connection_point:
module: EnergyNode
config_module:
fuse_limit_A: 40.0
phase_count: 3In EnergyNode module located at mnt/user_data/opt/everest/libexec/everest/modules/EnergyNode/manifest.yaml
It provides external_energy_limits as output:
provides:
energy_grid:
description: This is the chain interface to build the energy supply tree
interface: energy
external_limits:
description: Additional external limits can be set via this interface.
interface: external_energy_limitsand there is a external_energy_limits.yaml file located at mnt/user_data/opt/everest/share/everest/interfaces/external_energy_limits.yaml which contains set_external_limits function.
description: >-
This interface allows to limit energy flow at a specific node of the energy tree from the outside (e.g. from ocpp).
cmds:
set_external_limits:
description: Set additional external energy flow limits at this node.
arguments:
value:
description: External limits object
type: object
$ref: /energy#/ExternalLimits
vars:
enforced_limits:
description: Enforced limits for this node (coming from the EnergyManager)
type: object
$ref: /energy#/EnforcedLimitsthis says it allows to limit energy flow at a node from outside (e.g ocpp)
Then I looked at the OCPP module at mnt/user_data/opt/everest/libexec/everest/modules/OCPP/manifest.yaml which is for OCPP 1.6 and found a required connector_zero_sink using external_energy_limits interface , shown below
requires:
evse_manager:
interface: evse_manager
min_connections: 1
max_connections: 128
connector_zero_sink:
interface: external_energy_limits
min_connections: 0
max_connections: 1However, this variable is not defined in OCPP201 module which we are using in our config.
faizanmir21
commented
2 months ago Removed OCPP module and getting this error on Connector_types
2024-09-17 01:56:16.177368 [INFO] evse_manager_1: :: Module evse_manager_1 initialized [3788ms]
2024-09-17 01:56:17.634769 [INFO] manager :: πππ All modules are initialized. EVerest up and running [6372ms] πππ
sh: 1: echo: echo: I/O error
terminate called after throwing an instance of 'std::out_of_range'
what(): No known string conversion for provided enum of type Connector_type
2024-09-17 01:56:17.662212 [INFO] energy_manager: :: returning optimized values vector of length 0Connector_types seems to be removed from modules/EvseManager/manifest.yaml as shown here and is defined in everest-core/types/evse_board_support.yaml here
connector_type:string <required>
Type of charging connector available at this EVSE
enum:
- IEC62196Type2Cable
- IEC62196Type2SocketThe commit says Yeti and umwc driver will need MCU firmware updates that will come soon
Do we need to update the firmware ?
Everest documentation says to stop everest before updating firmware
systemctl stop basecamp
yeti_fwupdate /dev/serial0 /usr/share/everest/modules/YetiDriver/firmware/yetiR1_2.1_firmware.bin@faizanmir21 I updated the firmware see comments around https://github.com/EVerest/everest-demo/issues/51#issuecomment-2167266067
I was able to start everest after the firmware update. If the firmware update didn't "stick", you should be able to run the old EVerest version.
Regardless of the firmware version, you should be able to run some version of EVerest
corneliusclaussen
commented
2 months ago With the new yocto image for umwc, this works the same way as for the belaybox described here under "Cross compile":
https://everest.github.io/nightly/hardware/pionix_belay_box.html#belaybox-use-cases
New everest versions will not work on the deprecated debian image.
Abby-Wheelis
commented
2 months ago Starting to work on this today, so far still seem to be stuck here: No known string conversion for provided enum of type Connector_type which is thrown by evse_board_support. Looking back through the discussion, I attempted to run the "old" everest with
The yeti_driver is exiting over an error, causing all modules to terminate
2024-10-02 22:42:24.101848 [ERRO] yeti_driver:Mic void module::MicroMegaWattBSP::ready() :: uMWC reset not successful.
terminate called after throwing an instance of 'boost::wrapexcept<Everest::EverestInternalError>'
what(): uMWC reset not successful.
2024-10-02 22:42:25.035466 [CRIT] manager int boot(const boost::program_options::variables_map&) :: Module yeti_driver (pid: 5646) exited with status: 134. Terminating all modules.Seeing if any of the config files will work:
Ok, clearly need to investigate what could be causing this error - it is the same @shankari encountered after she "Hardcoded and rebuilt" to recover from the connector type error.
That occurance was with the "new" everest, however, and this is the original version ...
It looks like updating the firmware ./umwc_fwupdate /dev/serial0 ../share/everest/modules/YetiDriver/firmware/yetiR1_2.1_firmware.bin might have solved the connector type error, but might have caused this secondary error, so I'm not sure that's the solution.
The other solution to the connector type error seems to have been "hardcode and rebuild" but I'm not sure I understand the codebase well enough yet to know exactly what to change. Maybe I need to start with running this on my computer, then might know enough to get it working on the hardware.
shankari
commented
2 months ago @Abby-Wheelis the firmware has been updated, so I would not expect the older version of EVerest to work. To get the older version to work, you would need to downgrade the firmware, and you would need to get the instructions for that from Pionix. I upgraded the firmware on a two hour long call with a Pionix engineer in which he had to get answers from other people in his team. Neither of us wrote down the steps.
From my notes, I believe that the uMWC reset error is due to the firmware incompatibility.
The connector_type issue is a C++ code issue.
Given that we are going to move to the Yocto version anyway if we can't get this to work, I would suggest hardcoding the value (caps.connector_type = types::evse_board_support::Connector_type::IEC62196Type2Cable;) found in modules/YetiDriver/board_support/evse_board_supportImpl.cpp into modules/MicroMegaWattBSP/board_support/evse_board_supportImpl.cpp
Note that you can should also be able to read the firmware version from the serial port
https://github.com/EVerest/everest-demo/issues/51#issuecomment-2167816099
https://github.com/EVerest/everest-demo/issues/51#issuecomment-2167831379
and verify that it has been updated (aka is not 2fcb731e)
Abby-Wheelis
commented
2 months ago Given that we are going to move to the Yocto version anyway if we can't get this to work, I would suggest hardcoding the value (caps.connector_type = types::evse_board_support::Connector_type::IEC62196Type2Cable;) found in modules/YetiDriver/board_support/evse_board_supportImpl.cpp into modules/MicroMegaWattBSP/board_support/evse_board_supportImpl.cpp
I see these files in the everest-core repo, but I have not found them in the directories on the hardware ... should I be editing and rebuilding on my laptop and then push it back over to the device?
Note that you can should also be able to read the firmware version from the serial port
I am not sure how to interpret this, but it is different than what you saw
7AC?7A??@?? ????(21.0=BE
??7A+?7A??@@???Σ?(21.0=BE@HP?????
??7Aw?7A??@??T?3???(21.0=BEHP?(9??
+?7A?7A??`??????(21.0=BE ?@HP1???
z?7Aw?7A??@??GpI???(21.0=BEP?N@?
??7A??7A??@?????(21.0=BE ?@HP????
w?7A??7A??`????Κ?(21.0=BE@HP?0???
_?7A??7A??@??X???(21.0=BE?@HP?OΘ?
??7A?7A??@@Ξ????(21.0=BE ?@HPVΞ1?I am not sure how to interpret this, but it is different than what you saw
Yup! So it is running the new firmware. I think I still have a copy of the new firmware sitting around somewhere if you want to check the version, but I don't think that is interesting now.
I see these files in the everest-core repo, but I have not found them in the directories on the hardware ... should I be editing and rebuilding on my laptop and then push it back over to the device?
Yes! You should spin up the cross-compile (https://en.wikipedia.org/wiki/Cross_compiler) image (https://github.com/US-JOET/everest-demo/pkgs/container/everest-demo%2Fmanager/255301468?tag=cross-compile-charin-e2e-demo), edit, rebuild and then copy over as in https://github.com/EVerest/everest-demo/issues/51#issuecomment-2167176765
Your "development host" is the docker container with the cross-compile chain set up. Your "embedded system" is the uMWC
Abby-Wheelis
commented
2 months ago Progress so far:
docker pull ghcr.io/us-joet/everest-demo/manager:cross-compile-charin-e2e-demodocker run -it ghcr.io/us-joet/everest-demo/manager:cross-compile-charin-e2e-demo/ext/source/modules/MicroMegaWattBSP/board_support/evse_board_supportImpl.cpp (right place?)sudo /mnt/user_data/oct3/everest/bin/manager --conf /mnt/user_data/opt/everest/etc/everest/config-sil-ocpp201-pnc.yamlGot error:
2024-10-03 18:53:40.210362 [ERRO] manager void Everest::Config::load_and_validate_manifest(const string&, const json&) :: Config entry 'connector_id' for module config not defined in manifest of module '"MicroMegaWattBSP"'!Initial guess is config incompatibility ... let's see what I can work out ...
connector_id line from the configsudo /mnt/user_data/oct3/everest/bin/manager --conf /mnt/user_data/oct3/everest/etc/everest/config-sil-ocpp201-pnc.yaml2024-10-03 19:08:58.399592 [ERRO] manager void Everest::Config::load_and_validate_manifest(const string&, const json&) :: Missing mandatory config entry 'connector_id' for module config in module "EvseManager"So now I'm thinking it has something to do with the way the config is expected to be formatted vs what is actually needed ...
Since the connector_id is both a "mandatory config entry 'connector_id' for module config in module "EvseManager"" and "not defined in manifest of module '"EvseManager"'!"
My next idea to try - can I either make it not mandatory or define it in the manifest?
shankari
commented
2 months ago @Abby-Wheelis note that the original error was not defined in manifest of module '"MicroMegaWattBSP"'!
but the second one is Missing mandatory config entry 'connector_id' for module config in module "EvseManager"
They are two separate modules with different expected configurations (as you may remember from the high level overview).
Abby-Wheelis
commented
2 months ago They are two separate modules
Ah, ok, I saw that it was the same parameter and overlooked the part where it was two different modules. I am out of time to work on this today, but will investigate more tomorrow, maybe it is just an issue with the config
Abby-Wheelis
commented
2 months ago Made some required updates to the config today with help from @faizanmir21, but still getting an error about the Connector_type:
sudo /mnt/user_data/oct3/everest/bin/manager --conf /mnt/user_data/oct3/everest/etc/everest/user-config/config-sil-ocpp201-pnc-new.yaml terminate called after throwing an instance of 'std::out_of_range'
what(): No known string conversion for provided enum of type Connector_typeI'm sure I changed the line, so either 1) I built wrong, or 2) there is some other error. I wish I could figure out where it is trying to convert to a string, but there is no stacktrace.
shankari
commented
2 months ago I'm sure I changed the line, so either 1) I built wrong, or 2) there is some other error. I wish I could figure out where it is trying to convert to a string, but there is no stacktrace.
Can you rebuild with a new log statement to see whether the newly built version is what is being launched? You can also search for where the config is being parsed in the module and add additional logs there
EDIT: Also, what were the changes that you made? I remember @faizanmir21 running into this error earlier https://github.com/EVerest/everest-demo/issues/51#issuecomment-2316886486 but it doesn't seem like we resolved it?
Here's a PR which changed some similar messages; maybe that is where we can find this error as well? https://github.com/EVerest/everest-framework/pull/194
Abby-Wheelis
commented
2 months ago Can you rebuild with a new log statement to see whether the newly built version is what is being launched? You can also search for where the config is being parsed in the module and add additional logs there
I had tried adding some std::cout messages, but did not see any of them come across (did see a build error when i originally introduced a syntax error, but resolved that) - not sure if I messed something up or if things just crashed before the messages were hit. I'm out of time again, so I'll have to try again on Monday.
shankari
commented
1 month ago Attempting to test the compiled version with SIL before copying out the cross-compiled version, I created a new docker-compose that launches the cross-compile container. We need to do this through docker-compose, since otherwise, the MQTT server will not be running for the manager to connect to.
we get the gcda warnings even when running successfully, so they are not an issue, but removing them anyway (find . -name \*.gcda | xargs rm) to avoid freaking out about them.
Is this because we are trying to run the cross-compiled version? No. There is no cross-compiled version yet.
/ext/source # find . -name \*manager
./build/_deps/everest-framework-build/src/manager
./build/generated/include/generated/interfaces/evse_manager
./build/generated/include/generated/interfaces/energy_manager
./build/dist/bin/manager
./build-cross/_deps/everest-framework-build/src/manager
./build-cross/generated/include/generated/interfaces/evse_manager
./build-cross/generated/include/generated/interfaces/energy_managerLet's delete build-cross and build and try to rebuild
shankari
commented
1 month ago Edited build/dist/etc/everest/default_logging.cfg and set the filter to Filter="%Severity% >= INFO or (%Process% contains energy_manager and %Severity% >= DEBG) "
This is either related to the MQTT subscription or to the returning optimized values vector of length 0. Let's find where they are and add additional logs...
The energy_manager module does not depend on anything else. Removing it from the config (by editing /ext/source/config/config-sil.yaml) results in not crashing, but multiple logs of 2024-10-08 19:27:53.014169 [WARN] connector_1:Evs bool module::Charger::power_available() :: Power budget expired, falling back to 0.
Abby-Wheelis
commented
1 month ago I didn't work on this much yesterday, but I'm going to attempt to retrace the above steps today and see if I can get SIL working:
Lots of libgcov Merge mismatch errors, but those seem safe to ignore, and erroring out the same way as this comment
So now I need to see if I can actually put in log statements and rebuild successfully
Abby-Wheelis
commented
1 month ago With the same docker compose file that @shankari was using, I now have things running, and I was able to add an arbitrary log statement and see it appear in the output.
Made the same modification to the config that was added to the comment above, and saw the same logs - I think I have the workflow working!
Curiously, can I get this to not crash on the hardware?
Copied everything over, including a the config file where I made the changes described in this comment
Crashing:
2024-10-10 17:27:53.451747 [ERRO] manager void Everest::Config::resolve_all_requirements() :: Requirement 'slac' of module connector_1:EvseManager not fulfilled: required module slac:EvseSlac does not provide an implementation for 'evse'! Clearly need to update the config: the implementation_id for slac in the connections of connector_1 needed to be updated from evse to main to match the changes I made to slac
Config is still bad:
2024-10-10 17:35:02.753718 [ERRO] manager void Everest::Config::resolve_all_requirements() :: Requirement 'slac' of module ev_manager:JsEvManager not fulfilled: required module slac:EvseSlac does not provide an implementation for 'ev'! Ironing out some more issues with the config with slac... I ended up commenting out the ev_manager since it was a simulator ...
Now getting a new error:
The error is about the "PersistentStore database" - I remember seeing comments about the database in this thread, lets see where I can get referring back to those.
Config in current state: config-sil-no-crash.txt
shankari
commented
1 month ago @Abby-Wheelis what code changes (if any) did you have to make to get the SIL to work? We should keep track of that.
Abby-Wheelis
commented
1 month ago @shankari I did not do anything differently than you did, just ignored the warnings after modifying the config the way you described.
The only change I made before cross building and copying over to the hardware was to add the hardcoding as described before:
I would suggest hardcoding the value (caps.connector_type = types::evse_board_support::Connector_type::IEC62196Type2Cable;) found in modules/YetiDriver/board_support/evse_board_supportImpl.cpp into modules/MicroMegaWattBSP/board_support/evse_board_supportImpl.cpp
I commented out the parts of the config that referred to the PersistentStore, just to see what would happen, and am now back to the same string conversion error:
terminate called after throwing an instance of 'std::out_of_range'
what(): No known string conversion for provided enum of type Connector_typeI just tried to add more log statements around where the new hardcoded assignment is, and in the initialization of those modules, but I'm still not seeing any of those log statements. Could I be doing something wrong in building? I have been following steps 6(c-e)&7 in this comment from Louis and the steps to create a release bundle and move it over in this comment from Shankari.
I'm concerned because I added logs to where "yeti driver" is initialized, but I'm not seeing any of the new log statements show up, I was hoping to narrow down where it tries to convert the Connector_type to a string, but no luck so far.
the-bay-kay
commented
1 month ago Let's see if we can't figure out this log discrepancy! (E.g.: changes are showing up within the SIL, but disappear after cross compilation. So, let's see if the issue is within the compilation, or the uploading / running on hardware).
I've successfully cross compiled using Abby et al.'s notes here, adding a log to /ext/source/modules/YetiDriver/YetiDriver.cpp before compilation. To see if this log shows up, we want to use objdump to inspect the compiled module. Unfortunately, running...
/bullseye-toolchain/crosstool/bin/armv8-rpi4-linux-gnueabihf-objdump/ -S /ext/source/build/modules/YetiDriver/YetiDriver... results in a "file format not recognized" error. Inspecting the executable with file or readelf, we find that the YetiDriver file is an ELF 64-bit x86-64 LSB pie executable. This explains why we could inspect the final manager file (ELF 32-bit LSB executable, ARM), but not the intermediate elf. I was unable to find a copy of objdump within the toolchain that is meant for x86_64 (see logs below cut), so let's go ahead and add one to the toolchain...
shankari
commented
1 month ago @the-bay-kay the modules in build-cross should be in arm format. The build directory is the original compiled version that we use in the SIL. build-cross contains the cross-compiled version that is copied over to the hardware.
the-bay-kay
commented
1 month ago ... the modules in
build-crossshould be in arm format. Thebuilddirectory is the original compiled version that we use in the SIL.
D'oh, right, thank you! That works OK with the objdump in our toolchain. I'm still not seeing the logs, or any line-by-line for that matter -- let me tweak some of the objdump settings... If all else fails I'll add a dummy function and look for its footprint in the assembly.
EDIT 2: Well, adding a function with bad syntax causes the build to fail, which would suggest that we are seeing changes in the cross compilation. Once I confirm this in the ARM code, I'll see what happens when we try to transfer the code over. (I also want to confirm that the changes occur between builds, let's do that before attempting to run on the VM)
the-bay-kay
commented
1 month ago Update: I've been able to successfully see changes made during the compilation process! The steps to reproduce may be found below the cut. Next, I plan to (i) look at the objdump of the full manager ELF, to confirm the logs are still there, and (ii) run this on an ARM VM, to confirm we see the logs.
shankari
commented
1 month ago @the-bay-kay that's great!
API module that @Abby-Wheelis used? Just to make sure that this is not specific to one module...the-bay-kay
commented
1 month ago Do you see the log statements are only the function name?
-g or -ggdb CMake flags -- since the debug info isn't included in the ELF file, we don't get the additional code. Looking at the file footprints...
/ext/source/build-cross/modules/API/API: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, not stripped
# ...
/ext/source/build/modules/API/API: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, with debug_info, not stripped ... the native build includes debug info, but cross compile doesn't. We could probably set this flag somewhere in the source tree / make files, but I haven't dug too deeply.
Yes! I just re-ran the test with /API/, and I'm seeing the function definition of the foo() I added. Checking the manager/ ELF next as a sanity check...
Abby-Wheelis
commented
1 month ago Update: I've been able to successfully see changes made during the compilation process!
I was able to replicate these steps and also see a function that I added in the logs!
the-bay-kay
commented
1 month ago When I go to check the manager...
/bullseye-toolchain/crosstool/bin/armv8-rpi4-linux-gnueabihf-objdump -S /ext/source/build-cross/dist/mnt/user_data/opt/everest/bin/manager > /tmp/temp.log...I'm not seeing the signatures of the functions I added. They still appear within the intermediate phases' ARM code (API & YetiDriver) -- am I checking the wrong file? We should expect to see these signatures within the final manager, no? I deleted and re-compiled to ensure that I'm not looking at an old version...
I suppose there's a non-zero chance that my dummy files got optimized out via gcc (I don't know what flags we use, but IRRC, higher levels of optimization may remove "useless" code during optimization). From what I know about GCC, this shouldn't be an issue since we're defining a whole function and writing to the log stream.
Anyway, I'm glad to hear that you can replicate my build process @Abby-Wheelis ! LMK if those functions show up on the hardware -- I'll spend a bit more time digging into the manager's ASM, then I'll try running on the VM.
Abby-Wheelis
commented
1 month ago LMK if those functions show up on the hardware
Yes, after copying over I can see my function when I objdump /mnt/user_data/oct14/everest/libexec/everest/modules/YetiDriver/YetiDriver
FWIW I can't find it in /mnt/user_data/oct14/everest/bin/manager
It seems like I am updating the code, and the updates are copied over, so I'm going to add LOTS of logs and see if I can get anything to show up on hardware since it seems like the changes are persisting so maybe it's just crashing too early for the logs I tried to add earlier.
Edit: added EVLOG_error calls to several of the modules, none showing on the hardware
the-bay-kay
commented
1 month ago Good to know that we're seeing changes to the individual ELFs after copying! That rules that out, at least.
... added
EVLOG_errorcalls to several of the modules, none showing on the hardware
Since we're not seeing the changes in manager, my hunch is that something hinky is going on during the final linking phase... I'll do some digging!
the-bay-kay
commented
1 month ago Let's trace the workflow to see how we compile manager...
When we run make -j$(nproc) -C build-cross, the Makefile within build-cross builds the manager ELF starting on line 458, using Makefile2 found in build-cross/CMakeFiles.
modules/ around line 1921...
We find the build rules for YetiDriver start all the way down on line 5920! This tells us the CMake files for YetiDriver are in modules/CMakeFiles (Which I should've noticed, in retrospect).
modules/CMakeFiles/YetiDriver.dir/build.make that we use a set of object files linked together (198) to the ELF we've already proven is OK. So, I don't think we have to worry about this sub-routine.The YetiDriver process defined in Makefile2 above seems to be the last mention of the module; likewise, nowhere in this section do I see a linking phase similar to YetiDriver.dir/build.make. I think this is all of the pieces, but I'm missing something -- let's read these closer and see if we can't find anything (If we can't, I may be barking up the wrong build tree...)
Abby-Wheelis
commented
1 month ago Investigate possibility of path being hardcoded:
1) running the manager in opt/ with the config that worked in SIL gives the same charger type error
2) removed opt/ and tried to run the manager in oct14/ (most recent cross-compiled copy) - failed immediately
$ sudo /mnt/user_data/oct14/everest/bin/manager --conf /mnt/user_data/oct14/everest/etc/everest/config-sil-no-crash-1.yaml
sudo: unable to resolve host umwcdbde: Name or service not known
/mnt/user_data/oct14/everest/bin/manager: error while loading shared libraries: libframework.so.0.14.0: cannot open shared object file: No such file or directory3) overwrite opt with contents of oct14
4) run it - does not crash -- seeing logs! (INIT API module is from me/new)
shankari
commented
1 month ago Yay! I knew it had worked for me! Such a weird issue, and so annoying Now it's time to go to the lab and plug it in?!!
Abby-Wheelis
commented
1 month ago Now it's time to go to the lab and plug it in?!!
I think so!
There were a few parts of the config that I commented out, so I want to make sure we're testing with the "right" config, and debug anything else that comes up, but it does run, and we now know that code must be in /mnt/user_data/opt to be called as expected.
shankari
commented
1 month ago I think we are tracking that as a separate issue, so once you have checked over the configs, and confirmed that it consistently runs, we can go ahead and close this one (finally!!)
shankari
commented
1 month ago I will also point out that this only works if we disable the energy manager, which results in a max current of 0. Not sure if this will be a problem.
2024-10-15 23:09:56.883206 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: In ISO15118 charger impl, after updating AC max current to 0.000000, we get 0.000000
2024-10-15 23:09:56.925627 [INFO] connector_1:Evs :: πππ Ready to start charging πππ
2024-10-15 23:09:57.009075 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: In ISO15118 charger impl, after updating AC max current to 0.000000, we get 0.000000
2024-10-15 23:10:06.898035 [WARN] connector_1:Evs bool module::Charger::power_available() :: Power budget expired, falling back to 0.
2024-10-15 23:10:06.998978 [WARN] connector_1:Evs bool module::Charger::power_available() :: Power budget expired, falling back to 0.In the SIL, if I enable the energy manager, it still crashes. @Abby-Wheelis can you confirm that this is true even on hardware?
Will spend a time-bounded effort tonight to see if I can get the energy manager to stop crashing so that we can try to actually send a real current value.
shankari
commented
1 month ago ok so the energy manager thing is weird. It looks like everything crashes ~ 10 seconds after the manager starts
I enabled debug logging in build/dist/etc/everest/default_logging.cfg and added several sleep statements to the main energy manager loop.
new_sleep_statements.patch
If none of the sleep statements are in place, the energy manager crashes after 999 runs no_sleeps.log
$ grep "Run energy optimizer" /tmp/no_sleeps.log | wc -l
999If all the sleep statements are in place, the energy manager crashes after a single run all_sleeps.log
$ grep "Run energy optimizer" /tmp/all_sleeps.log | wc -l
1In both cases, the crash occurred after around 10 seconds
$ head -n1 /tmp/no_sleeps.log
2024-10-16 00:40:36.868416 [INFO] manager :: ________ __ _
$ tail -n1 /tmp/no_sleeps.log
2024-10-16 00:40:42.411856 [CRIT] manager int boot(const boost::program_options::variables_map&) :: Exiting manager.
$ head -n1 /tmp/all_sleeps.log
2024-10-16 00:52:21.105189 [INFO] manager :: ________ __ _
$ tail -n1 /tmp/all_sleeps.log
2024-10-16 00:52:33.012974 [CRIT] manager int boot(const boost::program_options::variables_map&) :: Exiting manager.So I suspect this is related to some kind of threading/timing issue and not a particular issue with the code.
Note also that the optimizer doesn't actually return any values; in the no_sleep case, it consistently returns a vector of length 0. Can we just hack this to set the energy limit once and disable the thread? that would be the most surgical change...
shankari
commented
1 month ago Looking at logs from a prior successful SIL run (non-cross-compiled), I can see that the optimizer returns a vector with a single value
2024-07-17 22:26:45.186727 [INFO] energy_manager: :: NO TRADE
2024-07-17 22:26:45.186918 [INFO] energy_manager: :: Sending enfored limits (import) to :evse_manager_1 {
"ac_max_current_A": 32.0
}
2024-07-17 22:26:45.186986 [INFO] energy_manager: :: returning optimized values vector of length 1
2024-07-17 22:26:45.187077 [INFO] energy_manager: :: evse_manager_1 Enforce limits 32A -9999Wlet's try to return that value and see how it goes.
shankari
commented
1 month ago After the changes, I see
2024-10-16 01:40:09.569465 [INFO] energy_manager: :: about to init globals again
2024-10-16 01:40:09.570124 [INFO] energy_manager: :: about to lock the energy mutex
2024-10-16 01:40:09.570222 [INFO] energy_manager: :: launched the mutex
2024-10-16 01:40:09.570526 [INFO] energy_manager: :: returning optimized values vector of length 1
2024-10-16 01:40:09.570684 [INFO] energy_manager: :: ran optimizer in the ready loop, got values 1
2024-10-16 01:40:09.570825 [INFO] energy_manager: :: my_fake_uuid Enforce limits 32A -9999W And then a crash
Will fix the UUID to evse_manager_1, but the issue is clearly not with the values returned but with the thread. Trying to set the values once and not launching the thread at all...
The EVerest project has open hardware as well https://everest.github.io/nightly/hardware/pionix_belay_box.html which is available as a kit from Pionix. Pionix also sells the uMWC (https://shop.pionix.com/products/umwc-micro-mega-watt-charger). This is is a non-open device in a housing that shares some hardware with the Belay Box although it has a different power module that is limited to 1W output.
In this issue, we will track the steps to run a custom build of EVerest on the uMWC so that we can perform HIL testing.
@faizanmir21 The instructions are here: https://everest.github.io/nightly/hardware/pionix_belay_box.html#developing-with-everest-and-belaybox
They are for the Belay Box, but I'm hoping that they will apply to the uMWC as well. If not, we can ask the community for help.
My suggested steps are:
everest-dev.serviceand verify that it starts the dev build from/mnt/user_data/opt/everestdocker exec -it ....manager /bin/bashORdocker run -it ghcr.io/everest/everest-demo/manager:0.0.16 /bin/bash@drmrd @couryrr-afs @wjmp for visibility