Support implementations that send invalid 0x00 byte at the end of JSON messages

[DerAndereAndi]

The PMCP device sends 0x00 bytes at the end of 2 SHIP messages:

{"accessMethodsRequest":[]}

and

{"accessMethods":[{"id":"Porsche-EVSE-00xxxxx"}]}

See 2nd log file in https://github.com/evcc-io/evcc/issues/15438#issuecomment-2294354359

A workaround for supporting this would be to check and remove any 0x00 byte at the end of messages containing json data. The 0x00 byte has to be present at the SHIP init data. That one should not be touched.

[Neverminda]

I am trying to fix this with a first brute patch. Since I don't know the code at all and can't estimate the effects either, I would want to test the whole thing locally on my end. Is there any documentation on how to test it with cvcc?

addtrimm.patch

[DerAndereAndi]

Thanks for the suggestion, but this is not a patch that I would want to use. There are no tests and it part of a specific part and wouldn't fix this in general if it occurs in other messages.

You can test this with evcc by pointing the SHIP repo in go.mod to your locally patched version of this repo.

[Neverminda]

Sure, this is only a first try, nothing I would put in a pull request. I never did anything with golang. First I need to to find out how I can build evcc locally

[Neverminda]

with the patch applied the PMCP connects to the evcc. But then the next invalid character '\x00' after top-level value occurs. I will start to patch me through, till it works.

[eebus ] TRACE 2024/08/17 13:44:49 Recv: 6c401f7671f64d682ed5a2e048a3efb9de7669db {"accessMethods":[{"id":"Porsche-EVSE-0048524"}]} [eebus ] TRACE 2024/08/17 13:44:49 Recv: 6c401f7671f64d682ed5a2e048a3efb9de7669db {"data":[{"header":[{"protocolId":"ee1.0"}]},{"payload":{"datagram":[{"header":[{"specificationVersion":"1.1.1"},{"addressSource":[{"device":"d:_i:19667_PorscheEVSE_0048524"},{"entity":[0]},{"feature":0}]},{"addressDestination":[{"entity":[0]},{"feature":0}]},{"msgCounter":1},{"cmdClassifier":"read"}]},{"payload":[{"cmd":[[{"nodeManagementDetailedDiscoveryData":[]}]]}]}]}}]} [eebus ] DEBUG 2024/08/17 13:44:49 6c401f7671f64d682ed5a2e048a3efb9de7669db error unmarshalling message: invalid character '\x00' after top-level value [eebus ] TRACE 2024/08/17 13:44:49 Recv: 6c401f7671f64d682ed5a2e048a3efb9de7669db {"data":[{"header":[{"protocolId":"ee1.0"}]},{"payload":{"datagram":[{"header":[{"specificationVersion":"1.1.1"},{"addressSource":[{"device":"d:_i:19667_PorscheEVSE_0048524"},{"entity":[0]},{"feature":0}]},{"addressDestination":[{"device":"d:_n:EVCC_HEMS-EVCC"},{"entity":[0]},{"feature":0}]},{"msgCounter":2},{"msgCounterReference":1},{"cmdClassifier":"reply"}]},{"payload":[{"cmd":[[{"nodeManagementDetailedDiscoveryData":[{"specificationVersionList":[{"specificationVersion":["1.1.1"]}]},{"deviceInformation":[{"description":[{"deviceAddress":[{"device":"d:_i:19667_PorscheEVSE_0048524"}]},{"deviceType":"ChargingStation"},{"networkFeatureSet":"smart"}]}]},{"entityInformation":[[{"description":[{"entityAddress":[{"entity":[0]}]},{"entityType":"DeviceInformation"}]}],[{"description":[{"entityAddress":[{"entity":[1]}]},{"entityType":"EVSE"}]}]]},{"featureInformation":[[{"description":[{"featureAddress":[{"entity":[0]},{"feature":0}]},{"featureType":"NodeManagement"},{"role":"special"},{"supportedFunction":[[{"function":"nodeManagementDetailedDiscoveryData"},{"possibleOperations":[{"read":[]}]}],[{"function":"nodeManagementSubscriptionRequestCall"},{"possibleOperations":[]}],[{"function":"nodeManagementBindingRequestCall"},{"possibleOperations":[]}],[{"function":"nodeManagementUseCaseData"},{"possibleOperations":[{"read":[]}]}]]}]}],[{"description":[{"featureAddress":[{"entity":[1]},{"feature":1}]},{"featureType":"Generic"},{"role":"client"}]}],[{"description":[{"featureAddress":[{"entity":[1]},{"feature":3}]},{"featureType":"DeviceDiagnosis"},{"role":"server"},{"supportedFunction":[[{"function":"deviceDiagnosisStateData"},{"possibleOperations":[{"read":[]}]}]]}]}],[{"description":[{"featureAddress":[{"entity":[1]},{"feature":2}]},{"featureType":"DeviceClassification"},{"role":"server"},{"supportedFunction":[[{"function":"deviceClassificationManufacturerData"},{"possibleOperations":[{"read":[]}]}]]}]}]]}]}]]}]}]}}]} [eebus ] DEBUG 2024/08/17 13:44:49 6c401f7671f64d682ed5a2e048a3efb9de7669db error unmarshalling message: invalid character '\x00' after top-level value

[Neverminda]

With this patch der connection works just fine. trimm.patch

Me suggestion to fix this more general. Adding the bytes.Trim(data, "\x00") to every data before calling json.Unmarshal(....) or create a function like

func Unmarshal(data []byte, v any) error {
    return json.Unmarshal([]byte(bytes.Trim(data, "\x00")), &v)
}

and use this instead of direct calls to json.Unmarshal(...)

[Neverminda]

But the bad news is that this "only" fixes the connection issues with the PMCP.

After adding this patch and connection a car with the PMCP

I got:

[lp-1 ] ERROR 2024/08/17 14:48:00 charge rater: not available [lp-1 ] ERROR 2024/08/17 14:48:00 charger status: timeout

looks like there are some missing values in the json response of the PMCP but this should be tracked in a different bug report, or this is an error in my evcc.yml

[DerAndereAndi]

The previous mentioned issues belong to evcc, not this repo.

[Neverminda]

I currently trying this:

func JsonFromEEBUSJson(json []byte) []byte { var result = bytes.ReplaceAll(json, []byte("[{"), []byte("{")) result = bytes.ReplaceAll(result, []byte("},{"), []byte(",")) result = bytes.ReplaceAll(result, []byte("}]"), []byte("}")) result = bytes.ReplaceAll(result, []byte("[]"), []byte("{}")) result = bytes.Trim(result, "\x00") return result }

[Neverminda]

I tested these changes and the connection worked fine (expect the missing "charger rate")

toEEBusJson.patch

I you are fine with this approach I would add a test and create a PR

[DerAndereAndi]

That would be awesome, thank you very much!

[Neverminda]

Maybe you can help with the missing "charger rate"

it looks like

that in ebbs-go/usecases/usecase/usecase.go:113 the func IsScenarioAvailableAtEntity returns false

[DerAndereAndi]

Please create a new discussion in evcc for that. The problem has nothing to do with this repository. Thanks for your understanding.

[andig]

Wäre es nicht sinnvoller, alle 0x00 aus dem JSON vollständig rauszuwerfen, idealerweise als zubuchbare Option? Falls keine utf8-Sequenzen enthalten sind sollte es in JSON eigentlich keine 0x00 geben.

[Neverminda]

Gute Frage, die kann ich dir aber nicht beantworten, da ist zu wenig Domainwissen zum EEBus Protocol habe

[DerAndereAndi]

Das ist ein Fehler in einem älteren Gerät welches wohl kein Software Update mehr bekommen wird. In neueren Geräten sollte dieser oder ähnliche Fehler nie auftreten und in Tests frühzeitig gefunden werden. Möchte daher ungern solche Fehlerfälle per se "erlauben". Für mich ist das momentane Handling daher gut genug.