Initial J1939 changes

[JohnLussmyer]

Initial set of changes, test failing

see #54

[JohnLussmyer]

I don't think the timestamps are working: name=0000000000000000, PGN=0000FFD4, addr=A1, data=00.E3.1C.A0.28.10.00.03 J1939ReceivedMessageHeader{sourceAddress=J1939Address{device=LinuxNetworkDevice(name='null', index=4), name=0, parameterGroupNumber=65492, address=-95}, bytesReceived=8, timestamp=1970-01-01T00:00:00Z, destinationAddress=-1, destinationName=0, priority=6} name=0000000000000000, PGN=0000FFD4, addr=A1, data=00.E3.1C.A0.28.10.00.03 J1939ReceivedMessageHeader{sourceAddress=J1939Address{device=LinuxNetworkDevice(name='null', index=4), name=0, parameterGroupNumber=65492, address=-95}, bytesReceived=8, timestamp=1970-01-01T00:00:00Z, destinationAddress=-1, destinationName=0, priority=6}

[pschichtel]

ok, I think timestamping needs to be enabled on the socket first, I'll have to investigate that.

[JohnLussmyer]

Just did a quick test of sendMessage() while I'm setting up my Java app to deal with this device. 0xA1 is the address reported when receiving packets. 0x00FEEB is the PGN I want to send.

    try (final J1939CanChannel channel = CanChannels.newJ1939Channel()) {
        J1939Address addr = new J1939Address(CAN_INTERFACE);
        channel.bind(addr);
        channel.setOption(J1939CanSocketOptions.SO_J1939_PROMISC, true);

        J1939Address addr2 = new J1939Address(CAN_INTERFACE, 0L, 0x00FEEB, (byte) 0xA1);
        ByteBuffer sendBuf = JavaCAN.allocateOrdered(32);
        long rc = channel.sendMessage(sendBuf, addr2);
        System.out.println("rc=" + rc);

fails on the sendMessage() with:

john@raspberrypi4:~/PiCanTest $ java -cp PiCanTest_Lib/* -jar PiCanTest.jar tel.schich.javacan.platform.linux.LinuxNativeOperationException: Unable to sendto to the socket - errorNumber=77, errorMessage='File descriptor in bad state' at tel.schich.javacan.SocketCAN.sendJ1939Message(Native Method) at tel.schich.javacan.J1939CanChannelImpl.sendMessage(J1939CanChannelImpl.java:133) at com.casadelgato.cantest.PiCANTest.main(PiCANTest.java:36)

I assume this is because I'm pretty clueless about CanBus. Want me to start asking questions on the forum?

[pschichtel]

There are 3 reasons for BADFD (error 77):

1. the socket has not been bound (not the case here)
2. source name == 0 or source addr == NO_ADDR (this might be the case, but I cannot tell from your snippet)
3. the socket's source address' interface (addr) and the destination address' interface (addr2) are different (this might also be the case, but I also cannot tell from your snippet)

[JohnLussmyer]

Umm, the snippet was the ENTIRE program. Note the name, pgn, addr fields being specifically set for the sendMessage().

It appears that the Source addr (new J1939Address(CAN_INTERFACE);) needs to be more fully specified. I'll try that.

I tried setting the sender addr to a valid addr different from the sendto addr, that worked for the send, but using the same channel to receive msgs stopped working. So I set the sender addr to the same as the sendto addr - that worked, and I could continue using the channel to receive msgs.

Though it appears that my receive code will then receive the packet that I sent, and not receive the response from the J1939 device. (or it didn't send one.) I need to finish setting up a separate listener thread. Hmm, I've been assuming that I could use multiple Channel objects to talk to the same can bus. Is that correct? or are multiple connections to the same can bus not allowed?

Shouldn't NetworkDevice.lookup(String name) return a @NonNull value? It appears that an exception would be thrown if it fails.

[pschichtel]

Hmm, I've been assuming that I could use multiple Channel objects to talk to the same can bus. Is that correct? or are multiple connections to the same can bus not allowed?

multiple channels on the same device are possible, yes.

Shouldn't NetworkDevice.lookup(String name) return a @nonnull value?

I just started annotating things with @NonNull and @Nullable, so there is plenty of places where it's still missing

It would also be great if you update to the latest build and try out the timestamp support (SO_TIMESTAMP, SO_TIMESTAMPNS or SO_TIMESTAMPING options on the channel).

[JohnLussmyer]

I had switched my git to use your repo directly - and it won't let me pull down the J19239 branch.

[pschichtel]

There is no branch, it's all on master

[JohnLussmyer]

Ok, a weird thing happened. When I switched from my copy of your repo, to your repo, my local GitHub desktop was showing my branches, in your repo. I had to shut it all down and restart it for that to work correctly.

Next you restructured the J1939 address support - It appears that when I create a J1939 address, I should use ImmutableJ1939Address now. The J1939AddressBuffer - is only used for the msg header return value, correct?

[pschichtel]

you can use either, they both implement the common J1939Address. if this pattern works here I will probably extend it to raw sockets as well.

[JohnLussmyer]

ok, Timestamp seems to work, but receive is not fully working.

    try (final J1939CanChannel channel = CanChannels.newJ1939Channel()) {
        J1939Address addr = new ImmutableJ1939Address(CAN_INTERFACE, 0L, 0, (byte) 0xA1);
        channel.bind(addr);
        channel.setOption(J1939CanSocketOptions.SO_J1939_PROMISC, true);
        channel.setOption(CanSocketOptions.SO_TIMESTAMP, true);

        while (true) {
            ByteBuffer buffer = JavaCAN.allocateOrdered(1024);
            J1939ReceiveMessageHeaderBuffer hdr = new J1939ReceiveMessageHeaderBuffer();
            buffer.flip();
            channel.receive(buffer, hdr);
            System.out.println(String.format(   "name=%016X, PGN=%08X, addr=%02X, data=%s",
                                                hdr.getSourceAddress().getName(),
                                                hdr.getSourceAddress().getParameterGroupNumber(),
                                                hdr.getSourceAddress().getAddress(),
                                                CanUtils.hexDump(buffer)));
            System.out.println(hdr.getTimestamp().toString());
        }

resulted in:

name=0000000000000000, PGN=0000FFD4, addr=A1, data= 2024-02-19T16:33:49.996097Z name=0000000000000000, PGN=0000FFD4, addr=A1, data= 2024-02-19T16:33:50.996757Z

Note that there was no data received.

[pschichtel]

you aren't flipping the buffer in the right place. you have to flip it after the receive.

[JohnLussmyer]

Yup, that fixed it. Trying to work too fast on multiple things...

[JohnLussmyer]

Sadly, I'm pretty sure that the @NonNull annotation library you are using, isn't compatible with the one built into Eclipse. I've done a little testing, and the Eclipse annotation checking doesn't seem to recognize your annotations. Minor PITA.

[pschichtel]

I assume you can configure eclipse to use these annotations, they are pretty standard. intellij auto detects them and asked me if it should use them instead of the default jetbrains annotations (which I intentionally didn't use).

[JohnLussmyer]

Well, Yes and No. While it is possible to use it from Eclipse, it's run as a separate Ant task, and reports to the ant console.
Not very useful, as it doesn't update the tags in the Eclipse editor. I could just use your source, and do a global replace on the relevant import statements - but that would also be a PITA. For now, I'm not going to worry about it.

[pschichtel]

which annotation does eclipse like to have?

[JohnLussmyer]

org.eclipse.jdt.annotation NonNullByDefault, NonNull, Nullable I tend to have package.info that sets the default for the source package, then just need to use @Nullable in the places needed.

[JohnLussmyer]

Just trying to send my first command, so this is probably a clueless newbie issue.

    J1939Address addr2 = new ImmutableJ1939Address(networkDevice, 0L, EPGN.INVCHG_Setpoint.to29bit(MY_ADDR), bcvAddr);
    log.debug("set12V: addr={}", addr2);
    ByteBuffer sendBuf = JavaCAN.allocateOrdered(8);
    sendBuf.put((byte) 0x10);
    sendBuf.putShort(voltsToBits(voltage));
    sendBuf.putShort(ampsToBits(maxA));
    sendBuf.put((byte) ((enable) ? 1 : 0)); // invCharge mode
    sendBuf.flip();
    log.debug("set12V: sendBuf={}", CanUtils.hexDump(sendBuf));
    long rc = sendChannel.send(sendBuf, addr2);
    log.debug("set12V: rc={}", rc);

My promiscuous listener sees this: (and the above logging lines are included)

16:04:20.237 [ main] DEBUG com.casadelgato.BCV200.BCV200 - set12V: addr=J1939Address{device=LinuxNetworkDevice(name='can1', index=4), name=0, parameterGroupNumber=418316354, address=-95} 16:04:20.241 [ main] DEBUG com.casadelgato.BCV200.BCV200 - set12V: sendBuf=10.F7.00.D0.84.00 16:04:20.448 [ Thread-0] DEBUG com.casadelgato.BCV200.BCV200 - handleMessage: name=0000000000000000, PGN=0000FFD4, addr=A1, data=00.E3.1C.A0.28.10.00.03 16:04:23.589 [ main] DEBUG com.casadelgato.BCV200.BCV200 - set12V: rc=6 16:04:23.608 [ Thread-0] DEBUG com.casadelgato.BCV200.BCV200 - handleMessage: name=0000000000000000, PGN=00000000, addr=42, data=10.F7.00.D0.84.00

So, it looks like the "From" address is correctly my computer, but the PGN received is 0.
Also the data length returned from the send is 2.

Note that the lines I'm receiving from the device look like: 16:04:18.445 [ Thread-0] DEBUG com.casadelgato.BCV200.BCV200 - handleMessage: name=0000000000000000, PGN=0000FFD4, addr=A1, data=00.E3.1C.A0.28.10.00.03

[pschichtel]

but the PGN received is 0.

that's something I noticed in the integration tests as well, feels like this is something that happens sending a message to a device and also reading that same message from the device. Apparently it didn't happen with messages sent by hardware?

[JohnLussmyer]

Not a clue, if it's normal, that fine, just thought it was suspicious.

[JohnLussmyer]

And, of course, this particular device doesn't seem to have a good way to verify the setting.

[pschichtel]

it's definitely suspicious, but I haven't seen anything obviously wrong in either the send or receive paths. Might be a question for the mailing list.

[JohnLussmyer]

I may have to go hook this thing up to a full test setup - which is a royal PITA. 330v battery pack and a 240VAC line.

[pschichtel]

I just switched the codebase over to eclipse annotations. I also used NonNullByDefault, nice suggestion.

[pschichtel]

I also just went through the entire send and receive path, I haven't found anything that might set the pgn to 0, not in my java code, not in my c code and not in the kernel sadly. if anything, then my last commit might have fixed something, but at least on x86 it behaves identically.

[JohnLussmyer]

Annotations, great! I assume that you must be using eclipse as well?

Yeah, the msg I'm sending may be working, but I can't tell unless the unit is connected to a full test setup. Which is out in the unheated shop, 200' from the house....

note on the Eclipse annotations, if you put a "package-info.java" in a package, it can set @NonNullByDefault for every file in the package (and maybe lower packages).

The file would contain similar to this:

@NonNullByDefault package com.casadelgato.BCV200; import org.eclipse.jdt.annotation.NonNullByDefault;

[pschichtel]

I assume that you must be using eclipse as well?

nope, always hated it. I don't really care about what annotations are used as long as they are reasonably well supported by IDEs. intellij supports pretty much all annotation libs.

note on the Eclipse annotations, if you put a "package-info.java" in a package, it can set @NonNullByDefault for every file in the package (and maybe lower packages).

that's exactly what I did

[JohnLussmyer]

Well, I'm getting closer. Receiving is working fine. Sending fails every time. I always get an return code of 8. Any suggestions on where to look? Or should I take it to the forums - and is so, which one?

[JohnLussmyer]

Been digging through my Send code. Creating the channel: J1939Address myAddr = new ImmutableJ1939Address(networkDevice, 0L, 0, MY_ADDR); J1939Address toAddr = new ImmutableJ1939Address(networkDevice, 0L, 0, addr); sendChannel = CanChannels.newJ1939Channel(myAddr, toAddr);

Code to do the send:

    J1939Address addr2 = new ImmutableJ1939Address(networkDevice, 0L, EPGN.DNC_Setpoint.to29bit(bcvAddr, MY_ADDR), MY_ADDR);
    log.debug("set12V: addr={}", addrHex(addr2));
    ByteBuffer sendBuf = JavaCAN.allocateOrdered(8);
    sendBuf.put(EPGN.DNC_Setpoint.getCmd());
    sendBuf.putShort(voltsToBits(voltage));
    sendBuf.putShort(ampsToBits(maxA));
    sendBuf.put((byte) ((enable) ? 1 : 0)); // invCharge mode
    sendBuf.put((byte) 0);
    sendBuf.put((byte) 0);
    sendBuf.flip();
    log.debug("set12V: sendBuf={}", CanUtils.hexDump(sendBuf));

    try {
        long rc = sendChannel.send(sendBuf, addr2);
        log.debug("set12V: rc={}", rc);
    } catch (IOException e) {
        throw new BCV200Exception("set12V failed", e);
    }

Log results:

DEBUG com.casadelgato.BCV200.BCV200 - set12V: addr=from: EE, PGN=18EFA1EE, dev=can1 DEBUG com.casadelgato.BCV200.BCV200 - set12V: sendBuf=10.04.01.C8.7D.01.00.00 DEBUG com.casadelgato.BCV200.BCV200 - set12V: rc=8

As far as I can tell, the PGN is correct. Sending from EE to A1 Docs say: 29 Bit identifier: 0x18EFyyxx (xx – Source address) yy is dest address

This is a sample msg from the device: DEBUG com.casadelgato.BCV200.BCV200 - handleMessage: name=0000000000000000, PGN=0000FFD4, addr=A1, data=00.E3.1C.A0.28.10.00.03

[pschichtel]

The 29 bit CAN address is constructed under the hood by the kernel module, I don't think you can just put a complete 29 bit address into the PGN field (the first line of your snippet). I was actually thinking about providing utilities to construct addresses similar to how I did it for other socket types, but I haven't gotten to it yet.

[JohnLussmyer]

What part of it do I need to supply? Any clues?

[pschichtel]

I think https://www.kernel.org/doc/html/latest/networking/j1939.html?highlight=j1939#j1939-concepts explains it well enough:

[JohnLussmyer]

Yeah, that looks like it matches what I thought. I used my can analyzer to see what was being sent. All 8 of the data bytes are correct, It's the PGN that is wrong. I just did multiple experiments, and found that what I need is: new ImmutableJ1939Address(networkDevice, 0L, 0xEF00, 0xA1);

sends 0x18EFA1EB Which is correct as far as I can tell.

Of course, the Docs that I have for the Device do NOT tell me what the response message means...

[pschichtel]

easy would be boring, right? :D

[pschichtel]

@JohnLussmyer Can you give the J1939Utils I just pushed a go? The functions extract PGN, source address and priority from a CAN ID

[JohnLussmyer]

I find it somewhat odd that I have to provide 0xFE00 so it can replace the 00 with the target address. Minor Note: You might want to have the J1929 address toString() use Hex for the values.

I've pulled and built with them, but not sure what I'd use those specific Utils for. It's creating the PGN to send something that is just weird. When receiving a msg, the .getSourceAddress().getParameterGroupNumber() method gets me the 16 bit PGN ID just fine.

I find it really odd that receiving msgs have a 16 bit value to specify what was sent, but SENDING msgs you only have an 8 bit field, as the other 8 bits are the address you are sending to.

[JohnLussmyer]

Apparently the response to the command I send is a "Standard J1939 ACK" - right. Except that what little docs I can find on the format for an ACK - don't match the data I'm seeing. I send "PGN=0000EF00, addr=EB, data=10.B4.00.00.7D.00.00.00" Ack was "PGN=0000E800, addr=A1, data=00.10.FF.FF.EB.00.EF.00"

Last 2 bytes look like the PGN cmd that was sent, and the "EB" is the address of the sender. No clue what the other fields are.

[pschichtel]

I find it somewhat odd that I have to provide 0xFE00 so it can replace the 00 with the target address.

I don't think that's the case, there is probably a misunderstanding somewhere (not that I could tell you where...).

I've pulled and built with them, but not sure what I'd use those specific Utils for.

the functions extract PGN, addr and priority from a 29bit CAN ID, exactly how the spec says it should be done. Yes, under normal circumstances you can get them directly, but if you mix J1939 and RAW for example you might have CAN IDs that you need to decompose, similar to how you had to manually decompose the address from your analyser.

I find it really odd that receiving msgs have a 16 bit value to specify what was sent, but SENDING msgs you only have an 8 bit field, as the other 8 bits are the address you are sending to.

I don't follow.

gets me the 16 bit PGN ID

the PGN is 18 bits (well 17, if you ignore the one reserved bit), that's why I used int values everywhere.

[JohnLussmyer]

When I'm receiving msgs from my device, the PGNs indicating what I'm receiving are 0xFExx through 0xFFxx, where xx is different for each msg. (though I have no idea what the bit restrictions are, as this device only has about 15 different msgs) When I SEND a msg, I use 0xEFxx - where xx is the address of whom I'm sending the message to.

[pschichtel]

if I understood correctly the PGN is basically the type of message, right? and a whole bunch of them are standardized. Have you checked if any of the messages by your device are standardized? I like the documentation from kvaser on the topic: https://www.kvaser.com/about-can/higher-layer-protocols/j1939-introduction/.

[splatch]

I may have to go hook this thing up to a full test setup - which is a royal PITA. 330v battery pack and a 240VAC line.

@JohnLussmyer Do you happen to have J1939 emulator which could be run under Linux? I don't think I will ever be able to play with protocol, but I am curious how tooling around it look a like.

[pschichtel]

@JohnLussmyer have you had any progress? I'm considering cutting a release soon-ish, that includes J1939.

@splatch from my limited research on j1939 open tooling around it seems even more limited than ISOTP.

[JohnLussmyer]

Well, I know the basics are working. I can receive msgs, and when I send them it's not reporting an error. I'm still building the test setup wiring. (HV safety takes effort!) I'm hoping to be able to fully test things by this weekend. I did ask a question of the company that makes the unit - but they insist on only looking at Can bus logs that are in the .asc or .bti format. (I believe those are from the Canoe toolset.)

[JohnLussmyer]

I'm making progress, and actually have the unit responding to commands - just not in the way expected. I do receive ACK's from the unit for each command sent, though the ACK doesn't contain any useful information. One thing I'm seeing is that EVERY time I .send(), I get an rc of 8. I haven't found the definition of 8 yet.

[pschichtel]

One thing I'm seeing is that EVERY time I .send(), I get an rc of 8.

rc?

[JohnLussmyer]

return code from the send command.

[JohnLussmyer]

ok, now I realize that the send method is returning the buffer size, not a linux return code. Not used to that style. So, the send is working. Now trying to figure out the crap documentation that this unit has. (lists the commands, but NOT any inter-dependencies between them, or what state the unit has to be in, etc...

Commands ARE being sent, and it is replying with ACK's.

[JohnLussmyer]

Oh, and yes it's my fault for not properly reading the javadocs. My only excuse is that my dev environment right now is spread of 3 computers, and I didn't HAVE the javadocs on the one that needed them. As far as I can tell, JavaCAN is working just fine.

[pschichtel]

ok, now I realize that the send method is returning the buffer size, not a linux return code. Not used to that style.

I think it's pretty common for read/write operations to return the bytes read/written. Java's various stream and channel APIs do the same, also all the posix IO functions.

Oh, and yes it's my fault for not properly reading the javadocs.

Don't worry

As far as I can tell, JavaCAN is working just fine.

great.