Describe the internet protocol used by the PyKOB server and clients.

[leskerr]

This will cover the UDP packet format, server commands, and extensions to the CWCom protocol to support closed-circuit telegraphy.

[leskerr]

I've created the first draft of a wiki page describing the internet protocol used by the pykob.internet module. Feel free to correct errors. Any feedback is always welcome.

[leskerr]

The new wiki page is located here.

[pwdirks]

Hi Les - looks great. A few random comments:

• Overall: you may want to mention that the data sent between client and servers represents intervals the key is open/closed, no more, no less? People may imagine ASCII dots/dashes being sent, or ASCII text to be converted back into morse, or whatever. Might be worth mentioning the idea is to move the receiver's sounder exactly the same timing as the sender to convey the morse being sent as faithfully as possible. You may want to talk about a unique "fist"?

• I see you explain in more detail below but when you introduce the various packet types, maybe say a few words about the reason/time each is invoked, or its (intended) effect? E.g. "Receipt of a 'CONNECT' packet starts the server sending traffic for the selected wire" or "Sending an ID packet controls the station identification string in the KOB window" etc.

• I don't know if there's a way to incorporate the packet layout PDF in-line but that may be more convenient than triggering a download. Is there someplace we can upload the PDF for reference in the Wiki (with [] IIRC?

• [not about the documentation] There's nothing preventing newer versions of PyKOB from being written to use the 'flags' field to distinguish ID packets? The server could be updated to recognize both mechanisms, at least for now, and ultimately drop the older if desired?

• I think the 'CODE' packet description needs more details: positive and negative time intervals, special intervals added to the original CWCom protocol to convey opening/closing key, etc.? Details of timing (mSec, IIRC), and maximum values? The convention of ending a CODE packet payload with a negative time value? Or starting with one?

• Does the server do anything in response to an ACK packet? What happens if you never send one?

• In the discussion of sequence numbers, it may be clearer to mention FIRST that every packet is sent twice, THEN mention that you can tell which is a repetition by the mechanism of the sequence numbers?

Thanks for pulling this together!

[AESilky]

Looks good on initial read, I'll re-read with Patrik's comments in mind. For the PDF, there might be a way to include it inline, but if not, we could generate an image version of it and put that inline and include the link to the PDF.

[AESilky]

For the PDF, I also think that there is a way to have it open in a new browser tab rather than kicking off a file download.

[leskerr]

For the PDF, there might be a way to include it inline, but if not, we could generate an image version of it and put that inline and include the link to the PDF.

I know the download of the PDF is clumsy. I couldn't come up with any way of including it inline, and I don't know why the link doesn't open a new browser tab automatically instead of downloading. I thought of representing the diagrams as Markdown tables, but they wouldn't look as nice as the PDF. I could convert the PDF to an image and include that with the text, but I was afraid the image size would come out wrong on some devices. So I'm stuck not knowing what to do...

[leskerr]

• [not about the documentation] There's nothing preventing newer versions of PyKOB from being written to use the 'flags' field to distinguish ID packets? The server could be updated to recognize both mechanisms, at least for now, and ultimately drop the older if desired?

I think this is a good suggestion. I'll create a separate issue for it. (I don't know how the dependency mechanism works between related issues, or whether it even applies in this case.)

[leskerr]

• Overall: you may want to mention that the data sent between client and servers represents intervals the key is open/closed, no more, no less? People may imagine ASCII dots/dashes being sent, or ASCII text to be converted back into morse, or whatever. Might be worth mentioning the idea is to move the receiver's sounder exactly the same timing as the sender to convey the morse being sent as faithfully as possible. You may want to talk about a unique "fist"?

I agree this is important information. Right now some of the details are in the Code sequences and Morse timing wiki pages. Maybe we need a higher level document that ties it all together, outlining the design goals and providing some historical context.

• I see you explain in more detail below but when you introduce the various packet types, maybe say a few words about the reason/time each is invoked, or its (intended) effect? E.g. "Receipt of a 'CONNECT' packet starts the server sending traffic for the selected wire" or "Sending an ID packet controls the station identification string in the KOB window" etc.

I'll try to clarify this on the next edit.

• I think the 'CODE' packet description needs more details: positive and negative time intervals, special intervals added to the original CWCom protocol to convey opening/closing key, etc.? Details of timing (mSec, IIRC), and maximum values? The convention of ending a CODE packet payload with a negative time value? Or starting with one?

I think this is all covered in the Code sequences page. I need to check to make sure. (A code sequence always starts with a negative value and ends with a positive one.)

• Does the server do anything in response to an ACK packet? What happens if you never send one?

The server never expects to receive an ACK packet, because it never sends a CON packet to the client. I'm not sure what happens if it does receive an ACK packet. Hopefully it gracefully ignores it, but I don't think it's ever happened so I'm not sure.

• In the discussion of sequence numbers, it may be clearer to mention FIRST that every packet is sent twice, THEN mention that you can tell which is a repetition by the mechanism of the sequence numbers?

I can switch the order, but I'm not sure it actually makes it clearer. Tell me one more time if you still think it helps and I'll make the change.

Hopefully I've addressed all the comments, one way or another. Please let me know if I've overlooked anything.

[pwdirks]

I agree this is important information. Right now some of the details are in the Code sequences and Morse timing wiki pages. Maybe we need a higher level document that ties it all together, outlining the design goals and providing some historical context.

Perhaps a pointer to the pages you refer to would be sufficiently helpful.

Does the server do anything in response to an ACK packet? What happens if you never send one? The server never expects to receive an ACK packet, because it never sends a CON packet to the client. I'm not sure what happens if it does receive an ACK packet. Hopefully it gracefully ignores it, but I don't think it's ever happened so I'm not sure.

I guess I figured the ACK was related to the timing data stream but you're right, that's not the use. Why does the server send an ACK? What should the client do if one is not received? Keep re-sending the CON packet until it gets an ACK? That might be good to spell out.

In the discussion of sequence numbers, it may be clearer to mention FIRST that every packet is sent twice, THEN mention that you can tell which is a repetition by the mechanism of the sequence numbers? I can switch the order, but I'm not sure it actually makes it clearer. Tell me one more time if you still think it helps and I'll make the change.

Until you realize packets are sent twice to mitigate expected packet loss, the description of the sequence numbers seems to be for no reason. I think explaining FIRST how and why packets are sent multiple times, and THEN getting into how to construct a single coherent data stream from them by ignoring packets w. the same sequence number will make more sense.

Besides warning clients to monitor sequence numbers and ignore duplicate packets, you may also want to advise clients on how to handle out-of-order packet receipt? If CODE packets are received with sequence numbers 1, 1, 2, 2, 4, should the packet be discarded while waiting for a 3? Or held back until #3 is received? Or advanced to #4 and ignore #3 if it's ever received?

[leskerr]

I agree this is important information. Right now some of the details are in the Code sequences and Morse timing wiki pages. Maybe we need a higher level document that ties it all together, outlining the design goals and providing some historical context.

Perhaps a pointer to the pages you refer to would be sufficiently helpful.

In the 'Code packet' section at https://github.com/MorseKOB/PyKOB/wiki/Internet-protocol#code-packet there's a link to the 'Code sequences' page, which has all the details on how timing is encoded. Does the link not work for you?

By the way, it looks like the hyperlinks in my comments aren't making it through to the quoted text.

Does the server do anything in response to an ACK packet? What happens if you never send one? The server never expects to receive an ACK packet, because it never sends a CON packet to the client. I'm not sure what happens if it does receive an ACK packet. Hopefully it gracefully ignores it, but I don't think it's ever happened so I'm not sure.

I guess I figured the ACK was related to the timing data stream but you're right, that's not the use. Why does the server send an ACK? What should the client do if one is not received? Keep re-sending the CON packet until it gets an ACK? That might be good to spell out.

The client keeps sending CON packets every ten seconds, no matter what, as long as the Connect button in the MorseKOB GUI window is depressed. If an ACK packet is received back from the server, then the connect light stays solid red. If no ACK packet is received, then the connect light flashes red/white to alert the user that the connection to the server has been lost.

In the discussion of sequence numbers, it may be clearer to mention FIRST that every packet is sent twice, THEN mention that you can tell which is a repetition by the mechanism of the sequence numbers? I can switch the order, but I'm not sure it actually makes it clearer. Tell me one more time if you still think it helps and I'll make the change.

Until you realize packets are sent twice to mitigate expected packet loss, the description of the sequence numbers seems to be for no reason. I think explaining FIRST how and why packets are sent multiple times, and THEN getting into how to construct a single coherent data stream from them by ignoring packets w. the same sequence number will make more sense.

In my mind, the primary function of the sequence numbers is so the user can be notified if packets are dropped. That's why I instinctively described that function first. In theory, it also serves to detect out-of-order packets, although in practice I've never seen that happen.

Dropped packets definitely do happen, though, especially with wireless connections. Sending the data packets twice is an attempt to mitigate that problem. What I've noticed, though, is that if the first packet is lost in the network, very often the second packet doesn't make it through either. I experimented with sending packets three or four times, but it didn't help. I left it the same as CWCom, which sends packets twice. The sequence number is necessary to recognize a duplicate packet, but I've always viewed that as a secondary purpose of the sequence number.

Such is my thinking, for what it's worth.

Besides warning clients to monitor sequence numbers and ignore duplicate packets, you may also want to advise clients on how to handle out-of-order packet receipt? If CODE packets are received with sequence numbers 1, 1, 2, 2, 4, should the packet be discarded while waiting for a 3? Or held back until #3 is received? Or advanced to #4 and ignore #3 if it's ever received?

Based on my experience, dealing with out-of-order packets isn't worth the effort because it happens so rarely. This is where my laziness factor kicks in.

I've often wondered why CWCom went with UDP instead of TCP/IP. But because I was migrating CWCom users to MorseKOB, I had no choice but to use the same protocol.

[AESilky]

I added an inline image of the packet diagram to the wiki and left a link to the PDF if people want to download it.

[leskerr]

I added an inline image of the packet diagram to the wiki and left a link to the PDF if people want to download it.

Thanks, @AESilky. It looks good. And thanks to @pwdirks for creating the diagrams in the first place.

[pwdirks]

Love the in-line packet diagram - I think that's a really nice improvement. It looks like you have "The sequence number field" linked to "code sequences", though, when I assume you meant to link that to data packets?

Actually UDP packets may be duplicated in flight, too. Perhaps the description of sequence numbers would be better if it was just expanded to read "This allows clients to detect missing, duplicated, or out of order packets" instead of just "... out of order or missing packets"?

I wonder if the page wouldn't be better with a "one stop shopping" model where ALL of the info, including the "code sequence" material and the "sequence number" material, was put into a section on the main page? Something like "Morse Timing" seems justifiably a completely different page but it might be better to combine EVERYTHING related to the on-the-wire format onto this page? The additional detail is easy to miss otherwise.

[leskerr]

Love the in-line packet diagram - I think that's a really nice improvement. It looks like you have "The sequence number field" linked to "code sequences", though, when I assume you meant to link that to data packets?

Good catch! I thought I checked everything, but that one slipped by. I've fixed it.

Actually UDP packets may be duplicated in flight, too. Perhaps the description of sequence numbers would be better if it was just expanded to read "This allows clients to detect missing, duplicated, or out of order packets" instead of just "... out of order or missing packets"?

This is a very good suggestion. I've changed the text accordingly.

I wonder if the page wouldn't be better with a "one stop shopping" model where ALL of the info, including the "code sequence" material and the "sequence number" material, was put into a section on the main page? Something like "Morse Timing" seems justifiably a completely different page but it might be better to combine EVERYTHING related to the on-the-wire format onto this page? The additional detail is easy to miss otherwise.

I agree there's a need for a high level description that ties all the details together. In the interim, is there a way we can maintain some sort of Q&A list to capture the essence of stuff someone might need to know, and worry about writing the documentation later? Writing a polished document is kind of painful for me.

If you like I can always append the Code sequences page to the end of the Internet protocol page. And we could add a Notes section to the end of the page, for details that aren't covered elsewhere.

[AESilky]

I really like this! Great to get it in place.

We need to think about when this can/should be the 'Go To' location for information. I'm assuming it will be when we have the PyKOB (MorseKOB 4.0) working to the point that people can use it rather than using MorseKOB 2.5?

Thoughts???