Control flow issues

[kapfab]

Software flow control (XON/XOFF) is set by default in the firmware, but looking at the code, it looks like it is not handled at all:

void handleFlowControl() {
  if (flowControl == F_NONE) return;
  if (flowControl == F_HARDWARE) {
    if (digitalRead(CTS_PIN) == pinPolarity) txPaused = true;
    else txPaused = false;
  }
  if (flowControl == F_SOFTWARE) {

  }
}

Hardware flow control (RTS/CTS) seems to be handled, though, but it’s often difficult to know if the cables and adapters we use have the related pins wired.

This leads to using no flow control at all, which is problematic when receiving hundreds/thousands of bytes at once when retrieving a web page, for example.

[kapfab]

I’d be happy to implement a few fixes and improvements, do you have some documentation on how to build the firmware?

EDIT: Just saw https://github.com/ssshake/vintage-computer-wifi-modem/blob/master/docs/build-firmward.md, I’ll have a try.

EDIT: On step 3, esp_modem/esp_modem.ino should probably be theoldnet_serial_wifi_modem/theoldnet_serial_wifi_modem.ino but I’m stuck on step 4. Choose your board here and make sure all of the settings in the Tools menu matches this screen shot as I don’t see what is meant here.

[ssshake]

Update that document link with an image showing which device to choose.

Every version of the modem going out now had hardware flow control.

Version 3 and under of the modem should be software flow control. Version 4 and up should be hardware flow. I don't see a config option to set the flow control.

I agree that software should be supported. I am no sure how to properly implement it but if you do please help out!

[kapfab]

Just a few thoughts: default handshaking should be set to none (AT&K0) because defaulting to RTS/CTS could prevent someone without a correctly wired cable to reconfigure and use the modem in its default configuration. No handshaking allows everyone to reconfigure the handshaking regardless of the cable used.

[ssshake]

@kapfab any idea how to implement the software flow control? I don't have a lot of spare time right now to address this and I would have to do some research.

[kapfab]

In the same way as what is done for hardware flow, as long as we only consider the XON/XOFF sent by the other side and we don’t bother with our own buffer (which is unlikely to get full), that should be quite simple. I’ll try a quick implementation in the next few days.

[ssshake]

The hardware flow control physically checks the properties of the rts cts pins, which don't exist of course for software flow.

if (digitalRead(CTS_PIN) == pinPolarity) txPaused = true;
    else txPaused = false;

I believe software flow has to check for a specific character and act on that. I'm just not sure which character and if the modem side should check for off or on.

https://en.wikipedia.org/wiki/Software_flow_control

I'm also going to look at this firmware as it seems far more robust and already supports flow control from the looks of it. https://github.com/bozimmerman/Zimodem.git

[kapfab]

Yes, it’s XOFF/XON (0x13/0x11). We have to pause/resume the data flow accordingly and remove these bytes from the actual data flow.

[kapfab]

I had a look at the current firmware code today and it seems that hardware flow control is OK in command mode and TCP mode but not with PPP (the ppp_output_cb function writes the whole buffer at once to the serial port without checking its status). That could explain why I encounter issues getting “large files” with PPP on EPOC32 even with hardware flow control enabled.

[kapfab]

I left the XON/XOFF topic aside while looking the overall hardware flow control problem.

The txPaused method to handle handshaking is fine for the TCP client (because we can decide when to retrieve the data to be sent) but it can’t be used for PPP (the data comes along with the ppp_output_cb callbacks and would be lost if ignored).

So I did some research and it looks like the following code activates the ESP8266’s hardware flow control:

      U0C0 |= 1<<UCTXHFE;
      U0C1 |= 1<<UCRXHFE;
      U0C1 |= 64<<UCRXHFT; // Not sure which is the best threshold value to be used (max is 127)
      PIN_FUNC_SELECT(PERIPHS_IO_MUX_MTCK_U, FUNC_UART0_CTS);
      PIN_FUNC_SELECT(PERIPHS_IO_MUX_MTDO_U, FUNC_UART0_RTS);

I removed the custom flow control methods so all data to be sent is simply added to the TX buffer and the UART manages the flow on its own. TX buffer overflows can happen, but it’s beyond our control (the baud rate should just be lowered).

I didn’t encounter any RX overflow on the computer side during my tests since using this (before that, even the AT? output was causing overflows at 115200 bauds).

But I still have some tests to run because I can’t manage to have PPP working at 115200 bauds on my Psion 5mx (probably an issue on the Psion side). And at 57600 bauds I don’t expect many overflows on the Psion side, even without flow control enabled. ATGEThttp://theoldnet.com at 115200 bauds is working flawlessly, though.

And then we’ll be back to the question: how to manage XON/XOFF with PPP (without having to implement an awful custom buffer)?

EDIT: I understand why the PIN_FUNC_SELECT() lines actually work. In the current firmware code we can read

#define RTS_PIN 4         // RTS Request to Send, connect to host's CTS pin
#define CTS_PIN 5         // CTS Clear to Send, connect to host's RTS pin

which made me think RTS and CTS pins were wired to GPIO 4 and 5. But looking at the tracks on the board, they are actually wired to GPIO 15 (U0RTS) and 13 (U0CTS). So that means the current custom RTS/CTS flow control is not working at all but the good news is that the native ESP8266 RTS/CTS flow control can be used.

My last tests seem conclusive. The few slowness issues I encountered yesterday were due to a lack of performance on the Psion side.

[ssshake]

Sorry that I haven't replied I've been very busy with personal stuff. If I understand what you've said my firmware has the wrong pins and I never changed them from 4,5 to 13, 15. My schematic is definitely going to 13, 15. I used to have them at 4,5 because I read that 13,15 has gotcha's on power up so I temporarily moved them and then moved them back when I realized 13 and 15 are native RTS/CTS as you pointed out.

Do you have a version of your changes to submit for a PR? I'm assuming from what you wrote that you removed using RTS_PIN/CTS_PIN all together, removed the hardware flow function, and added the U0C0 stuff so that the uart takes over the job. Is that right?

[kapfab]

I have a version with hardware RTS/CTS but I still wonder whether this is a good or bad thing to do with the ESP8266 because serial writes are blocking and they don’t yield.

So if for any reason the computer on the other side is stuck in a non-ready state (what actually happened a few times on my EPOC devices for obscure reasons when using PPP), writes are waiting until the watchdog bites.

What is your opinion?

[ssshake]

Honestly this is over my head at this point. I would have to learn to much to get up to baseline with you.

[kapfab]

That’s fine. I’ll open a few PRs on different topics in a few weeks (hardware RTS/CTS, additional AT commands, different use of the two LEDs). Hopefully around end of August, I’ll be busy for a few weeks.

[ssshake]

OK thank you, much appreciated!

Again I do wonder if it's worth looking at another project that claims to support what we are going for here. Maybe they have over come these limitations or maybe they are in the same boat for the same reasons.

Sometimes I find it helps to compare something to an existing working solution because then it eliminates any uncertainty between the thing not working because we aren't doing it right or not working because of something out of our control.

I could load up this firmware but I'm ill equipped to evaluate it at this point.

https://github.com/bozimmerman/Zimodem

[kapfab]

Zimodem seems fine but it does not support PPP for now. And PPP is where the flow control problem lies (because of lwIP’s PPP callbacks that run outside of the loop).

[ssshake]

My bad I thought they did have PPP support.

I wonder is there any reason to look at an ESP32 instead, would it still have the same constraints? I suppose if the chip is architected the same way it would.

[kapfab]

Yes, that would be the same with ESP32.

The solution would be to find another PPP implementation to be used with lwIP but I don’t think there is.

It looks like old lwIP PPP implementations were allowing data retrieval without callbacks but this method was removed purposely so…

Anyway, lost/invalid packets are handled by PPP so I’m inclined to think that using no flow control is the only way to go with PPP and hardware flow control should be used for all other uses where we can safely pause the network data flow from within the loop.

[ssshake]

OK thanks for the valuable research. It's unfortunate that hardware flow can't work with PPP because PPP is a feature where speed would matter the most.

Where did you find this information btw? I should do some reading.

[kapfab]

Hardware flow can work with PPP but this can lead to instabilities or resets. Let me share my understanding of how things work.

With PPP, packets from the network stack are retrieved via callback functions which are triggered when data is available.

These callback functions are running outside the loop context, in the same context as the WiFi stack. If callback functions take too long to return, I understand this could make the WiFi stack unstable (which I might be what sometimes happens to me with my devices).

So unless we implement a custom buffer to make the callback function stack this data to make it available from within the loop, we have to send it ASAP directly from the callback function. Which is an issue as Serial.write loops until it can actually send the data, what can get blocked by hardware flow control.

And implementing a custom buffer would either quickly eat the available memory or just replace a watchdog reset with memory full issues when the computer on the other side is stuck/refusing incoming data for too long.

[kapfab]

You can find interesting comments there : https://github.com/esp8266/Arduino/issues/3042

[ssshake]

Worth looking at how flow control was implemented in a similar firmware found here: https://github.com/8bit-bruno/WiFiModem/commit/32d3bd7286c8d2daa36b83fb77ffb9c8589bf37f

May be worth trying that firmware on my device and seeing if perhaps it's a better solution all around. The current firmware is originally based off of a commodore 64 modem and it was initially meant to be just a starting point. It's not necessarily the most suitable option.

I'll try to set aside some time this fall exploring this.