Problems reading from FT230X

[Windwoes]

I'm trying to use this library to talk to a device which uses an FT230X chip. It shows up as /dev/ttyUSB0 when I plug it in, and I can successfully use this library to open it, set the baudrate, and send data to the device. (The device reacts to the commands as expected.) However, I'm having issues reading from it. If I manually call the read functions, I get something like 1 byte every 5 seconds, when I am expecting several bytes every 50ms. If I use the event-driven reader, I get nothing back. Any ideas what might be going on? OS is Ubuntu 18.04 x64 with kernel 5.3.0-42.

[ircole]

I have a similar problem with a FT232R adapter on a Raspberry PI (Buster). The application only reads the serial data. It sometimes gets the whole packet, sometimes none, and most times seems to get a varying amount of data but missing some off the tail end. Running 9600 baud. A QinHeng Electronics HL-340 USB-Serial adapter works fine. The FT232R works fine with minicom so seems the problem is related to Java and jSerialComm. Haven't confirmed it but suspect the problem may be what you're observing, too long between bytes. The packets are small 11 bytes max. My program is reading 1 byte at a time in a loop.
if (comPort.bytesAvailable() > 0) { comPort.readBytes(rdBuffer, 1); value = rdBuffer[0]; } else

[ircole]

The problem occurs on Linux Mint 19.3 too so isn't RPI specific in my case. I added a Thread.sleep(1); in my polling loop which seems to resolve the problem which implies some thread being starved. In my case I should've had the sleep anyway as my past experience in the world of devices in the POS Java world often had problems if there was a tight loop without a sleep.

[hedgecrw]

@Windwoes, I know it's been forever since you raised this issue, but wanted to check in and see if you were able to solve it? From your description, it sounds most like a problem with the way the timeouts are set up.

@ircole, your problem is more concerning since you receive differing amounts of data each time. Were you able to resolve your problem? If not, would you be able to post your port setup/initialization code? Thanks!

[Windwoes]

@hedgecrw

I know it's been forever since you raised this issue, but wanted to check in and see if you were able to solve it? From your description, it sounds most like a problem with the way the timeouts are set up.

I messed with it a bit more but wasn't able to get it to work. IIRC, I saw the same behavior on Windows 7 as well.

More recently however (this summer) I was working on firmware for an Arduino Mega which used a custom command-response serial protocol to talk to a Java application on the computer, and I didn't have any issues with reading serial data. So I feel like it might be something specific to the FT230X? I did however run into a different issue when communicating with the Arduino, which was that I had to send a throwaway command to the device before calling close() on the serial port object, otherwise it would just hang.

[ircole]

The solution for my problem was to add a Thread.sleep(20) to a tight loop in my code 'polling' for data. In Java it's not uncommon to cause some low level device driver problems due to starving their thread of CPU cycles which is easy to do with a tight loop that doesn't have a sleep. The problems resulting from the starvation vary from poor performance to intermittent error to not working at all. Been a while since I had that problem but I spent a lot of years with Point of Sale where problems occurred with devices due to starvation. What's funny is I'm retired and was developing some software for a client on my own and I committed the sin that I used to chastise developers for committing and it took me a while to figure it out. Actually the tight loop doesn't have to be related to the device. /**

• Read one byte from the com port if available within the timeout period.
• @param timeout
• @return / protected byte readByte(long timeout) { byte value = -1; byte[] rdBuffer = new byte[2]; int exitFlg = 0; long startTime = System.currentTimeMillis(); while (exitFlg == 0) { if (comPort.bytesAvailable() > 0) { comPort.readBytes(rdBuffer, 1); value = rdBuffer[0]; exitFlg = 1; } else { if (ckTimeout(startTime, timeout)) { exitFlg = 1; } } sleep(20); / Method sleep() is just Thread.sleep() wrapped in try catch */ } return value; }

[hedgecrw]

@Windwoes, regarding sending your throwaway command or else close() would hang, I think that was a problem others were experiencing as well. I've implemented a number of changes recently attempting to fix this issue. If you get a chance, please test the following library version and see if it resolves this problem for you. If so, it should end up in the next official release:

https://drive.google.com/file/d/17nqSfZDaLu7EJQhe25NXLTepWKG8dFfz/view?usp=sharing

[Windwoes]

@Windwoes, regarding sending your throwaway command or else close() would hang, I think that was a problem others were experiencing as well. I've implemented a number of changes recently attempting to fix this issue. If you get a chance, please test the following library version and see if it resolves this problem for you. If so, it should end up in the next official release:

https://drive.google.com/file/d/17nqSfZDaLu7EJQhe25NXLTepWKG8dFfz/view?usp=sharing

Hmm, I don't actually have an Arduino Mega currently that I can deploy my firmware onto to test, but I will see if I can get something going on my Uno to either reproduce the issue (or not).

[Windwoes]

@hedgecrw So I tested that beta JAR on my Linux box with a similar firmware loaded onto my Uno..... And I'm very confused. Not only can I not reproduce the issue I was having before (it closes perfectly well without sending a throwaway command), but also your beta version actually has a bug where although closing succeeds, it does not kick an ongoing read out of the wait state. (I am using a timeout of 0 (infinite?) with BLOCKING reads).

[hedgecrw]

@Windwoes, I'm glad that closing seems to work okay for you now, but I'm trying to figure out your second issue where a close() does not cause a blocking read in another thread to return. I wrote the following test application and am unable to replicate this issue on Mac, Windows, or Linux. Would you be able to test out the following and see if it still doesn't work for you? (I'm also attaching a precompiled JAR file with essentially the same code that you should just be able to use directly from the command line...it'll let you select your desired serial port).

final SerialPort ubxPort = SerialPort.getCommPorts()[0];
boolean openedSuccessfully = ubxPort.openPort(0);
System.out.println("Opening " + ubxPort.getSystemPortName() + ": " + ubxPort.getDescriptivePortName() + " - " + ubxPort.getPortDescription() + ": " + openedSuccessfully);
if (!openedSuccessfully)
   return;
System.out.println("\nReading for 5 seconds then closing from a separate thread...");
ubxPort.setComPortTimeouts(SerialPort.TIMEOUT_READ_BLOCKING, 0, 0);
Thread thread = new Thread(new Runnable()
{
    @Override
    public void run()
    {
        byte[] buffer = new byte[2048];
        while (ubxPort.isOpen())
        {
            System.out.println("\nStarting blocking read...");
            int numRead = ubxPort.readBytes(buffer, buffer.length);
            System.out.println("Read " + numRead + " bytes");
        }
        System.out.println("\nPort was successfully closed from a separate thread");
    }
});
thread.start();
try { Thread.sleep(5000); } catch (Exception e) { e.printStackTrace(); }
System.out.println("\nClosing " + ubxPort.getDescriptivePortName() + ": " + ubxPort.closePort());
try { thread.join(); } catch (Exception e) { e.printStackTrace(); }

Test JAR: https://www.dropbox.com/t/i3wCyM0boIAv3Vp6

If this program works for you, it'd be good to see some of your test code to figure out what the difference is. Thanks!

[Windwoes]

@hedgecrw Your test JAR prints this output, then hangs:

Using Library Version v2.8.0

Available Ports:

   [0] ttyACM0: USB-Based Serial Port - USB-Based Serial Port

Choose your desired serial port or enter -1 to specify a port directly: 0

Opening ttyACM0: USB-Based Serial Port - USB-Based Serial Port: true

Reading for 5 seconds then closing from a separate thread...

Starting blocking read...

Closing USB-Based Serial Port: true

At that point, if I then physically unplug the Arduino, the program will exit with

Read 0 bytes

Port was successfully closed from a separate thread

[Windwoes]

I think it's probably specific to the Linux version, because if I run it on Windows 7 it works fine:

Available Ports:

   [0] COM1: Communications Port (COM1) - Serial0
   [1] COM8: Arduino UNO R3 (COM8) - USBSER000

Choose your desired serial port or enter -1 to specify a port directly: 1

Opening COM8: Arduino UNO R3 (COM8) - USBSER000: true

Reading for 5 seconds then closing from a separate thread...

Starting blocking read...
Read -1 bytes

Port was successfully closed from a separate thread

Closing Arduino UNO R3 (COM8): true

[hedgecrw]

@Windwoes, very interesting. My Linux test setup is primarily Ubuntu on x86_64. Which Linux distro, version, and architecture are you using? I'd like to compile a number of C-only programs to pinpoint where exactly the native code is freezing. It'd be really good if we could get this issue fixed for ALL architectures and OSs, but it seems like there's always some setup that causes a problem! Thanks again for your help.

[Windwoes]

@hedgecrw My Linux machine is on Ubuntu 20.04.3, kernel version 5.11.0-38-generic, on x86_64.

[ircole]

I tried this on my Linux Mint 20.1, 5.4.0-91-generic (#102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021) and it works properly in a clean 'termina' with a fresh plugged USB device, but it hangs if I attempt to run the program a second time without unplugging and plugging the device. I've tried a MKR1500 and an ESP32 with the same results. If I blow away the terminal session and open a new one without unplugging/plugging the device the device doesn't show up in the list of devices. I don't know if my testing is relevant to the problem or not. Being I get notified on updates to this I decided to give it a try just for the heck of it.

[hedgecrw]

@ircole, thanks a ton for testing this as well. I'm not sure if your issue is exactly the same (especially regarding not re-enumerating the port after a problem, since that shouldn't require being able to actually open/close the port), but the hanging on close() portion might be related.

Could you and @Windwoes try testing one additional change I made here and see if it clears anything up:

https://www.dropbox.com/t/ESbGTTTcYDPB1ZMs

[Windwoes]

@hedgecrw That change seems to have fixed the issue :)

I'll try to find some time in the near-ish future to try talking to the FT230X again.

[hedgecrw]

Great! Here is a link to a full v2.8.0 beta JAR you can use for testing: https://www.dropbox.com/t/xlyLiDkcTXyLLZmE

[ircole]

The new code works great in my environment too. Don't see the problem with having to unplug and plug the device. Great work!

[Windwoes]

Great! Here is a link to a full v2.8.0 beta JAR you can use for testing: https://www.dropbox.com/t/xlyLiDkcTXyLLZmE

@hedgecrw Can confirm that this JAR does not experience the read hang when closing issue :)

Also - I pulled out the device with the FT230X chip again, and now things seem to be working properly with that as well! I am currently using blocking reads though, and I think I may been trying non-blocking before. I might mess with non-blocking reads and see if those work as expected too.

[hedgecrw]

Great news on all fronts - looks like we might finally be close to having all these closing issues resolved. Again, thanks you guys for your testing help.

[Windwoes]

I might mess with non-blocking reads and see if those work as expected too.

Okay update on this.... turns out I had a bug in my FIFO buffer that was causing it to not populate my destination buffer completely when using a non-zero offset. Once I fixed that, reading seems to be working well with semi-blocking mode.

However, there seems to be another closing hang issue with semi-blocking mode. serialPort.closePort(); returns just fine, but then the serialPort.readBytes() call in another thread hangs. I have the timeout set to 100ms, but it still doesn't return after that time. But in any case, I would expect the readBytes() call to return immediately when closePort() is called, regardless of whether the timeout has elapsed or not.

[hedgecrw]

I'm not able to reproduce this on my end. I've written the following test code which I think mimics what you're saying:

final SerialPort port = SerialPort.getCommPorts()[0];
port.setComPortTimeouts(SerialPort.TIMEOUT_READ_SEMI_BLOCKING, 100, 0);
System.out.println("\nOpening " + port.getSystemPortName() + ": " + port.getDescriptivePortName() + " - " + port.getPortDescription() + ": " + port.openPort(0));
System.out.println("\nReading for 3 seconds then closing from a separate thread...");
Thread thread = new Thread(new Runnable()
{
    @Override
    public void run()
    {
        byte[] buffer = new byte[2048];
        while (port.isOpen())
        {
            System.out.println("\nStarting semi-blocking read...");
            int numRead = port.readBytes(buffer, buffer.length);
            System.out.println("Read " + numRead + " bytes");
        }
        System.out.println("\nPort was successfully closed from a separate thread");
        System.out.println("\nAttempting a further read on the closed port...");
        int numRead = port.readBytes(buffer, buffer.length);
        System.out.println("Read " + numRead + " bytes");
    }
});
thread.start();
try { Thread.sleep(3000); } catch (Exception e) { e.printStackTrace(); }
System.out.println("\nClosing " + port.getDescriptivePortName() + ": " + port.closePort());
try { thread.join(); } catch (Exception e) { e.printStackTrace(); }

When running this code, I get a bunch of successful reads, followed by a "Closing XXX: true" message, followed by "Port was successfully closed from a separate thread. Attempting a further read on the closed port...", followed immediately by "Read -1 bytes" which is all as expected.

I'm currently testing against my own Release Candidate 1 version of the library though, so maybe something got fixed in the interim. Could you please retest with the same version I'm using and see what your results are:

https://www.dropbox.com/t/xK0OjqsvWTHzhDac

[Windwoes]

Sigh my apologies. I ran your test code and it worked fine. I took another look at my project I had a bit of a git mixup and missed an important part of a diff that released a reentrant lock in a finally clause. It created the illusion that the read was hanging when it actually wasn't. Sorry for the trouble.

[hedgecrw]

Closing this issue as resolved with the release of jSerialComm v2.8.0. Thanks for everyone's help in diagnosing and testing!