Data Extractor not processing all packets

[Jesse-Millwood]

First check out our main documentation site at https://openc3.com.

Describe the bug When exporting all of our telemetry packets for a certain time period I see the following:

Here it says it processed 17,026 packets. However, if I change to just looking at one packet, from the same target and time period I see:

Here, it is reporting that it processed 42,177 packets. This is larger than the set that it is supposed to be a part of.

To Reproduce Steps to reproduce the behavior:

1. Log a lot of packets
2. Go to Data Extractor
3. Extract all packets and save csv
4. Extract one packet type and save csv
5. Compare packets processed

Expected behavior I expect that the set of all packets contain more processed packets than a subset.

Screenshots Besides the others shared, these are the settings:

Environment (please complete the following information):

• OS: Mac OS 13.6.1
• OpenC3 COSMOS Version 5.11.0
• Browser: Firefox

[jmthomas]

Are you starting with an empty list and in the first instance clicking "Add Target", then "Process". In the second example, you clear the list, select the packet and "Add Packet", then "Process". Does the list make sense in both cases? Does the CSV have the items you expect but just not the time range? Is the time range correct but not the items?

[Jesse-Millwood]

Are you starting with an empty list and in the first instance clicking "Add Target", then "Process". In the second example, you clear the list, select the packet and "Add Packet", then "Process". Does the list make sense in both cases? Does the CSV have the items you expect but just not the time range? Is the time range correct but not the items?

That's correct, sorry I missed a couple steps. I do select add packet in the first screen shot and then process and download the CSV. Then I delete them all and select add packet in the second screenshot and then process and download the CSV.

I can verify that the number of lines in the csv's match the reported processed packets. Neither of them match what I expect though. I would expect the subset to have fewer packets than the one that has "[ALL]" packets. I also have one packet that is supposed to be sending out at a high rate (around 1600 packets/second) but those don't all seem to be picked up in the extraction of just that packet because when I just do a max() and min() on the PACKET_TIMESECONDS field the difference is only 7 seconds when I had set the time period to be a 30 second period.

[ryanmelt]

In the past, issues like this were caused by one of the packets in the playback completing, and sending an all done (empty array) to the frontend.

I thought we had all of those edge cases corrected though.

For that kind of problem, I would expect the single packet playbacks to work perfectly. And the multiple packet playbacks to end short.

Is that what you always see?

[Jesse-Millwood]

In the past, issues like this were caused by one of the packets in the playback completing, and sending an all done (empty array) to the frontend.

I thought we had all of those edge cases corrected though.

For that kind of problem, I would expect the single packet playbacks to work perfectly. And the multiple packet playbacks to end short.

Is that what you always see?

I was trying to to export other single packet sets and dig into those too. It seems that the packets that send at a relatively low rate (~1Hz) seem to include all of the packets just fine in the export, I see what I expect. However, the one set of packets that are sending at a very high rate (1600 pkts per second), I only get 7 seconds of the requested 30 seconds of data.

To be transparent here, I am trying to stress test parts of my system with the high rate packets and I'm trying to detect drops and characterize when and where they happen. I'm trying to analyze the exported data for drops in sequence counters.

I am confident that I am getting more packets down because I have a Python script (outside of cosmos) that listens to the same port and logs packets to a csv file. It gets much more of my high rate packets than the Data Exporter exports for the same time period. Where my python script, which is very single purpose grabs the packets off the stream, collects ~36k packets, the Data Extractor is giving me 409 packets.

The other part of my test involves a script in the scripting interface that initiates the sending of the packets to the ground. I print timestamps at the beginning and end of the test that are in UTC. I use that UTC time in the Data Exporter and make sure the UTC radio button is selected.

I also see this:

[ryanmelt]

closing from lack of activity. Please reopen if issue continues.