We want to analyze / process log messages from a hardware logger. Unfortunately the logs are quite big and due to the logger, some messages might come out of order (regarding the time).
Now we would need to sort the messages after time. But since the logs are quite big we can not collect the whole dataframe.
We would rather sort based on time within the individual batch of messages in the LazyFrame. At least for us this would be sufficient since messages are only slightly out of order. So a message from the start might not appear at the far end of the file.
Description
We want to analyze / process log messages from a hardware logger. Unfortunately the logs are quite big and due to the logger, some messages might come out of order (regarding the time). Now we would need to sort the messages after time. But since the logs are quite big we can not collect the whole dataframe. We would rather sort based on time within the individual batch of messages in the LazyFrame. At least for us this would be sufficient since messages are only slightly out of order. So a message from the start might not appear at the far end of the file.
Is this somehow possible?