The CCM Service processes incoming messages from Chocolatey Agent machines as described here. The intention was always the case that should there be a corrupted/invalid XML file located in the incoming folder, that an attempt would be made to process it, and if a failure happened, that processing of other messages should continue, and the file be left in place, to be attempted again.
However, on testing, it was found that if there is an invalid XML file in the queue, that messages that appear after it, won't be processed. i.e. if there is a message from today that is invalid, any messages that came in yesterday that are still in the queue, won't be processed. The default ordering of messages is by date, newest to oldest.
The end result here is that there can be a backlog of messages in the processing folder, that never get processed.
Steps To Reproduce
In the related support ticket, there are a set of 3 XML files that can be used for testing.
Get a system that has 0.6.2 of CCM Service installed
Copy the three XML files from the support ticket into the computer_information_messages folder
Watch as the CCM Service attempts to process these
In the log you should see something similar to the output log below
After finishing processing, there should still be 3 files in the folder
Delete the first file from the folder
Watch as the CCM Service attempts to process these
These two files should be correctly processed and there should be no files left in this folder
Output Log
2021-09-10 06:49:49,455 19 [INFO ] - Found 3 computer information messages to process...
2021-09-10 06:49:49,456 19 [ERROR] - Error deserializing response of type chocolatey.console.service.infrastructure.app.messaging.ReportComputerInformationMessage:
'.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
2021-09-10 06:49:49,457 19 [WARN ] - This is try 1/3. Retrying after 400 milliseconds.
Error converted to warning:
'.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
2021-09-10 06:49:49,861 19 [ERROR] - Error deserializing response of type chocolatey.console.service.infrastructure.app.messaging.ReportComputerInformationMessage:
'.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
2021-09-10 06:49:49,862 19 [WARN ] - This is try 2/3. Retrying after 600 milliseconds.
Error converted to warning:
'.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
2021-09-10 06:49:50,471 19 [ERROR] - Error deserializing response of type chocolatey.console.service.infrastructure.app.messaging.ReportComputerInformationMessage:
'.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
2021-09-10 06:49:50,472 19 [ERROR] - Maximum tries of 3 reached. Throwing error.
2021-09-10 06:49:50,473 19 [ERROR] - '.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
Notice how three files were found to be processed, but only the first one, which is invalid, is attempted, the other two files are not even looked at.
Workarounds
A manual workaround would be to identify which files are corrupt, and to delete them from the folder. This should allow the other messages in the queue to be processed. This is far from ideal though, as there may be multiple corrupt files in the queue.
Proposed Solution
A change is needed to the CCM Service to handle the case where there is a problem with a corrupt file, this should be logged, and then the next file in the queue needs to be moved to.
The file should be left in place, which would give customers the ability to attempt to fix the problem with the corrupt file, and allow it to be processed.
Description
The CCM Service processes incoming messages from Chocolatey Agent machines as described here. The intention was always the case that should there be a corrupted/invalid XML file located in the incoming folder, that an attempt would be made to process it, and if a failure happened, that processing of other messages should continue, and the file be left in place, to be attempted again.
However, on testing, it was found that if there is an invalid XML file in the queue, that messages that appear after it, won't be processed. i.e. if there is a message from today that is invalid, any messages that came in yesterday that are still in the queue, won't be processed. The default ordering of messages is by date, newest to oldest.
The end result here is that there can be a backlog of messages in the processing folder, that never get processed.
Steps To Reproduce
In the related support ticket, there are a set of 3 XML files that can be used for testing.
computer_information_messagesfolderOutput Log
Notice how three files were found to be processed, but only the first one, which is invalid, is attempted, the other two files are not even looked at.
Workarounds
A manual workaround would be to identify which files are corrupt, and to delete them from the folder. This should allow the other messages in the queue to be processed. This is far from ideal though, as there may be multiple corrupt files in the queue.
Proposed Solution
A change is needed to the CCM Service to handle the case where there is a problem with a corrupt file, this should be logged, and then the next file in the queue needs to be moved to.
The file should be left in place, which would give customers the ability to attempt to fix the problem with the corrupt file, and allow it to be processed.
Related Issues and Tickets
Zendesk Ticket
Done Checklist
Documentation has been updated.┆Issue is synchronized with this Gitlab issue by Unito ┆Milestone: 0.6.3