Common Criteria Gap - FAU_STG_EXT.1 External Audit Trail Storage

[RushanNanayakkara]

Describe the issue: To be eligible to be granted the Common Criteria Certification as per the certification requirement FAU_STG_EXT.1 External Audit Trail Storage the following should be facilitated by the identity server.

The evaluator shall also make the connection to the external audit storage unavailable, perform audited events on the TOE, re-establish the connection, and observe that the external audit trail storage is synchronized with the local storage. Similar to the testing for FAU_GEN.1, this testing can be done in conjunction with the exercise of other functionality. Finally, since the requirement specifically calls for the audit records to be transmitted over the trusted channel established by FTP_ITC.1, verification of that requirement is sufficient to demonstrate this part of this one.

Current behaviour: The current implementation uses an in-memory queue to buffer logs, resulting in the potential loss of log batches if the queue limit is reached. Furthermore, the existing design does not adequately handle remote server failures, which is inconsistent with the above mentioned standards.

Expected behaviour: The implementation should ensure the synchronisation of the locally stored logs with the remote server logs in the event of a remote server failure and subsequent recovery

---

Related issues:

https://github.com/wso2/product-is/issues/16697

[RushanNanayakkara]

Update Summary - 3rd of Oct 2023

Completed flows / Current progress:

1. Research for a possible solution using log4j2 for the requirement. ✅
2. Draft implementation. ✅

Details

Findings of the research

1. Implement an appender for log4j2
   • Appender uses a queue to buffer the logs before sending them to the server.
   • Periodically send the logs to the server from the queue.
   • In case of a server failure the queue will be kept and the retry mechanism will be triggered with the defined time period.
   • In case of a long failure in the server the memory usage of the queue and the frequency of the logs being generated will be considered to consider storing the logs in a temporary file which will then be sent to the server when available.
   • The upload speed of the logs to the server has to be faster than the frequency of the logs being generated. ( This is to ensure that the queued logs in the failure period will be dequeued eventually. ) Appender Choice :
   • An asynchronous appender needs to be used to be not blocking. (eg: AsyncAppender )
   • The AsyncAppenders by default uses ArrayBlockingQueue for buffering. However this is prone to deadlocks when used in multi threaded environments. Performance of AsyncAppenders
   • https://logging.apache.org/log4j/2.x/performance.html#asyncLogging
   • Therefore in that case the use of lock-free Async Loggers
   • https://logging.apache.org/log4j/2.x/manual/async.html
   • In case the logging rate cannot be handled by the queue log4j2.AsyncQueueFullPolicy can be used to configure to handle the filtering process for the logs to be preserved.
2. Improve the current implementation of SecuredHttpAppender with a file based queue.
   • Use a file based queue in place of the current on memory queue for the log events.
   • Change implementation to not drop any logs.
   • Improve the log publishing implementation with a fallback mechanism to deal with the remote server failures.
   • Implement a cleaning mechanism to remove the expired local temporary log files.

Draft implementation description.

• The draft implementation[2] enhances the existing SecuredHttpAppender implementation to use a memory mapped file based queue.
• Chronicle-Queue is used to implement the local file based queue.
  • This is a persisted low-latency messaging framework for high performance applications
• No logs will be dropped.
• Local server restarts does not result in loss of the log queue.
• Failure to push to the remote server withheld the log entries in the local files until the log publish attempt is a success.

TODO

• Finalise the discussion on the solution architecture.
• Complete the implementation.

[1] https://github.com/OpenHFT/Chronicle-Queue [2] https://github.com/wso2/carbon-commons/pull/477

[RushanNanayakkara]

Update Summary - 16th of Oct 2023

Completed flows / Current progress:

1. Finalise the discussion on the solution architecture. ✅
2. Take permission from the security team to introduce chronicle-queue dependency to the product. ✅
3. Analyse the disk usage and decide on how to impose limitations while adhering to certificate guidelines. ✅
4. Complete the implementation. 🚧

Details

Finalise the discussion on the solution architecture.

• Implementation is to be done using chronicle-queue
• Other options considered :
  • Option : Using a message broke such as Kafka.
  • Exclusion Reasons :
    • Heavy weight
    • If used the broker server has to be shipped with the product itself to be compliant with the guidelines.

Permission taken from the security team to introduce the dependency to the product.

• A transitive dependency vulnerability was found which would not affect the runtime since it is a test dependency.

  Analyse the disk usage and decide on how to impose limitations while adhering to certificate guidelines.

• The file size of the chronicle queue file is determined by the block size and the amount of messages stored.
• The cycle time can be defined in order to create new files for the queue.
• Limiting the disk usage :
  • Files are to be deleted as soon as it is consumed by the publisher.
  • The maximum number of messages that will be stored in the queue in case of a remote server failure is defined through a configuration.
  • When considering the limit the file usage should be considered.
  • The average disk usage is 1MB for 1000 messages.

Complete the implementation.

• Queue implementation : Completed using chronicle queue
• Cleanup process for used queue data files :
  • A cleanup process implemented to delete the used data files when the dequeue pointer moves to a new file.
  • This uses a meta data file which is used to persist cleaner meta data through server restarts.

TODO

• Complete the implementation. 🚧
  • Finalise error handling mechanism. 🚧
• Test the implementation. 🚧
• Finalise the design decisions
  • Where to store temporary data files. 🚧
  • How the configurations are added for the disk usages (eg: deployment.toml configuration, log4j configuration). 🚧

[RushanNanayakkara]

Update Summary - 31st of Oct 2023

Update

• Implementation using chronicle queue was completed
• A major concern was raised due to chronicle queue sending analytics data to their google analytics servers.
• This was discussed and decided that it could possibly cause a security or privacy concern.
• The option to disable the analytics data sharing through java config was considered and dismissed since the library offers no guarantee of allowing deactivation of this feature in the future updates.
• As an alternative approach a custom implementation of an memory mapped file queue was decided to be implemented.

Completed flows / Current progress:

1. Implementation of the custom file queue 🚧

Todo

1. Test the custom file queue performance.

[RushanNanayakkara]

Update Summary - 31st of Oct 2023

Update

• Implementation is complete ✅
• Testing complete ✅

Implemented design

High level architecture

• Persistent Queue(PQ) is implemented as a queue of queues.
• The high level PQ is a queue of sub-level queues(QueueBlocks).
• QueueBlocks(QB) is a queue of messages being stored in serialized form.

Disk and Memory Management

Memory Usage

• Only the appender block and the tailer block are maintained on memory on runtime.
• The new appender and tailer blocks are loaded from disk as being written to or being consumed.
• The list of ids of the queue blocks being used are maintained on memory.
• The memory usage may differ according to the QB size(configurable).

Disk Usage

• The meta data related to the PQ is stored in a json file in the root of the provided directory.
• The QB files are stored inside the 'blocks' sub-directory.
• The QB files are created and deleted on demand. Therefore the only stored QB files are the ones that are not yet consumed.
• Average disk usage : 1MB per 1000 log entries.

QueueBlock File Structure

• Implemented using memory mapped files.
• Two sections of the file include the meta data block and the messages blocks.
• The meta data block includes the first 8 bytes of the file.
• Rest of the file is used for message blocks where serialized objects are stored.

Thread Safety

Read operations are allowed for multiple threads. Write operations are synchronized.

Performance

Performance test done using audit logs generated by Asgardeo. Enqueue Rate: 13500+ operations per second Dequeue Rate: 8000+ operations per second

Testing

The implementation is tested for the following scenarios

Start up

• ✅ 1. When activated through console, the directories and files are created as expected and logs start to get published.
• ✅ 2. On server startup without the directories already present, the directories and files are created and logs are getting published as expected.

General Case

• ✅ 3. On remote server failure, logs get queued to the log files.
• ✅ 4. On remote server start up, logs are published to the remote server.
• ✅ 5. A new block is created once the current block is completely used.
• ✅ 6. Once log entries are dequeued from the queue the consumed files are deleted as consumed.
• ✅ 7. At least one queue block file is always maintained even when the queue is empty.

Configurations

• ✅ 8. Values are properly read through the log4j2.properties file.
• ✅ 9. If the configured batch size is less than the minimum value, the value should be set to the minimum value. A warning message should be printed.
• ✅ 13. If the configured max disk usage size is less than the minimum value, the value should be set to the minimum value. A warning message should be printed.

Event logs

• ✅ 10. A warning should be printed in the following instances.
  • ✅ Once the remote server is down.
  • ✅ Once the disk usage reaches 50%.
  • ✅ Once the disk usage reaches 90%.
  • ✅ Once the full disk space allocated is used.
  • ✅ Periodic warning about logs being lost each one minute.
• ✅ 11. An info message printed when remote server is reconnected.
• ✅ 12. After a restart the correct level of warning messages are printed in case of a failure again.

[RushanNanayakkara]

Closing as completed.