Performance considerations with large messages (in particular CLMSG_save_fai_log)

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 08:29:22

Performance considerations with large messages (in particular CLMSG_save_fai_log)

Original issue: http://code.google.com/p/go-susi/issues/detail?id=75

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 00:32:34

Do a load test of gosa-si-server and go-susi with and without -vvv. The load test program should send a here_i_am with a key and then send CLMSG_save_fai_log messages as fast as possible, printing on the console the rate of messages and MB/s. The program should take as input the size of the log file to send.

While the program is running, test using GOsa on the server and perform other tests to see if the server keeps working, how much memory and CPU load it has.

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 00:41:04

In go-susi.go:handle_request, when the buffer exceeds a certain size (config.MessageSizeSingleThreadThreshold), take a large_message_mutex, thereby effectively making processing (of large messages) become single threaded and limiting the maximum memory amount buffered to number_of_parallel_jobs * config.MessageSizeSingleThreadThreshold. This computation should also be used when defining config.MessageSizeSingleThreadThreshold. Let's assume max 1000 clients installing in parallel. With 1GB RAM that's 1MB message threshold. Sounds reasonable.

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 00:45:37

Add a feature to util.Log that suppresses log messages larger than util.LogMessageMaxSize. This prevents DEBUG messages with dumps of CLMSG_save_fai_log from filling the hard disk. A reasonable max size seems to be to look at the largest gosa_query_fai_release reply and take twice that size for safety.

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 00:50:53

Trying to decrypt large messages (in particular CLMSG_save_fai_log) with multiple keys is a possible time waster. Possibly optimize this by only decrypting the first block and then checking for so that we notice sooner when the key is wrong.

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 01:53:43

Because clients drop all messages from foreign servers anyway, there's no sense in transmitting elements in new_foreign_client messages. Removing this feature will speed up the last resort attempt in ProcessEncryptedMessage() because fewer client keys need to be examined.

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 01:59:00

In ProcessEncryptedMessage() try db.ClientKeys(host) first in an extra case. When (see previous comment) the transmission of in new_foreign_client messages has been removed, db.ClientKeys(host) will return an empty list unless the sender is actually a client registered with us. And in that case the key most likely to decrypt the message is the key belonging to that client. More importantly the biggest messages (CLMSG_save_fai_log) are encrypted with the client key, so trying this key first will give the biggest savings in wasted cycles for failed decryptions.

[mbenkmann]

From mux2...@gmail.com on March 12, 2013 02:02:44

In addition to the config.MessageSizeSingleThreadThreshold described above there should be a config.MaxMessageSize. If the buffer in handle_request exceeds this size, then the connection should either be dropped or handle_request should switch to a mode where it simply discards all bytes until the next \n is seen (at which point the next message starts).

[mbenkmann]

From matthias...@gmail.com on March 14, 2013 23:54:08

Instead of effectively single-threading large messages, allow up to N threads to process a large message in parallel. This avoids the problem of a single stalled connection with a large message blocking all processing of large messages. A simple way to implement this would be a deque.Deque, pre-filled with N tokens. Every goroutine that wants to process a large message Pop()s a token and when its done Push()es it again.

[mbenkmann]

From matthias...@gmail.com on March 22, 2013 02:09:12

Possibly relevant for MS 5

Labels: Milestone-5

[mbenkmann]

From matthias...@gmail.com on March 24, 2013 23:45:26

See related issue #104

[mbenkmann]

From matthias...@gmail.com on April 02, 2013 08:16:06

In the decryption process, include a shortcut. After decrypting and before parsing the XML, check if the string contains

CLMSG_save_fai_log

or the element and if so, directly send it to a specialized handler that doesn't take xml.Hash but a raw bytes.Buffer (assuming the process is rewritten so that everything until the parsing is done within bytes.Buffer)

[mbenkmann]

From matthias...@gmail.com on April 03, 2013 11:09:48

Implemented since 65cb1db691fc:

1) bytes.Buffer is used for messages from handleConnection() to GosaDecrypt and then via a shortcut even into clmsg_save_fai_log(). The only missing piece here is StringToHash() which is still used for the other messages and requires conversion of the bytes.Buffer into a string.

2) GosaDecrypt has been optimized so that it first tries to decrypt only the first 2-3 AES blocks and then checks for .

3) A load test running all 3 susikillers has shown that memory is no longer an issue, but requests that are important to GOsa become too slow (in particular gosa_query_packages_list). I'm not sure if the cause was susikiller or susikiller3. If it was susikiller3, then restricing large message processing to a pool of workers may help.

[mbenkmann]

From mux2...@gmail.com on July 12, 2013 03:15:35

Performance seems to be adequate => Closing

Status: Done