CoAP Bench support for PUT requests + DTLS

[Jalal-Assaly]

Greetings dear contributors.

I developed a CoAP server enabled with DTLS PSK using the Californium library. It all works perfectly well and I am capable of sending a DTLS PUT request from an embedded ESP32 client. Now I want to benchmark my server using CoAP Bench. However, it seems to me that at the current time, it is only supporting GET requests with no DTLS encryption. Is it correct ?

If yes, how can I go about benchmarking my server with PUT requests (including payload) and DTLS encryption ?

Thank you very much for your collaboration.

[boaks]

I mainly use Cf-Extplugtest-client/BenchmarkClient for that.

Usually the users, which try to implement their own benchmark, should obey, that using a single client doesn't work well. On the server side it's usually a question of the heap consumption, caused by the request deduplication (detection of retransmissions). Read through Californium - issue #2167 for some details.

In order to reduce that heap consumption on the server side, you may try following settings:

# Deduplicator algorithm.
# [MARK_AND_SWEEP, PEERS_MARK_AND_SWEEP, CROP_ROTATION, NO_DEDUPLICATOR].
# Default: MARK_AND_SWEEP
COAP.DEDUPLICATOR=PEERS_MARK_AND_SWEEP
# Maximum messages kept per peer for PEERS_MARK_AND_SWEEP.
# Default: 64
COAP.PEERS_MARK_AND_SWEEP_MESSAGES=16
# Use message off-loading, when data is not longer required.
# Default: false
COAP.USE_MESSAGE_OFFLOADING=true

[Jalal-Assaly]

Thank you very much for your help ! I am now able to benchmark my CoAP server using DTLS with PSK and POST requests. However, there's still an issue I am facing. Whenever I try to benchmark the server with concurrent requests, even with 2 virtual clients only, the server is unable to process requests incoming from the second client. This is my command: java -jar cf-extplugtest-client-3.12.0-SNAPSHOT.jar BenchmarkClient coaps://localhost:5684/accessControl --method POST --payload-file payload.json --json --auth PSK --identity Access_Point --secret secretPSK --clients 2 --requests 10

And I get the following logs: Benchmark clients, first request successful. Benchmark clients created. 1 ms, 1000 clients/s 20:36:03.366: register shutdown hook. Benchmark started. [0001]: 11 requests (1 reqs/s, 0 retransmissions (0.00%), 0 transmission errors (0.00%), 2 clients, 1 pending) 20:36:17.385: client-1 : failed after 0 CON-requests! Handshake flight 5 failed! Stopped by timeout after 2 retransmissions! Wrong PSK identity or secret? [0002]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0003]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0004]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0005]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0006]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0007]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0008]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0009]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0010]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0011]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0012]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0013]: 11 requests (0 reqs/s, 0 retransmissions, 0 transmission errors, 1 clients) [0014]: 11 requests, stale (1 clients, 0 pending) 2 benchmark clients stopped. 1 clients with 0 requests. 1 clients with 10 requests. 20:38:24.498: uptime: 10002 ms, 16 processors 20:38:24.499: cpu-time: 140 ms (per-processor: 8 ms, load: 0%) 20:38:24.502: gc: 3 ms, 1 calls 20:38:24.503: 10 requests sent, 20 expected 20:38:24.503: 10 requests in 10002 ms, 1 reqs/s 20:38:24.503: Stale at 10 messages (50%) 20:38:24.505: connects : #: 1, avg.: 34.00 ms, 95%: 34 ms, 99%: 34 ms, 99.9%: 34 ms, max.: 34 ms 20:38:24.505: success-responses : #: 9, avg.: 1.56 ms, 95%: 2 ms, 99%: 2 ms, 99.9%: 2 ms, max.: 2 ms 20:38:24.505: errors-responses : no values available! 20:38:24.506: single-blocks : #: 9, avg.: 1.44 ms, 95%: 2 ms, 99%: 2 ms, 99.9%: 2 ms, max.: 2 ms 20:38:24.507: coaps endpoint statistic: coaps send statistic: coaps requests : 10 ( 0 overall). coaps responses : 0 ( 0 overall). coaps acks : 0 ( 0 overall). coaps rejects : 0 ( 0 overall). coaps request retransmissions : 0 ( 0 overall). coaps response retransmissions : 0 ( 0 overall). coaps errors : 1 ( 0 overall). coaps receive statistic: coaps requests : 0 ( 0 overall). coaps responses : 10 ( 0 overall). coaps acks : 0 ( 0 overall). coaps rejects : 0 ( 0 overall). coaps duplicate requests : 0 ( 0 overall). coaps duplicate responses : 0 ( 0 overall). coaps offloaded : 0 ( 0 overall). coaps ignored : 0 ( 0 overall).

I can't lay my finger on the proper configuration to allow for concurrent DTLS requests. Here is also my server's properties: COAP.ACK_INIT_RANDOM=1.5 COAP.ACK_TIMEOUT=2[s] COAP.ACK_TIMEOUT_SCALE=2.0 COAP.BLOCKWISE_ENTITY_TOO_LARGE_AUTO_FAILOVER=true COAP.BLOCKWISE_REUSE_TOKEN=false COAP.BLOCKWISE_STATUS_INTERVAL=5[s] COAP.BLOCKWISE_STATUS_LIFETIME=5[min] COAP.BLOCKWISE_STRICT_BLOCK2_OPTION=false COAP.COAP_PORT=5683 COAP.COAP_SECURE_PORT=5684 COAP.CONGESTION_CONTROL_ALGORITHM=NULL COAP.CROP_ROTATION_PERIOD=247[s] COAP.DEDUPLICATOR=PEERS_MARK_AND_SWEEP COAP.DEDUPLICATOR_AUTO_REPLACE=true COAP.EXCHANGE_LIFETIME=247[s] COAP.LEISURE=5[s] COAP.MARK_AND_SWEEP_INTERVAL=10[s] COAP.MAX_ACK_TIMEOUT=1[min] COAP.MAX_ACTIVE_PEERS=150000 COAP.MAX_LATENCY=100[s] COAP.MAX_MESSAGE_SIZE=1024 COAP.MAX_PEER_INACTIVITY_PERIOD=10[min] COAP.MAX_RESOURCE_BODY_SIZE=8192 COAP.MAX_RETRANSMIT=4 COAP.MAX_SERVER_OBSERVES=50000 COAP.MAX_SERVER_RESPONSE_DELAY=250[s] COAP.MAX_TRANSMIT_WAIT=93[s] COAP.MID_TACKER=GROUPED COAP.MID_TRACKER_GROUPS=16 COAP.MULTICAST_BASE_MID=65000 COAP.NON_LIFETIME=145[s] COAP.NOTIFICATION_CHECK_INTERVAL=2[min] COAP.NOTIFICATION_CHECK_INTERVAL_COUNT=100 COAP.NOTIFICATION_REREGISTRATION_BACKOFF=2[s] COAP.NSTART=100 COAP.PEERS_MARK_AND_SWEEP_MESSAGES=16 COAP.PREFERRED_BLOCK_SIZE=512 COAP.PROBING_RATE=1.0 COAP.PROTOCOL_STAGE_THREAD_COUNT=16 COAP.RESPONSE_MATCHING=STRICT COAP.STRICT_EMPTY_MESSAGE_FORMAT=true COAP.TCP_NUMBER_OF_BULK_BLOCKS=4 COAP.TOKEN_SIZE_LIMIT=8 COAP.USE_MESSAGE_OFFLOADING=true COAP.USE_RANDOM_MID_START=true DTLS.ADDITIONAL_ECC_TIMEOUT=0[ms] DTLS.AUTO_HANDSHAKE_TIMEOUT= DTLS.CERTIFICATE_KEY_ALGORITHMS= DTLS.CERTIFICATE_TYPES=RAW_PUBLIC_KEY, X_509 DTLS.CIPHER_SUITES= DTLS.CLIENT_AUTHENTICATION_MODE=NEEDED DTLS.CONNECTION_ID_LENGTH= DTLS.CONNECTION_ID_NODE_ID= DTLS.CONNECTOR_THREAD_COUNT=16 DTLS.CURVES= DTLS.DEFAULT_HANDSHAKE_MODE= DTLS.EXTENDED_MASTER_SECRET_MODE=ENABLED DTLS.MAC_ERROR_FILTER_QUIET_TIME=0[ms] DTLS.MAC_ERROR_FILTER_THRESHOLD=0 DTLS.MAX_CONNECTIONS=150000 DTLS.MAX_DEFERRED_INBOUND_RECORDS=8192 DTLS.MAX_DEFERRED_OUTBOUND_APPLICATION_MESSAGES=10 DTLS.MAX_FRAGMENTED_HANDSHAKE_MESSAGE_LENGTH=8192 DTLS.MAX_FRAGMENT_SIZE= DTLS.MAX_PENDING_HANDSHAKE_RESULT_JOBS=5000 DTLS.MAX_PENDING_INBOUND_JOBS=50000 DTLS.MAX_PENDING_OUTBOUND_JOBS=50000 DTLS.MAX_RETRANSMISSIONS=4 DTLS.MAX_RETRANSMISSION_TIMEOUT=1[min] DTLS.MAX_TRANSMISSION_UNIT= DTLS.MAX_TRANSMISSION_UNIT_LIMIT=1500 DTLS.PRESELECTED_CIPHER_SUITES=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256, TLS_ECDHE_PSK_WITH_AES_256_GCM_SHA378, TLS_ECDHE_PSK_WITH_AES_128_CCM_SHA256, TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA256, TLS_PSK_WITH_AES_128_GCM_SHA256, TLS_PSK_WITH_AES_256_GCM_SHA378, TLS_PSK_WITH_AES_128_CCM, TLS_PSK_WITH_AES_256_CCM, TLS_PSK_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8, TLS_ECDHE_PSK_WITH_AES_128_CCM_8_SHA256, TLS_PSK_WITH_AES_128_CCM_8, TLS_PSK_WITH_AES_256_CCM_8 DTLS.READ_WRITE_LOCK_CONNECTION_STORE=true DTLS.RECEIVER_THREAD_COUNT=2 DTLS.RECEIVE_BUFFER_SIZE= DTLS.RECOMMENDED_CIPHER_SUITES_ONLY=false DTLS.RECOMMENDED_CURVES_ONLY=true DTLS.RECOMMENDED_SIGNATURE_AND_HASH_ALGORITHMS_ONLY=true DTLS.RECORD_SIZE_LIMIT= DTLS.REMOVE_STALE_DOUBLE_PRINCIPALS=false DTLS.RETRANSMISSION_BACKOFF= DTLS.RETRANSMISSION_INIT_RANDOM=1.0 DTLS.RETRANSMISSION_TIMEOUT=2[s] DTLS.RETRANSMISSION_TIMEOUT_SCALE=2.0 DTLS.ROLE=SERVER_ONLY DTLS.SEND_BUFFER_SIZE= DTLS.SERVER_USE_SESSION_ID=true DTLS.SESSION_TIMEOUT=1[d] DTLS.SIGNATURE_AND_HASH_ALGORITHMS= DTLS.STALE_CONNECTION_THRESHOLD=30[min] DTLS.SUPPORT_DEPRECATED_CID=false DTLS.SUPPORT_KEY_MATERIAL_EXPORT=false DTLS.TRUNCATE_CERTIFICATE_PATH_FOR_VALIDATION=true DTLS.TRUNCATE_CLIENT_CERTIFICATE_PATH=true DTLS.UPDATE_ADDRESS_USING_CID_ON_NEWER_RECORDS=true DTLS.USE_DEFAULT_RECORD_FILTER=true DTLS.USE_DEPRECATED_CID= DTLS.USE_DISABLED_WINDOW_FOR_ANTI_REPLAY_FILTER=0 DTLS.USE_EARLY_STOP_RETRANSMISSION=true DTLS.USE_HELLO_VERIFY_REQUEST=true DTLS.USE_HELLO_VERIFY_REQUEST_FOR_PSK=true DTLS.USE_MULTI_RECORD_MESSAGES= DTLS.USE_NEWER_FILTER=false DTLS.USE_SERVER_NAME_INDICATION=false DTLS.USE_USE_ANTI_REPLAY_FILTER=true DTLS.VERIFY_PEERS_ON_RESUMPTION_THRESHOLD=30 DTLS.VERIFY_SERVER_CERTIFICATES_SUBJECT=true SYS.HEALTH_STATUS_INTERVAL=0[ms]

Thank you very much for your help, I appreciate it !

[boaks]

20:36:17.385: client-1 : failed after 0 CON-requests! Handshake flight 5 failed! Stopped by timeout after 2 retransmissions! Wrong PSK identity or secret?

Ok, the docu is missing ;-(.

PSK client setup

if (config.pskStore != null) {
    int pskIndex = (pskOffset + index) % config.pskStore.size();
    identity = config.pskStore.getIdentity(pskIndex);
    secret = config.pskStore.getSecret(pskIndex);
} else if (index == 0) {
    identity = config.identity;
    secret = config.getPskSecretKey();
} else {
    random.nextBytes(id);
    identity = ConnectorConfig.PSK_IDENTITY_PREFIX + StringUtil.byteArray2Hex(id);
    secret = config.getPskSecretKey();
}

If you don't provide a "--psk-store", only the first client uses the provided identity. The others use "cali.${random}".

The idea behind is then a wildcard PSK, but just for testing.

[Jalal-Assaly]

Thank you very much for your help, I sincerely appreciate it !

I created the psk-store file and all is working great.

Again thank you for the support. I wish you all the best

[boaks]

You're welcome!