fschade
commented
9 months ago nats allowDirect: true
k6 run packages/k6-tests/artifacts/koko-platform-000-mixed-ramping-k6.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: packages/k6-tests/artifacts/koko-platform-000-mixed-ramping-k6.js
output: -
scenarios: (100.00%) 8 scenarios, 75 max VUs, 6m30s max duration (incl. graceful stop):
* add_remove_tag_100: Up to 5 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: add_remove_tag_100, gracefulStop: 30s)
* create_remove_group_share_090: Up to 5 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: create_remove_group_share_090, gracefulStop: 30s)
* create_space_080: Up to 5 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: create_space_080, gracefulStop: 30s)
* create_upload_rename_delete_folder_and_file_040: Up to 5 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: create_upload_rename_delete_folder_and_file_040, gracefulStop: 30s)
* download_050: Up to 5 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: download_050, gracefulStop: 30s)
* navigate_file_tree_020: Up to 10 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: navigate_file_tree_020, gracefulStop: 30s)
* sync_client_110: Up to 20 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: sync_client_110, gracefulStop: 30s)
* user_group_search_070: Up to 20 looping VUs for 6m0s over 3 stages (gracefulRampDown: 30s, exec: user_group_search_070, gracefulStop: 30s)
✓ authn -> logonResponse - status
✓ authn -> authorizeResponse - status
✓ authn -> accessTokenResponse - status
✗ client -> search.searchForSharees - status
↳ 98% — ✓ 396 / ✗ 6
✓ client -> role.getMyDrives - status
✗ client -> resource.getResourceProperties - status
↳ 99% — ✓ 3384 / ✗ 11
✗ client -> application.createDrive - status
↳ 95% — ✓ 84 / ✗ 4
✗ client -> resource.createResource - status
↳ 98% — ✓ 90 / ✗ 1
✗ client -> drive.deactivateDrive - status
↳ 95% — ✓ 84 / ✗ 4
✓ client -> tag.getTags - status -- (SKIPPED)
✓ client -> tag.createTag - status -- (SKIPPED)
✗ client -> drive.deleteDrive - status
↳ 95% — ✓ 84 / ✗ 4
✓ client -> tag.addTagToResource - status
✗ client -> share.createShare - status
↳ 84% — ✓ 38 / ✗ 7
✓ client -> tag.removeTagToResource - status
✗ client -> share.deleteShare - status
↳ 82% — ✓ 37 / ✗ 8
✗ client -> resource.deleteResource - status
↳ 98% — ✓ 90 / ✗ 1
✗ client -> resource.downloadResource - status
↳ 96% — ✓ 89 / ✗ 3
✓ client -> resource.uploadResource - status
✓ client -> resource.moveResource - status
checks.........................: 99.03% ✓ 5030 ✗ 49
data_received..................: 1.5 GB 3.8 MB/s
data_sent......................: 832 MB 2.1 MB/s
http_req_blocked...............: avg=221.9µs min=0s med=0s max=83.35ms p(90)=1µs p(95)=1µs
http_req_connecting............: avg=17.75µs min=0s med=0s max=5.19ms p(90)=0s p(95)=0s
http_req_duration..............: avg=152.33ms min=13.29ms med=46.77ms max=10.02s p(90)=163.51ms p(95)=286.28ms
{ expected_response:true }...: avg=108.78ms min=13.29ms med=46.49ms max=8.95s p(90)=158.11ms p(95)=262.55ms
http_req_failed................: 0.98% ✓ 49 ✗ 4936
http_req_receiving.............: avg=20.74ms min=14µs med=55µs max=8.87s p(90)=96µs p(95)=157µs
http_req_sending...............: avg=5.67ms min=20µs med=67µs max=5.16s p(90)=108µs p(95)=148µs
http_req_tls_handshaking.......: avg=190.59µs min=0s med=0s max=81.33ms p(90)=0s p(95)=0s
http_req_waiting...............: avg=125.92ms min=13.18ms med=46.54ms max=10.02s p(90)=157.26ms p(95)=265.42ms
http_reqs......................: 4985 12.783076/s
iteration_duration.............: avg=5.9s min=1.01s med=1.05s max=44.25s p(90)=15.24s p(95)=17.28s
iterations.....................: 3900 10.000802/s
vus............................: 2 min=0 max=75
vus_max........................: 75 min=33 max=75
running (6m30.0s), 00/75 VUs, 3900 complete and 3 interrupted iterations
add_remove_tag_100 ✓ [======================================] 0/5 VUs 6m0s
create_remove_group_share_090 ✓ [======================================] 1/5 VUs 6m0s
create_space_080 ✓ [======================================] 0/5 VUs 6m0s
create_upload_rename_delete... ✓ [======================================] 0/5 VUs 6m0s
download_050 ✓ [======================================] 0/5 VUs 6m0s
navigate_file_tree_020 ✓ [======================================] 00/10 VUs 6m0s
sync_client_110 ✓ [======================================] 00/20 VUs 6m0s
user_group_search_070 ✓ [======================================] 00/20 VUs 6m0s
butonic
micbar
wkloucek
Nats jetstream does not guarantee read-after-write consistency OOTB: https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-8.md#consistency-guarantees
When updating filemetadata we lock the node, write to disk, write the cache (to NATS) and then unlock the file. Until NATS has propagated the value to all nodes other requests might read the old metadata.
oCIS web does a propfind after uploading a file. If the requests hits a different nats cluster node the default consistency guarantees might return old metadata when propagation is slow, eg. due to load.
In a single node deployment we won't see a problem, but in a cluster we might.
There are two scenarios:
Creating new files
When file metadata is not present in the cache we will go to tho storage. No inconsistency problem in this case.
Updating existing files
When updating file metadata of an existing file reading from a different NATS node might return old data: fileid (ok), mtime/etag (no change, search won't index), grants?
Next steps
Build a minimal test scenario that
Long term solution
Use groupcache like sharding of the filemetadata key space so that requests will always be handled by the same responsible pod.
Related issues
Distributed locks
AFAICT NATS does not support distributed locks. kraud cloud wrote a blog post in Nov 2023 describing how to build locks based on NATS streams: https://docs.kraudcloud.com/blog/category/tech/ It links to this comment by the nats devs: https://github.com/nats-io/nats-server/discussions/4803#discussioncomment-7612323:
decomposedfs currently locks msgpack metadata using the filesystem. So the locking part is safe for now.
Cache Directory Metadata
Currently, decomposedfs relies on NFS to provide parent / child relationships, which is slow as we have to disable directory caches. Putting directory listings into the cache is subject to the same read-after-write consistency guarantees. Topic for a new ADR.