ODF push/pull protocol fails on repos restricted to HTTPS

[sergiimk]

Tracking issue: steps

• [x] https://github.com/kamu-data/kamu-cli/pull/591
• [x] https://github.com/kamu-data/kamu-deploy/pull/77
• [x] Wait for Jupyter image release
• [x] Re-merge, after rollback of changes https://github.com/kamu-data/kamu-deploy/commit/0fc0ee15409a6805341b38a025dae34a33526bc4
  • https://github.com/kamu-data/kamu-deploy/pull/86

Describe the bug

In our new demo environment we restricted HTTP ports and configured load balancers to auto-redirect to HTTPS. This broke smart push/pull protocol.

Steps To Reproduce

1. Log in to https://jupyter.demo.kamu.dev
2. kamu init
3. kamu repo add kamu-node odf+https://node.demo.kamu.dev/
4. kamu search covid - note that search is working (correctly goes over HTTPs)
5. kamu pull kamu-node/kamu/covid19.ontario.case-details

Current Behavior

> Failed to sync covid19.ontario.case-details from kamu-node/kamu/covid19.ontario.case-details:
  0: Internal error
  1: HTTP error: 301 Moved Permanently

In logs we can observe:

Connecting smart pull protocol web socket, pull_url: ws://node.demo.kamu.dev/kamu/covid19.ontario.case-details/pull

While it should likely be wss.

Offending line: https://github.com/kamu-data/kamu-cli/blob/c1c183787d59da2eb428a0ae777c38f4d0f22214/src/adapter/http/src/smart_protocol/ws_tungstenite_client.rs#L481

Expected Behavior

Push/pull works for ODF repos restricted to HTTPS

kamu system info

$ kamu version
appVersion: 0.163.0
buildTimestamp: 2024-03-01T10:47:57.507449661Z
gitDescribe: a77f5d9
gitSha: a77f5d92ec712f1d31fc3a75d1961aa1bc649d4f
gitCommitDate: 2024-03-01
gitBranch: HEAD
rustcSemver: 1.78.0-nightly
rustcChannel: nightly
rustcHostTriple: x86_64-unknown-linux-gnu
rustcCommitSha: 1a648b397dedc98ada3dd3360f6d661ec2436c56
cargoTargetTriple: x86_64-unknown-linux-gnu
cargoFeatures: default,ftp,web_ui
cargoOptLevel: '3'

Logs

No response

Anything else?

Current protocol detection on server side that relies on KAMU_PROTOCOL_SCHEME env var could be improved to use x-forwarded-proto header that is being passed down by some proxies (including AWS ALB).

[sergiimk]

@zaychenko-sergei pointed out that this is likely because KAMU_PROTOCOL_SCHEME env var is not set.

This task is now to:

• Support extraEnv in kamu-api-server helm chart (so we could patch such things faster in future)
• Add dedicated value for this into values.yaml of the chart (done)
• Re-enable SSL redirect in demo/api-server deployment

[sergiimk]

@zaychenko-sergei FYI setting KAMU_PROTOCOL_SCHEME on the server does not solve the issue.

ws:// protocol is chosen on the client side: https://github.com/kamu-data/kamu-cli/blob/c1c183787d59da2eb428a0ae777c38f4d0f22214/src/adapter/http/src/smart_protocol/ws_tungstenite_client.rs#L481

Updated ticket description and my last comment.

[s373r]

@sergiimk , I can't reproduce at the current demo environment

https://jupyter.demo.kamu.dev/user/s373r/terminals/1


Locally as well, btw


[s373r]

Just double-checked after the last deploy (https://github.com/kamu-data/kamu-deploy/pull/85), at https://jupyter.demo.kamu.dev/user/s373r/terminals/1

Works as expected:

(base) jovyan@jupyter-s373r:~$ kamu --version
kamu 0.176.3

(base) jovyan@jupyter-s373r:~$ RUST_LOG=debug kamu pull kamu-node/kamu/covid19.ontario.case-details

  Dataset is up-to-date (kamu-node/kamu/covid19.ontario.case-details > covid19.ontario.case-details)1 dataset(s) up-to-date

(base) jovyan@jupyter-s373r:~$ cat .kamu/run/kamu.log | grep wss

{"v":0,"name":"kamu","msg":"[SYNC - EVENT] Connecting smart pull protocol web socket","level":20,"hostname":"jupyter-s373r","pid":171,"time":"2024-04-19T08:13:13.564728888Z","target":"kamu_adapter_http::smart_protocol::ws_tungstenite_client","line":490,"file":"src/adapter/http/src/smart_protocol/ws_tungstenite_client.rs","access_token":"Some(\"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzM4NCJ9.eyJleHAiOjE3MTM1MzExMjQsImlhdCI6MTcxMzUxMzEyNCwiaXNzIjoiZGV2LmthbXUiLCJzdWIiOiJzMzczciIsImFjY2Vzc19jcmVkZW50aWFscyI6eyJsb2dpbl9tZXRob2QiOiJvYXV0aF9naXRodWIiLCJwcm92aWRlcl9jcmVkZW50aWFsc19qc29uIjoie1wiYWNjZXNzX3Rva2VuXCI6XCJnaG9fanJVWmJmZ1E2bk5NUHhxbndYbUZJZWVrZENwOU5yM3MybjBRXCJ9In19.LW_3lZR9mKCCV2wEOQ5M--sQyFR-F1kf6VfPe0RT06mK0TJdkqM8ZuNJw_i_O-kh\")","pull_url":"wss://node.demo.kamu.dev/kamu/covid19.ontario.case-details/pull","dst":"covid19.ontario.case-details","src":"kamu-node/kamu/covid19.ontario.case-details"}

[s373r]

acceptance testing successful