Open ghost opened 2 years ago
bmd3k
commented
2 years ago The request is triggering the logic in werkzeug that parses chunked HTTP requests but the request is presumably not actually chunked encoded.
Which browser are you using?
If you're using Chrome, would it be ok for you could use the "Copy as cURL" option in the network tab of the developer console and paste the contents here? https://developer.chrome.com/blog/replay-a-network-request-in-curl/
ghost
commented
2 years ago Hello @bmd3k, here is the guilty request (sensitive info replaced by xxx):
curl 'https://xxx.notebook.eu-west-1.sagemaker.aws/proxy/6006/experiment/defaultExperimentId/data/plugin/timeseries/timeSeries' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-GB,en;q=0.9,fr;q=0.8,de;q=0.7' \
-H 'Connection: keep-alive' \
-H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryWA6twK652Fzuc5rB' \
-H 'Cookie: _xsrf=xxx; domain=prod; region=eu-west-1; expiryTime=1668196217000; authToken=xxx; JSESSIONID=xxx; redirectURL="https://console.aws.amazon.com/sagemaker/home?region=eu-west-1¬ebookState=xxx%3D%3D#/notebook-instances/openNotebook/xxx"' \
-H 'Origin: https://xxx.notebook.eu-west-1.sagemaker.aws' \
-H 'Referer: https://xxx.notebook.eu-west-1.sagemaker.aws/proxy/6006/?darkMode=true' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36' \
-H 'X-TensorBoard-Feature-Flags: {"enabledColorGroup":true,"enabledColorGroupByRegex":true,"enabledExperimentalPlugins":[],"enabledLinkedTime":false,"enabledCardWidthSetting":true,"enabledScalarDataTable":false,"forceSvg":false,"enableDarkModeOverride":null,"defaultEnableDarkMode":true,"isAutoDarkModeAllowed":true,"inColab":false,"metricsImageSupportEnabled":true,"enableTimeSeriesPromotion":false}' \
-H 'X-XSRF-Protected: 1' \
-H 'sec-ch-ua: "Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "Windows"' \
--data-raw $'------WebKitFormBoundaryWA6twK652Fzuc5rB\r\nContent-Disposition: form-data; name="requests"\r\n\r\n[{"plugin":"scalars","tag":"epoch"}]\r\n------WebKitFormBoundaryWA6twK652Fzuc5rB--\r\n' \
--compressedAnd I can confirm that the server struggle with this WebKitForm:
ValueError: invalid literal for int() with base 16: '------WebKitFormBoundaryWA6twK652Fzuc5rB'EDIT:
I looked at the same package when running locally, and it seems similar. I don't get why it doesn't work on AWS.
I also compared werkzeug version and it's 2.2.2 in both cases, locally and on AWS.
japie1235813
commented
2 years ago Might or might not relevant: in your diagnose shows Werkzeug==2.0.3 not 2.2.2
I'm trying to reproduce but jupyter is running for me. Here's my env versions:
$ pip freeze --all | grep Werkzeug
Werkzeug==2.2.2
$ pip freeze --all | grep jupyter
jupyter-server==1.23.1
jupyter-server-proxy==3.2.1
jupyter_client==7.4.4
jupyter_core==5.0.0
jupyterlab==3.3.2
jupyterlab-pygments==0.2.2
jupyterlab-server==2.11.2
jupyterlab-widgets==3.0.3This is my request:
curl 'http://localhost:8888/proxy/6006/' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Accept-Language: en-US,en;q=0.9,hi;q=0.8,ja;q=0.7' \
-H 'Cache-Control: no-cache' \
-H 'Connection: keep-alive' \
-H 'Cookie: _xsrf=...; username-localhost-8888="2|1:0|10:1668116972|23:username-localhost-8888|44:..."' \
-H 'Pragma: no-cache' \
-H 'Sec-Fetch-Dest: document' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-User: ?1' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' \
-H 'sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "macOS"' \
--compressedI don't have "data-raw"
ghost
commented
1 year ago @japie1235813 : I did a fresh reinstall of my conda on an AWS instance and I can confirm Werkzeug==2.2.2.
The problem is still there.
But, like you, I don't have the issue if I run it locally, it seems tied to AWS, somehow.
mkserge
commented
1 year ago Facing the same issue with tensorboard 2.11.0 and Werkzeug 2.2.2 on AWS Sagemaker.
akshaylive
commented
1 year ago Same issue and versions as above. I wonder if this has more to do with SageMaker than TB.
mayankanand111
commented
1 year ago I am having same OSError: Invalid chunk header, while loading scalers on sagemaker notebook instance. Any fix yet?
tcourat
commented
1 year ago Same issue there.
dimagor
commented
1 year ago Same issue, hoping for traction on this one 🤞
tsdev
commented
1 year ago Same issue here on Tensorboard 2.13.0 and werkzeug 2.3.6 on AWS SageMaker notebook
hadriensablon
commented
1 year ago Same here. TB 2.13.0 and werkzeug 2.3.6
datasith
commented
1 year ago +1 Tensorboard 2.13.0 and werkzeug 2.3.6 on AWS SageMaker notebook
Tootiya
commented
1 year ago +1 I tried different versions of Tensorboard and werkzeug but none worked out.
linzzzzzz
commented
1 year ago +1 Tensorboard 2.14 on AWS SageMaker notebook instance. doesn't work
Environment information (required)
I am running Tensorboard in a Jupyter Lab, on AWS Sagemaker.
Tensorboard is installed via Pytorch.
Tensorboard version: 2.10.1
Diagnostics
Diagnostics output
`````` --- check: autoidentify INFO: diagnose_tensorboard.py version 516a2f9433ba4f9c3a4fdb0f89735870eda054a1 --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=10, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='ip-xxx.eu-west-1.compute.internal', release='5.10.102-99.473.amzn2.x86_64', version='#1 SMP Wed Mar 2 19:14:12 UTC 2022', machine='x86_64') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: True INFO: $VIRTUAL_ENV: None --- check: installed_packages WARNING: no installation among: ['tb-nightly', 'tensorboard', 'tensorflow-tensorboard'] WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview'] WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly'] --- check: tensorboard_python_version Traceback (most recent call last): File "tb_diag.py", line 528, in main suggestions.extend(check()) File "tb_diag.py", line 81, in wrapper result = fn() File "tb_diag.py", line 277, in tensorboard_python_version from tensorboard import version ModuleNotFoundError: No module named 'tensorboard' --- check: tensorflow_python_version Traceback (most recent call last): File "tb_diag.py", line 528, in main suggestions.extend(check()) File "tb_diag.py", line 81, in wrapper result = fn() File "tb_diag.py", line 284, in tensorflow_python_version import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow' --- check: tensorboard_data_server_version INFO: no data server installed --- check: tensorboard_binary_path INFO: which tensorboard: b'/home/ec2-user/SageMaker/envs/a2s-aws/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC =Issue description
I start tensorboard via
tensorboard --logdir trainings/Tensorboard starts correctly.
Then I open the UI via https://xxx.notebook.eu-west-1.sagemaker.aws/proxy/6006/
The UI shows up, the runs are listed, but there is no scalar or hparam.
The scalars plots are there, but empty.
The terminal returns:
This is the same error than described here, 2 years ago, on tensorboard 2.4:
https://stackoverflow.com/questions/66220426/tensorboard-not-working-with-after-upgrading-tensorflow-from-2-1-2-to-2-3-0
I can also confirm that tensorboard 2.3 runs and return scalars, in the same context.