[ml release] lightning user test consistently failing

[xwjiang2010]

What happened + What you expected to happen

https://buildkite.com/ray-project/release-tests-branch/builds/752#0181d064-d4a0-4642-82cd-6e16ce855f75

Versions / Dependencies

master

Reproduction script

NA

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[xwjiang2010]

Seems to have something to do with tensorboard/summary/writer/event_file_writer.py

[xwjiang2010]

problematic one:

(base) ray:~/oss-release-tests% pip freeze
absl-py==1.1.0
accelerate==0.5.1
adal==1.2.7
aioboto3==8.3.0
aiobotocore==1.2.2
aiohttp==3.8.1
aiohttp-cors==0.7.0
aiohttp-middlewares==1.2.1
aioitertools==0.8.0
aioredis==1.3.1
aiorwlock==1.3.0
aiosignal==1.2.0
ale-py==0.7.5
alembic==1.4.1
anyio==3.6.1
anyscale @ file:///home/ray/anyscale-0.5.31.tar.gz
anyscale-node-provider @ file:///home/ray/anyscale_node_provider-0.0.1.tar.gz
applicationinsights==0.11.10
argcomplete==1.12.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asgiref==3.5.2
astunparse==1.6.3
async-timeout==4.0.2
asynctest==0.13.0
attrs==21.4.0
autocfg==0.0.8
autogluon.core==0.2.0
autograd==1.4
autopage==0.5.1
AutoROM==0.4.2
AutoROM.accept-rom-license==0.4.2
awscli==1.25.6
ax-platform==0.2.4
azure-cli-core==2.29.1
azure-cli-telemetry==1.0.6
azure-common==1.1.28
azure-core==1.24.2
azure-identity==1.7.0
azure-mgmt-compute==23.1.0
azure-mgmt-core==1.2.2
azure-mgmt-network==19.0.0
azure-mgmt-resource==20.0.0
backcall==0.2.0
backoff==1.10.0
bayesian-optimization==1.2.0
bcrypt==3.2.2
beautifulsoup4==4.11.1
bleach==5.0.1
blessed==1.19.1
bokeh==2.3.3
boto3==1.16.52
botocore==1.19.52
botorch==0.6.2
brotlipy==0.7.0
cached-property==1.5.2
cachetools==5.2.0
catboost==1.0.6
certifi==2022.6.15
cffi @ file:///opt/conda/conda-bld/cffi_1642701102775/work
chardet==4.0.0
charset-normalizer==2.1.0
chex==0.1.3
clang==5.0
click==8.0.4
cliff==3.10.1
cloudpickle==2.1.0
cma==2.7.0
cmaes==0.8.2
cmd2==2.4.1
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
colorful==0.5.4
colorlog==6.6.0
comet-ml==3.23.0
conda==4.13.0
conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/work
conda-pack==0.6.0
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1649105789509/work
configobj==5.0.6
ConfigSpace==0.4.18
cryptography @ file:///tmp/build/80754af9/cryptography_1652083456434/work
cycler==0.11.0
Cython==0.29.26
dask==2022.2.0
databricks-cli==0.17.0
datasets==2.0.0
debugpy==1.6.0
decorator==5.1.1
decord==0.6.0
defusedxml==0.7.1
dill==0.3.3
distlib==0.3.4
distributed==2022.2.0
dm-tree==0.1.7
docker==5.0.3
docker-pycreds==0.4.0
docutils==0.16
dopamine-rl==4.0.5
dragonfly-opt==0.1.6
dulwich==0.20.44
entrypoints==0.4
etils==0.6.0
everett==3.0.0
expiringdict==1.2.2
fastapi==0.78.0
fastjsonschema==2.15.3
filelock==3.7.1
FLAML==0.9.7
Flask==2.1.2
Flask-Cors==3.0.10
flatbuffers==1.12
flax==0.4.1
fonttools==4.33.3
freezegun==1.1.0
frozenlist==1.3.0
fsspec==2022.5.0
future==0.18.2
gast==0.4.0
gin-config==0.5.0
gitdb==4.0.9
GitPython==3.1.27
gluoncv==0.10.1.post0
google-api-core==2.8.2
google-api-python-client==1.7.8
google-auth==2.9.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-core==2.3.1
google-cloud-storage==2.4.0
google-crc32c==1.3.0
google-oauth==1.0.1
google-pasta==0.2.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.3
gpustat==1.0.0rc1
GPy==1.10.0
gpytorch==1.6.0
graphviz==0.8.4
greenlet==1.1.2
grpcio==1.43.0
grpcio-tools==1.47.0
gunicorn==20.1.0
gym==0.23.1
gym-notices==0.0.7
h11==0.13.0
h5py==3.1.0
halo==0.0.31
HeapDict==1.0.1
HEBO==0.3.2
higher==0.2.1
hiredis==2.0.0
hpbandster==0.7.4
httplib2==0.20.4
huggingface-hub==0.8.1
humanfriendly==9.2
hyperopt==0.2.5
idna==3.3
imageio==2.19.3
imageio-ffmpeg==0.4.5
importlib-metadata==4.12.0
importlib-resources==5.8.0
iniconfig==1.1.1
ipykernel==6.15.0
ipython==7.34.0
ipython-genutils==0.2.0
ipywidgets==7.7.1
iso8601==1.0.2
isodate==0.6.1
itsdangerous==2.1.2
jax==0.3.14
jaxlib==0.3.14
jedi==0.18.1
Jinja2==3.1.2
jmespath==0.10.0
joblib==1.1.0
json5==0.9.8
jsonpatch==1.32
jsonpointer==2.3
jsonschema==4.6.1
jupyter==1.0.0
jupyter-client==7.3.4
jupyter-console==6.4.4
jupyter-core==4.10.0
jupyterlab==2.2.5
jupyterlab-pygments==0.2.2
jupyterlab-server==1.2.0
jupyterlab-widgets==1.1.1
kaggle-environments==1.7.11
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.3
knack==0.8.2
kopf==1.35.5
kubernetes==17.17.0
lightgbm==3.2.1
lightgbm-ray==0.1.4
lightning-bolts==0.5.0
locket==1.0.0
log-symbols==0.0.14
lz4==4.0.1
Mako==1.2.1
Markdown==3.3.7
MarkupSafe==2.1.1
matplotlib==3.4.3
matplotlib-inline==0.1.3
mistune==0.8.4
mlagents-envs==0.28.0
mlflow==1.21.0
modin==0.12.1
moto==2.3.1
msal==1.18.0
msal-extensions==0.3.1
msgpack==1.0.4
msrest==0.7.1
msrestazure==0.6.4
multidict==6.0.2
multipledispatch==0.6.0
multiprocess==0.70.11.1
mxnet==1.8.0.post0
nbclient==0.6.6
nbconvert==6.5.0
nbformat==5.4.0
nest-asyncio==1.5.5
netifaces==0.11.0
networkx==2.6.3
nevergrad==0.4.3.post7
notebook==6.0.3
numexpr==2.8.3
numpy==1.21.6
nvidia-ml-py==11.495.46
nvidia-ml-py3==7.352.0
oauthlib==3.2.0
onnx==1.9.0
onnxruntime==1.9.0
open-spiel==1.0.2
opencensus==0.9.0
opencensus-context==0.1.2
opencv-python==3.4.18.65
opentelemetry-api==1.1.0
opentelemetry-exporter-otlp==1.1.0
opentelemetry-exporter-otlp-proto-grpc==1.1.0
opentelemetry-proto==1.1.0
opentelemetry-sdk==1.1.0
opentelemetry-semantic-conventions==0.20b0
opt-einsum==3.3.0
optax==0.0.9
optuna==2.10.0
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
paramiko==2.11.0
paramz==0.9.5
parso==0.8.3
partd==1.2.0
pathspec==0.9.0
pathtools==0.1.2
patsy==0.5.2
pbr==5.9.0
PettingZoo==1.15.0
pexpect==4.8.0
pickle5==0.0.12
pickleshare==0.7.5
Pillow==9.2.0
pkginfo==1.8.3
platformdirs==2.5.2
plotly==5.9.0
pluggy==1.0.0
portalocker==1.7.1
prettytable==3.3.0
prometheus-client==0.13.1
prometheus-flask-exporter==0.20.2
promise==2.3
prompt-toolkit==3.0.30
protobuf==3.19.4
psutil==5.9.1
ptyprocess==0.7.0
py==1.11.0
py-spy==0.3.12
py4j==0.10.9.5
pyaml==21.10.1
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybullet==3.2.0
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydantic==1.9.1
pyDeprecate==0.3.1
pygame==2.1.2
pyglet==1.5.15
Pygments==2.12.0
PyJWT==2.4.0
pymars==0.9.0
pymoo==0.5.0
pymunk==6.0.0
PyNaCl==1.5.0
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
pyparsing==3.0.9
pyperclip==1.8.2
pypng==0.0.21
Pyro4==4.82
pyrsistent==0.18.1
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
pyspark==3.3.0
pytest==7.1.2
pytest-remotedata==0.3.2
pytest-repeat==0.9.1
python-dateutil==2.8.2
python-editor==1.0.4
python-json-logger==2.0.2
pytorch-lightning==1.5.10
pytz==2022.1
PyWavelets==1.3.0
PyYAML==6.0
pyzmq==23.2.0
qtconsole==5.3.1
QtPy==2.1.0
querystring-parser==1.2.4
ray @ https://s3-us-west-2.amazonaws.com/ray-wheels/master/88a219c7f29f817850f802da9f3b790ad1573e1a/ray-3.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl
ray-lightning==0.2.0
raydp-nightly==2022.6.30.dev1
recsim==0.2.4
regex==2022.6.2
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
responses==0.13.4
rsa==4.8
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016701961/work
s3fs==2021.7.0
s3transfer==0.3.7
scikit-image==0.19.3
scikit-learn==0.24.2
scikit-optimize==0.8.1
scipy==1.6.3
semantic-version==2.10.0
Send2Trash==1.8.0
sentencepiece==0.1.96
sentry-sdk==1.6.0
serpent==1.41
setproctitle==1.2.3
shared-memory38==0.1.2
shortuuid==1.0.1
sigopt==7.5.0
six==1.16.0
smart-open==6.0.0
smmap==5.0.0
sniffio==1.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
spinners==0.0.24
SQLAlchemy==1.4.39
sqlparse==0.4.2
starlette==0.16.0
statsmodels==0.13.2
stevedore==3.5.0
SuperSuit==3.3.3
tabulate==0.8.10
tblib==1.7.0
tenacity==8.0.1
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.4.1
tensorflow==2.6.2
tensorflow-estimator==2.6.0
tensorflow-probability==0.14.1
termcolor==1.1.0
terminado==0.10.1
tf-slim==1.1.0
tf2onnx==1.8.5
threadpoolctl==3.1.0
tifffile==2021.11.2
timm==0.4.5
tinycss2==1.1.1
tokenizers==0.12.1
tomli==2.0.1
toolz==0.11.2
torch==1.9.0
torch-geometric==2.0.4
torch-scatter==2.0.9
torch-sparse==0.6.12
torchmetrics==0.9.2
torchvision==0.10.0+cu111
tornado==6.2
tqdm==4.64.0
traitlets==5.3.0
transformers==4.19.1
tune-sklearn==0.4.1
typeguard==2.13.3
typing_extensions==4.3.0
uritemplate==3.0.1
urllib3==1.26.9
uvicorn==0.16.0
uvloop==0.16.0
virtualenv==20.15.1
wandb==0.12.15
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.3
Werkzeug==2.1.2
widgetsnbextension==3.6.1
wrapt==1.12.1
wurlitzer==3.0.2
xgboost==1.3.3
xgboost-ray==0.1.9
xmltodict==0.13.0
xxhash==3.0.0
yacs==0.1.8
yarl==1.7.2
zict==2.2.0
zipp==3.8.0
zoopt==0.4.1

working one:

(base) ray:~/workspace-project-xiaowei-oncall% pip freeze
absl-py==0.15.0
accelerate==0.5.1
adal==1.2.7
aioboto3==8.3.0
aiobotocore==1.2.2
aiohttp==3.7.4.post0
aiohttp-cors==0.7.0
aiohttp-middlewares==1.2.1
aioitertools==0.8.0
aioredis==1.3.1
aiorwlock==1.3.0
aiosignal==1.2.0
ale-py==0.7.5
alembic==1.4.1
anyio==3.6.1
anyscale @ file:///home/ray/anyscale-0.5.31.tar.gz
anyscale-node-provider @ file:///home/ray/anyscale_node_provider-0.0.1.tar.gz
applicationinsights==0.11.10
argcomplete==1.12.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asgiref==3.5.2
astunparse==1.6.3
async-timeout==3.0.1
asynctest==0.13.0
attrs==21.4.0
autocfg==0.0.8
autogluon.core==0.2.0
autograd==1.4
autopage==0.5.1
AutoROM==0.4.2
AutoROM.accept-rom-license==0.4.2
awscli==1.25.6
ax-platform==0.2.4
azure-cli-core==2.29.1
azure-cli-telemetry==1.0.6
azure-common==1.1.28
azure-core==1.24.2
azure-identity==1.7.0
azure-mgmt-compute==23.1.0
azure-mgmt-core==1.2.2
azure-mgmt-network==19.0.0
azure-mgmt-resource==20.0.0
backcall==0.2.0
backoff==1.10.0
bayesian-optimization==1.2.0
bcrypt==3.2.2
beautifulsoup4==4.11.1
bleach==5.0.1
blessed==1.19.1
bokeh==2.3.3
boto3==1.16.52
botocore==1.19.52
botorch==0.6.2
brotlipy==0.7.0
cached-property==1.5.2
cachetools==4.2.4
catboost==1.0.6
certifi @ file:///opt/conda/conda-bld/certifi_1655968806487/work/certifi
cffi @ file:///opt/conda/conda-bld/cffi_1642701102775/work
chardet==4.0.0
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
chex==0.1.3
clang==5.0
click==8.0.4
cliff==3.10.1
cloudpickle==2.1.0
cma==2.7.0
cmaes==0.8.2
cmd2==2.4.1
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
colorful==0.5.4
colorlog==6.6.0
comet-ml==3.23.0
conda==4.13.0
conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/work
conda-pack==0.6.0
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1649105789509/work
configobj==5.0.6
ConfigSpace==0.4.18
cryptography @ file:///tmp/build/80754af9/cryptography_1652083456434/work
cycler==0.11.0
Cython==0.29.26
dask==2022.2.0
databricks-cli==0.17.0
datasets==2.0.0
debugpy==1.6.0
decorator==5.1.1
decord==0.6.0
defusedxml==0.7.1
dill==0.3.3
distlib==0.3.4
distributed==2022.2.0
dm-tree==0.1.7
docker==5.0.3
docker-pycreds==0.4.0
docutils==0.16
dopamine-rl==4.0.5
dragonfly-opt==0.1.6
dulwich==0.20.44
entrypoints==0.4
etils==0.6.0
everett==3.0.0
expiringdict==1.2.2
fastapi==0.78.0
fastjsonschema==2.15.3
filelock==3.7.1
FLAML==0.9.7
Flask==2.1.2
Flask-Cors==3.0.10
flatbuffers==1.12
flax==0.4.1
fonttools==4.34.0
freezegun==1.1.0
frozenlist==1.3.0
fsspec==2021.7.0
future==0.18.2
gast==0.4.0
gin-config==0.5.0
gitdb==4.0.9
GitPython==3.1.27
gluoncv==0.10.1.post0
google-api-core==2.8.2
google-api-python-client==1.7.8
google-auth==1.35.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-core==2.3.1
google-cloud-storage==2.4.0
google-crc32c==1.3.0
google-oauth==1.0.1
google-pasta==0.2.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.3
gpustat==1.0.0rc1
GPy==1.10.0
gpytorch==1.6.0
graphviz==0.8.4
greenlet==1.1.2
grpcio==1.43.0
grpcio-tools==1.47.0
gunicorn==20.1.0
gym==0.23.1
gym-notices==0.0.7
h11==0.13.0
h5py==3.1.0
halo==0.0.31
HeapDict==1.0.1
HEBO==0.3.2
higher==0.2.1
hiredis==2.0.0
hpbandster==0.7.4
httplib2==0.20.4
huggingface-hub==0.8.1
humanfriendly==9.2
hyperopt==0.2.5
idna==2.10
imageio==2.19.3
imageio-ffmpeg==0.4.5
importlib-metadata==4.12.0
importlib-resources==5.8.0
iniconfig==1.1.1
ipykernel==6.15.0
ipython==7.34.0
ipython-genutils==0.2.0
ipywidgets==7.7.1
iso8601==1.0.2
isodate==0.6.1
itsdangerous==2.1.2
jax==0.3.14
jaxlib==0.3.14
jedi==0.18.1
Jinja2==3.1.2
jmespath==0.10.0
joblib==1.1.0
json5==0.9.8
jsonpatch==1.32
jsonpointer==2.3
jsonschema==4.6.1
jupyter==1.0.0
jupyter-client==7.3.4
jupyter-console==6.4.4
jupyter-core==4.10.0
jupyterlab==2.2.5
jupyterlab-pygments==0.2.2
jupyterlab-server==1.2.0
jupyterlab-widgets==1.1.1
kaggle-environments==1.7.11
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.3
knack==0.8.2
kopf==1.35.5
kubernetes==17.17.0
lightgbm==3.2.1
lightgbm-ray==0.1.4
lightning-bolts==0.4.0
locket==1.0.0
log-symbols==0.0.14
lz4==4.0.1
Mako==1.2.1
Markdown==3.3.7
MarkupSafe==2.1.1
matplotlib==3.4.3
matplotlib-inline==0.1.3
mistune==0.8.4
mlagents-envs==0.28.0
mlflow==1.21.0
modin==0.12.1
moto==2.3.1
msal==1.18.0
msal-extensions==0.3.1
msgpack==1.0.4
msrest==0.7.1
msrestazure==0.6.4
multidict==6.0.2
multipledispatch==0.6.0
multiprocess==0.70.11.1
mxnet==1.8.0.post0
nbclient==0.6.6
nbconvert==6.5.0
nbformat==5.4.0
nest-asyncio==1.5.5
netifaces==0.11.0
networkx==2.6.3
nevergrad==0.4.3.post7
notebook==6.0.3
numexpr==2.8.3
numpy==1.19.5
nvidia-ml-py==11.495.46
nvidia-ml-py3==7.352.0
oauthlib==3.2.0
onnx==1.9.0
onnxruntime==1.9.0
open-spiel==1.0.2
opencensus==0.10.0
opencensus-context==0.1.2
opencv-python==3.4.18.65
opentelemetry-api==1.1.0
opentelemetry-exporter-otlp==1.1.0
opentelemetry-exporter-otlp-proto-grpc==1.1.0
opentelemetry-proto==1.1.0
opentelemetry-sdk==1.1.0
opentelemetry-semantic-conventions==0.20b0
opt-einsum==3.3.0
optax==0.0.9
optuna==2.10.0
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
paramiko==2.11.0
paramz==0.9.5
parso==0.8.3
partd==1.2.0
pathspec==0.9.0
pathtools==0.1.2
patsy==0.5.2
pbr==5.9.0
PettingZoo==1.15.0
pexpect==4.8.0
pickle5==0.0.12
pickleshare==0.7.5
Pillow==9.2.0
pkginfo==1.8.3
platformdirs==2.5.2
plotly==5.9.0
pluggy==1.0.0
portalocker==1.7.1
prettytable==3.3.0
prometheus-client==0.13.1
prometheus-flask-exporter==0.20.2
promise==2.3
prompt-toolkit==3.0.30
protobuf==3.20.1
psutil==5.9.1
ptyprocess==0.7.0
py==1.11.0
py-spy==0.3.12
py4j==0.10.9.5
pyaml==21.10.1
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybullet==3.2.0
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydantic==1.9.1
pyDeprecate==0.3.1
pygame==2.1.2
pyglet==1.5.15
Pygments==2.12.0
PyJWT==2.4.0
pymars==0.9.0
pymoo==0.5.0
pymunk==6.0.0
PyNaCl==1.5.0
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
pyparsing==3.0.9
pyperclip==1.8.2
pypng==0.0.21
Pyro4==4.82
pyrsistent==0.18.1
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
pyspark==3.3.0
pytest==7.1.2
pytest-remotedata==0.3.2
pytest-repeat==0.9.1
python-dateutil==2.8.2
python-editor==1.0.4
python-json-logger==2.0.2
pytorch-lightning==1.5.10
pytz==2022.1
PyWavelets==1.3.0
PyYAML==5.4.1
pyzmq==23.2.0
qtconsole==5.3.1
QtPy==2.1.0
querystring-parser==1.2.4
ray @ file:///home/ray/ray-3.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl
ray-lightning==0.2.0
raydp-nightly==2022.6.30.dev1
recsim==0.2.4
regex==2022.6.2
requests==2.25.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
responses==0.13.4
rsa==4.7.2
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016701961/work
s3fs==2021.7.0
s3transfer==0.3.7
scikit-image==0.19.3
scikit-learn==0.24.2
scikit-optimize==0.8.1
scipy==1.6.3
semantic-version==2.10.0
Send2Trash==1.8.0
sentencepiece==0.1.96
sentry-sdk==1.6.0
serpent==1.41
setproctitle==1.2.3
shared-memory38==0.1.2
shortuuid==1.0.1
sigopt==7.5.0
six==1.15.0
smart-open==6.0.0
smmap==5.0.0
sniffio==1.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
spinners==0.0.24
SQLAlchemy==1.4.39
sqlparse==0.4.2
starlette==0.16.0
statsmodels==0.13.2
stevedore==3.5.0
SuperSuit==3.3.3
tabulate==0.8.10
tblib==1.7.0
tenacity==8.0.1
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.4.1
tensorflow==2.6.2
tensorflow-estimator==2.6.0
tensorflow-probability==0.14.1
termcolor==1.1.0
terminado==0.10.1
tf-slim==1.1.0
tf2onnx==1.8.5
threadpoolctl==3.1.0
tifffile==2021.11.2
timm==0.4.5
tinycss2==1.1.1
tokenizers==0.12.1
tomli==2.0.1
toolz==0.11.2
torch==1.9.0+cu111
torch-geometric==2.0.4
torch-scatter==2.0.9
torch-sparse==0.6.12
torchmetrics==0.9.2
torchvision==0.10.0+cu111
tornado==6.2
tqdm @ file:///opt/conda/conda-bld/tqdm_1647339053476/work
traitlets==5.3.0
transformers==4.19.1
tune-sklearn==0.4.1
typeguard==2.13.3
typing-extensions==3.7.4.3
uritemplate==3.0.1
urllib3 @ file:///opt/conda/conda-bld/urllib3_1643638302206/work
uvicorn==0.16.0
uvloop==0.16.0
virtualenv==20.15.1
wandb==0.12.15
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.3
Werkzeug==2.1.2
widgetsnbextension==3.6.1
wrapt==1.12.1
wurlitzer==3.0.2
xgboost==1.3.3
xgboost-ray==0.1.9
xmltodict==0.13.0
xxhash==3.0.0
yacs==0.1.8
yarl==1.7.2
zict==2.2.0
zipp==3.8.0
zoopt==0.4.1