search

mosaicml / composer

Supercharge Your Model Training
http://docs.mosaicml.com
Apache License 2.0
5.17k stars 420 forks source link

Unable to script model #3584

Open callmekris opened 2 months ago

callmekris commented 2 months ago

Currently running mosaic composer in a databricks environment but experiencing different behavior of torch.jit.script after importing composer depending on the DBR installed.

Code to reproduce

import torch
class MyModule(torch.nn.Module):
    def forward(self, x):
        return x + 10

m = torch.jit.script(MyModule())
print(m)
print(type(m))

and

import torch
from composer.models import ComposerModel
class MyModule(torch.nn.Module):
    def forward(self, x):
        return x + 10

m = torch.jit.script(MyModule())
print(m)
print(type(m))

This is just a minimum example to reproduce, but the same happens when training a composer model and using export_for_inference.

Expected Behavior DBR 14.3

Running with both with or without the composer import prints as expected:

RecursiveScriptModule(original_name=MyModule)
<class 'torch.jit._script.RecursiveScriptModule'>

On DBR >= 15.1

Running with the composer import prints:

MyModule()
<class '__main__.MyModule'>

And without the composer import it scripts correctly as expected.

On both runtimes the libary versions are: torch 2.4.0 mosaicml 0.24.1 mosaicml-cli 0.6.41

I'm not exactly sure how to pinpoint the cause it only occurs on the newer DBRs even 15.4 LTS, but only after the import of composer.

cli99 commented 2 months ago

@callmekris can you also check the transformers version in both envs?

callmekris commented 2 months ago

@cli99 looks like on DBR 14.3 4.36.1 is installed and on DBR 15.4 it's 4.41.2. Tried to downgrade the transformers version of 15.4 but still gave same behavior.

callmekris commented 2 months ago

Here is the full list of installed packages in each environment:

DBR 15.4:

Package                            Version
---------------------------------- --------------
absl-py                            1.0.0
accelerate                         0.31.0
aiohttp                            3.8.5
aiohttp-cors                       0.7.0
aiosignal                          1.2.0
alembic                            1.13.2
anyio                              3.5.0
argcomplete                        3.5.0
argon2-cffi                        21.3.0
argon2-cffi-bindings               21.2.0
arrow                              1.3.0
astor                              0.8.1
asttokens                          2.0.5
astunparse                         1.6.3
async-timeout                      4.0.2
attrs                              22.1.0
audioread                          3.0.1
azure-core                         1.30.2
azure-cosmos                       4.3.1
azure-identity                     1.17.1
azure-storage-blob                 12.19.1
azure-storage-file-datalake        12.14.0
backcall                           0.2.0
backoff                            2.2.1
bcrypt                             3.2.0
beautifulsoup4                     4.12.2
black                              23.3.0
bleach                             4.1.0
blinker                            1.4
blis                               0.7.11
boto3                              1.34.39
botocore                           1.34.39
Brotli                             1.0.9
cachetools                         5.4.0
catalogue                          2.0.10
category-encoders                  2.6.3
certifi                            2023.7.22
cffi                               1.15.1
chardet                            4.0.0
charset-normalizer                 2.0.4
circuitbreaker                     1.4.0
click                              8.0.4
cloudpathlib                       0.16.0
cloudpickle                        2.2.1
cmdstanpy                          1.2.2
colorful                           0.5.6
colorlog                           6.8.2
comm                               0.1.2
confection                         0.1.4
configparser                       5.2.0
contourpy                          1.0.5
coolname                           2.2.0
cryptography                       41.0.3
cycler                             0.11.0
cymem                              2.0.8
Cython                             0.29.32
dacite                             1.8.1
databricks-automl-runtime          0.2.21
databricks-feature-engineering     0.6.0
databricks-sdk                     0.20.0
dataclasses-json                   0.6.7
datasets                           2.19.1
dbl-tempo                          0.1.26
dbus-python                        1.2.18
debugpy                            1.6.7
decorator                          5.1.1
deepspeed                          0.14.4
defusedxml                         0.7.1
Deprecated                         1.2.14
dill                               0.3.6
diskcache                          5.6.3
distlib                            0.3.8
distro                             1.7.0
distro-info                        1.1+ubuntu0.2
dm-tree                            0.1.8
einops                             0.8.0
entrypoints                        0.4
evaluate                           0.4.2
executing                          0.8.3
facets-overview                    1.1.1
Farama-Notifications               0.0.4
fastjsonschema                     2.20.0
fasttext                           0.9.2
filelock                           3.13.4
flash-attn                         2.5.9.post1
Flask                              2.2.5
flatbuffers                        24.3.25
fonttools                          4.25.0
frozenlist                         1.3.3
fsspec                             2023.5.0
future                             0.18.3
gast                               0.4.0
gitdb                              4.0.11
GitPython                          3.1.27
google-api-core                    2.18.0
google-auth                        2.21.0
google-auth-oauthlib               1.0.0
google-cloud-core                  2.4.1
google-cloud-storage               2.10.0
google-crc32c                      1.5.0
google-pasta                       0.2.0
google-resumable-media             2.7.1
googleapis-common-protos           1.63.0
gql                                3.5.0
graphql-core                       3.2.3
greenlet                           2.0.1
grpcio                             1.60.0
grpcio-status                      1.60.0
gunicorn                           20.1.0
gviz-api                           1.10.0
gymnasium                          0.28.1
h11                                0.14.0
h5py                               3.10.0
hjson                              3.1.0
holidays                           0.45
horovod                            0.28.1+db1
htmlmin                            0.1.12
httpcore                           1.0.5
httplib2                           0.20.2
httpx                              0.27.0
huggingface-hub                    0.23.4
idna                               3.4
ImageHash                          4.3.1
imageio                            2.31.1
imbalanced-learn                   0.11.0
importlib-metadata                 6.0.0
importlib_resources                6.4.0
ipyflow-core                       0.0.198
ipykernel                          6.25.1
ipython                            8.15.0
ipython-genutils                   0.2.0
ipywidgets                         7.7.2
isodate                            0.6.1
itsdangerous                       2.0.1
jax-jumpy                          1.0.0
jedi                               0.18.1
jeepney                            0.7.1
Jinja2                             3.1.2
jmespath                           0.10.0
joblib                             1.2.0
joblibspark                        0.5.1
jsonpatch                          1.33
jsonpointer                        3.0.0
jsonschema                         4.17.3
jupyter_client                     7.4.9
jupyter_core                       5.3.0
jupyter-server                     1.23.4
jupyterlab-pygments                0.1.2
kaleido                            0.2.1
keras                              3.2.1
keyring                            23.5.0
kiwisolver                         1.4.4
langchain                          0.1.20
langchain-community                0.0.38
langchain-core                     0.1.52
langchain-text-splitters           0.0.2
langcodes                          3.4.0
langsmith                          0.1.63
language_data                      1.2.0
launchpadlib                       1.10.16
lazr.restfulclient                 0.14.4
lazr.uri                           1.0.6
lazy_loader                        0.2
libclang                           15.0.6.1
librosa                            0.10.1
lightgbm                           4.3.0
lightning-utilities                0.11.6
linkify-it-py                      2.0.0
llvmlite                           0.40.0
lxml                               4.9.2
lz4                                4.3.2
Mako                               1.2.0
marisa-trie                        1.1.1
Markdown                           3.4.1
markdown-it-py                     2.2.0
MarkupSafe                         2.1.1
marshmallow                        3.21.2
matplotlib                         3.7.2
matplotlib-inline                  0.1.6
mdit-py-plugins                    0.3.0
mdurl                              0.1.0
memray                             1.13.4
mistune                            0.8.4
ml-dtypes                          0.3.2
mlflow-skinny                      2.13.1
more-itertools                     8.10.0
mosaicml                           0.24.1
mosaicml-cli                       0.6.41
mosaicml-streaming                 0.7.4
mpmath                             1.3.0
msal                               1.30.0
msal-extensions                    1.2.0
msgpack                            1.0.8
multidict                          6.0.2
multimethod                        1.12
multiprocess                       0.70.14
murmurhash                         1.0.10
mypy-extensions                    0.4.3
namex                              0.0.8
nbclassic                          0.5.5
nbclient                           0.5.13
nbconvert                          6.5.4
nbformat                           5.7.0
nest-asyncio                       1.5.6
networkx                           3.1
ninja                              1.11.1.1
nltk                               3.8.1
notebook                           6.5.4
notebook_shim                      0.2.2
numba                              0.57.1
numpy                              1.23.5
nvidia-cublas-cu12                 12.1.3.1
nvidia-cuda-cupti-cu12             12.1.105
nvidia-cuda-nvrtc-cu12             12.1.105
nvidia-cuda-runtime-cu12           12.1.105
nvidia-cudnn-cu12                  9.1.0.70
nvidia-cufft-cu12                  11.0.2.54
nvidia-curand-cu12                 10.3.2.106
nvidia-cusolver-cu12               11.4.5.107
nvidia-cusparse-cu12               12.1.0.106
nvidia-ml-py                       12.555.43
nvidia-nccl-cu12                   2.20.5
nvidia-nvjitlink-cu12              12.5.82
nvidia-nvtx-cu12                   12.1.105
oauthlib                           3.2.0
oci                                2.126.4
openai                             1.35.3
opencensus                         0.11.4
opencensus-context                 0.1.3
opentelemetry-api                  1.25.0
opentelemetry-sdk                  1.25.0
opentelemetry-semantic-conventions 0.46b0
opt-einsum                         3.3.0
optree                             0.12.1
optuna                             3.6.1
orjson                             3.10.6
packaging                          23.2
pandas                             1.5.3
pandocfilters                      1.5.0
paramiko                           3.4.0
parso                              0.8.3
pathspec                           0.10.3
patsy                              0.5.3
petastorm                          0.12.1
pexpect                            4.8.0
phik                               0.12.4
pickleshare                        0.7.5
pillow                             10.4.0
pip                                23.2.1
platformdirs                       3.10.0
plotly                             5.9.0
pmdarima                           2.0.4
pooch                              1.8.1
portalocker                        2.10.1
preshed                            3.0.9
prometheus-client                  0.14.1
prompt-toolkit                     3.0.36
prophet                            1.1.5
proto-plus                         1.24.0
protobuf                           4.24.1
psutil                             5.9.0
psycopg2                           2.9.3
ptyprocess                         0.7.0
pure-eval                          0.2.2
py-cpuinfo                         8.0.0
py-spy                             0.3.14
pyarrow                            14.0.1
pyarrow-hotfix                     0.6
pyasn1                             0.4.8
pyasn1-modules                     0.2.8
pybind11                           2.13.1
pyccolo                            0.0.52
pycparser                          2.21
pydantic                           1.10.6
Pygments                           2.15.1
PyGObject                          3.42.1
PyJWT                              2.3.0
PyNaCl                             1.5.0
pyodbc                             4.0.38
pyOpenSSL                          23.2.0
pyparsing                          3.0.9
pyrsistent                         0.18.0
pytesseract                        0.3.10
python-apt                         2.4.0+ubuntu3
python-dateutil                    2.8.2
python-editor                      1.0.4
python-lsp-jsonrpc                 1.1.1
python-snappy                      0.6.1
pytorch-ranger                     0.1.1
pytz                               2022.7
PyWavelets                         1.4.1
PyYAML                             6.0
pyzmq                              23.2.0
questionary                        2.0.1
ray                                2.20.0
regex                              2022.7.9
requests                           2.31.0
requests-oauthlib                  1.3.1
rich                               13.7.1
rsa                                4.9
ruamel.yaml                        0.18.6
ruamel.yaml.clib                   0.2.8
s3transfer                         0.10.2
safetensors                        0.4.2
scikit-image                       0.20.0
scikit-learn                       1.3.0
scipy                              1.11.1
seaborn                            0.12.2
SecretStorage                      3.3.1
Send2Trash                         1.8.0
sentence-transformers              2.7.0
sentencepiece                      0.1.99
setuptools                         68.0.0
shap                               0.44.0
simplejson                         3.17.6
six                                1.16.0
slicer                             0.0.7
smart-open                         5.2.1
smmap                              5.0.0
sniffio                            1.2.0
soundfile                          0.12.1
soupsieve                          2.4
soxr                               0.3.7
spacy                              3.7.2
spacy-legacy                       3.0.12
spacy-loggers                      1.0.5
spark-tensorflow-distributor       1.0.0
SQLAlchemy                         1.4.39
sqlparse                           0.4.2
srsly                              2.4.8
ssh-import-id                      5.11
stack-data                         0.2.0
stanio                             0.5.1
statsmodels                        0.14.0
sympy                              1.11.1
tabulate                           0.9.0
tangled-up-in-unicode              0.2.0
tenacity                           8.2.2
tensorboard                        2.16.2
tensorboard-data-server            0.7.2
tensorboard_plugin_profile         2.15.1
tensorboardX                       2.6.2.2
tensorflow                         2.16.1
tensorflow-estimator               2.15.0
tensorflow-io-gcs-filesystem       0.37.1
termcolor                          2.4.0
terminado                          0.17.1
textual                            0.63.3
tf_keras                           2.16.0
thinc                              8.2.3
threadpoolctl                      2.2.0
tifffile                           2021.7.2
tiktoken                           0.5.2
tinycss2                           1.2.1
tokenize-rt                        4.2.1
tokenizers                         0.19.0
torch                              2.4.0
torch-optimizer                    0.3.0
torcheval                          0.0.7
torchmetrics                       1.4.0.post0
torchvision                        0.19.0
tornado                            6.3.2
tqdm                               4.65.0
traitlets                          5.7.1
transformers                       4.41.2
triton                             3.0.0
typeguard                          2.13.3
typer                              0.9.4
types-python-dateutil              2.9.0.20240821
typing_extensions                  4.10.0
typing-inspect                     0.9.0
tzdata                             2022.1
uc-micro-py                        1.0.1
ujson                              5.4.0
unattended-upgrades                0.1
urllib3                            1.26.16
validators                         0.33.0
virtualenv                         20.24.2
visions                            0.7.5
wadllib                            1.3.6
wasabi                             1.1.2
wcwidth                            0.2.5
weasel                             0.3.4
webencodings                       0.5.1
websocket-client                   0.58.0
websockets                         11.0.3
Werkzeug                           2.2.3
wheel                              0.38.4
wordcloud                          1.9.3
wrapt                              1.14.1
xgboost                            2.0.3
xxhash                             3.4.1
yarl                               1.8.1
ydata-profiling                    4.5.1
zipp                               3.11.0
zstd                               1.5.5.1

DBR 14.3:

Package                        Version
------------------------------ --------------
absl-py                        1.0.0
accelerate                     0.25.0
aiohttp                        3.9.1
aiosignal                      1.3.1
alembic                        1.13.2
anyio                          3.5.0
appdirs                        1.4.4
argcomplete                    3.5.0
argon2-cffi                    21.3.0
argon2-cffi-bindings           21.2.0
arrow                          1.3.0
astor                          0.8.1
asttokens                      2.0.5
astunparse                     1.6.3
async-timeout                  4.0.3
attrs                          22.1.0
audioread                      3.0.1
azure-core                     1.29.1
azure-cosmos                   4.3.1
azure-storage-blob             12.19.0
azure-storage-file-datalake    12.14.0
backcall                       0.2.0
backoff                        2.2.1
bcrypt                         3.2.0
beautifulsoup4                 4.11.1
black                          22.6.0
bleach                         4.1.0
blinker                        1.4
blis                           0.7.11
boto3                          1.24.28
botocore                       1.27.96
cachetools                     5.3.2
catalogue                      2.0.10
category-encoders              2.6.3
certifi                        2022.12.7
cffi                           1.15.1
chardet                        4.0.0
charset-normalizer             2.0.4
click                          8.0.4
cloudpathlib                   0.16.0
cloudpickle                    2.0.0
cmake                          3.28.1
cmdstanpy                      1.2.0
colorlog                       6.8.2
comm                           0.1.2
confection                     0.1.4
configparser                   5.2.0
contourpy                      1.0.5
coolname                       2.2.0
cryptography                   39.0.1
cycler                         0.11.0
cymem                          2.0.8
Cython                         0.29.32
dacite                         1.8.1
databricks-automl-runtime      0.2.20
databricks-cli                 0.18.0
databricks-feature-engineering 0.2.1
databricks-sdk                 0.1.6
dataclasses-json               0.6.3
datasets                       2.15.0
dbl-tempo                      0.1.26
dbus-python                    1.2.18
debugpy                        1.6.7
decorator                      5.1.1
deepspeed                      0.12.4
defusedxml                     0.7.1
dill                           0.3.6
diskcache                      5.6.3
distlib                        0.3.7
distro                         1.7.0
distro-info                    1.1+ubuntu0.2
docstring-to-markdown          0.11
einops                         0.7.0
entrypoints                    0.4
evaluate                       0.4.1
executing                      0.8.3
facets-overview                1.1.1
fastjsonschema                 2.19.1
fasttext                       0.9.2
filelock                       3.9.0
flash-attn                     2.3.6
Flask                          2.2.5
flatbuffers                    23.5.26
fonttools                      4.25.0
frozenlist                     1.4.1
fsspec                         2023.6.0
future                         0.18.3
gast                           0.4.0
gitdb                          4.0.11
GitPython                      3.1.27
google-api-core                2.15.0
google-auth                    2.21.0
google-auth-oauthlib           1.0.0
google-cloud-core              2.4.1
google-cloud-storage           2.11.0
google-crc32c                  1.5.0
google-pasta                   0.2.0
google-resumable-media         2.7.0
googleapis-common-protos       1.62.0
gql                            3.5.0
graphql-core                   3.2.3
greenlet                       2.0.1
grpcio                         1.48.2
grpcio-status                  1.48.1
gunicorn                       20.1.0
gviz-api                       1.10.0
h5py                           3.7.0
hjson                          3.1.0
holidays                       0.38
horovod                        0.28.1
htmlmin                        0.1.12
httplib2                       0.20.2
huggingface-hub                0.19.4
idna                           3.4
ImageHash                      4.3.1
imbalanced-learn               0.11.0
importlib_metadata             8.4.0
importlib-resources            6.1.1
ipykernel                      6.25.0
ipython                        8.14.0
ipython-genutils               0.2.0
ipywidgets                     7.7.2
isodate                        0.6.1
itsdangerous                   2.0.1
jedi                           0.18.1
jeepney                        0.7.1
Jinja2                         3.1.2
jmespath                       0.10.0
joblib                         1.2.0
joblibspark                    0.5.1
jsonpatch                      1.33
jsonpointer                    2.4
jsonschema                     4.17.3
jupyter-client                 7.3.4
jupyter_core                   5.2.0
jupyter-server                 1.23.4
jupyterlab-pygments            0.1.2
jupyterlab-widgets             1.0.0
kaleido                        0.2.1
keras                          2.14.0
keyring                        23.5.0
kiwisolver                     1.4.4
langchain                      0.0.348
langchain-core                 0.0.13
langcodes                      3.3.0
langsmith                      0.0.79
launchpadlib                   1.10.16
lazr.restfulclient             0.14.4
lazr.uri                       1.0.6
lazy_loader                    0.3
libclang                       15.0.6.1
librosa                        0.10.1
lightgbm                       4.1.0
lightning-utilities            0.11.6
lit                            17.0.6
llvmlite                       0.39.1
lxml                           4.9.1
Mako                           1.2.0
Markdown                       3.4.1
markdown-it-py                 3.0.0
MarkupSafe                     2.1.1
marshmallow                    3.20.2
matplotlib                     3.7.0
matplotlib-inline              0.1.6
mccabe                         0.7.0
mdurl                          0.1.2
mistune                        0.8.4
ml-dtypes                      0.2.0
mlflow-skinny                  2.9.2
more-itertools                 8.10.0
mosaicml                       0.24.1
mosaicml-cli                   0.6.41
mpmath                         1.2.1
msgpack                        1.0.7
multidict                      6.0.4
multimethod                    1.10
multiprocess                   0.70.14
murmurhash                     1.0.10
mypy-extensions                0.4.3
nbclassic                      0.5.2
nbclient                       0.5.13
nbconvert                      6.5.4
nbformat                       5.7.0
nest-asyncio                   1.5.6
networkx                       2.8.4
ninja                          1.11.1.1
nltk                           3.7
nodeenv                        1.8.0
notebook                       6.5.2
notebook_shim                  0.2.2
numba                          0.56.4
numpy                          1.23.5
nvidia-cublas-cu12             12.1.3.1
nvidia-cuda-cupti-cu12         12.1.105
nvidia-cuda-nvrtc-cu12         12.1.105
nvidia-cuda-runtime-cu12       12.1.105
nvidia-cudnn-cu12              9.1.0.70
nvidia-cufft-cu12              11.0.2.54
nvidia-curand-cu12             10.3.2.106
nvidia-cusolver-cu12           11.4.5.107
nvidia-cusparse-cu12           12.1.0.106
nvidia-nccl-cu12               2.20.5
nvidia-nvjitlink-cu12          12.6.68
nvidia-nvtx-cu12               12.1.105
oauthlib                       3.2.0
openai                         0.28.1
opt-einsum                     3.3.0
optuna                         3.6.1
packaging                      23.2
pandas                         1.5.3
pandocfilters                  1.5.0
paramiko                       2.9.2
parso                          0.8.3
pathspec                       0.10.3
patsy                          0.5.3
petastorm                      0.12.1
pexpect                        4.8.0
phik                           0.12.4
pickleshare                    0.7.5
pillow                         10.4.0
pip                            22.3.1
platformdirs                   2.5.2
plotly                         5.9.0
pluggy                         1.0.0
pmdarima                       2.0.4
pooch                          1.4.0
preshed                        3.0.9
prompt-toolkit                 3.0.36
prophet                        1.1.5
protobuf                       4.24.0
psutil                         5.9.0
psycopg2                       2.9.3
ptyprocess                     0.7.0
pure-eval                      0.2.2
py-cpuinfo                     9.0.0
pyarrow                        8.0.0
pyarrow-hotfix                 0.5
pyasn1                         0.4.8
pyasn1-modules                 0.2.8
pybind11                       2.11.1
pycparser                      2.21
pydantic                       1.10.6
pyflakes                       3.1.0
Pygments                       2.18.0
PyGObject                      3.42.1
PyJWT                          2.3.0
PyNaCl                         1.5.0
pynvml                         11.5.0
pyodbc                         4.0.32
pyparsing                      3.0.9
pyright                        1.1.294
pyrsistent                     0.18.0
pytesseract                    0.3.10
python-apt                     2.4.0+ubuntu3
python-dateutil                2.8.2
python-editor                  1.0.4
python-lsp-jsonrpc             1.1.1
python-lsp-server              1.8.0
pytoolconfig                   1.2.5
pytorch-ranger                 0.1.1
pytz                           2022.7
PyWavelets                     1.4.1
PyYAML                         6.0
pyzmq                          23.2.0
questionary                    2.0.1
regex                          2022.7.9
requests                       2.28.1
requests-oauthlib              1.3.1
responses                      0.18.0
rich                           13.8.0
rope                           1.7.0
rsa                            4.9
ruamel.yaml                    0.18.6
ruamel.yaml.clib               0.2.8
s3transfer                     0.6.2
safetensors                    0.4.1
scikit-learn                   1.1.1
scipy                          1.10.0
seaborn                        0.12.2
SecretStorage                  3.3.1
Send2Trash                     1.8.0
sentence-transformers          2.2.2
sentencepiece                  0.1.99
setuptools                     65.6.3
shap                           0.44.0
simplejson                     3.17.6
six                            1.16.0
slicer                         0.0.7
smart-open                     5.2.1
smmap                          5.0.0
sniffio                        1.2.0
soundfile                      0.12.1
soupsieve                      2.3.2.post1
soxr                           0.3.7
spacy                          3.7.2
spacy-legacy                   3.0.12
spacy-loggers                  1.0.5
spark-tensorflow-distributor   1.0.0
SQLAlchemy                     1.4.39
sqlparse                       0.4.2
srsly                          2.4.8
ssh-import-id                  5.11
stack-data                     0.2.0
stanio                         0.3.0
statsmodels                    0.13.5
sympy                          1.11.1
tabulate                       0.9.0
tangled-up-in-unicode          0.2.0
tenacity                       8.1.0
tensorboard                    2.14.1
tensorboard-data-server        0.7.2
tensorboard-plugin-profile     2.14.0
tensorflow                     2.14.1
tensorflow-estimator           2.14.0
tensorflow-io-gcs-filesystem   0.35.0
termcolor                      2.4.0
terminado                      0.17.1
thinc                          8.2.2
threadpoolctl                  2.2.0
tiktoken                       0.5.2
tinycss2                       1.2.1
tokenize-rt                    4.2.1
tokenizers                     0.15.0
tomli                          2.0.1
torch                          2.4.0
torch-optimizer                0.3.0
torchmetrics                   1.4.0.post0
torchvision                    0.19.0
tornado                        6.1
tqdm                           4.64.1
traitlets                      5.7.1
transformers                   4.36.1
triton                         3.0.0
typeguard                      2.13.3
typer                          0.9.0
types-python-dateutil          2.9.0.20240821
typing_extensions              4.12.2
typing-inspect                 0.9.0
ujson                          5.4.0
unattended-upgrades            0.1
urllib3                        1.26.14
validators                     0.33.0
virtualenv                     20.16.7
visions                        0.7.5
wadllib                        1.3.6
wasabi                         1.1.2
wcwidth                        0.2.5
weasel                         0.3.4
webencodings                   0.5.1
websocket-client               0.58.0
websockets                     11.0.3
Werkzeug                       2.2.2
whatthepatch                   1.0.2
wheel                          0.38.4
widgetsnbextension             3.6.1
wordcloud                      1.9.3
wrapt                          1.14.1
xgboost                        1.7.6
xxhash                         3.4.1
yapf                           0.33.0
yarl                           1.9.4
ydata-profiling                4.2.0
zipp                           3.11.0
mvpatel2000 commented 2 months ago

@callmekris if you install the same package versions on the older image (pip freeze > requirements.txt, copy paste it into the old image, pip install -r requirements.txt), can you see if you still get the error?

If so, mind binary searching the package diff to try to narrow it down?

I'm confused how an import could do anything. The dependencies in ComposerModel are quite minimal, and iirc we don't do any tweaking to packages on import of Composer.... The other thing I'd try is simplifying the import. Do you get the same effect with import composer? Or is it just ComposerModel

callmekris commented 2 months ago

@callmekris if you install the same package versions on the older image (pip freeze > requirements.txt, copy paste it into the old image, pip install -r requirements.txt), can you see if you still get the error?

If so, mind binary searching the package diff to try to narrow it down?

I'm confused how an import could do anything. The dependencies in ComposerModel are quite minimal, and iirc we don't do any tweaking to packages on import of Composer.... The other thing I'd try is simplifying the import. Do you get the same effect with import composer? Or is it just ComposerModel

@mvpatel2000 sure, I will try that. I am confused as well. The issue does occur as well when importing just composer or any composer sub modules.