Model cannot be saved

[miranska]

System information

• OS Platform and Distribution: MacOS 14.2.1
• Python version: 3.11.7
• DeepTables version: 0.2.6
• Other Python packages(run pip list):

  Package                      Version                                                                                                                                                                                                                                                                                                                                                        
  ---------------------------- ------------
  absl-py                      2.0.0
  asttokens                    2.4.1
  astunparse                   1.6.3
  attrs                        23.2.0
  bcrypt                       4.1.2
  cachetools                   5.3.2
  category-encoders            2.6.3
  certifi                      2023.11.17
  cffi                         1.16.0
  charset-normalizer           3.3.2
  cryptography                 41.0.7
  decorator                    5.1.1
  deeptables                   0.2.6
  eli5                         0.13.0
  executing                    2.0.1
  flatbuffers                  23.5.26
  fsspec                       2023.12.2
  gast                         0.5.4
  google-auth                  2.26.1
  google-auth-oauthlib         1.2.0
  google-pasta                 0.2.0
  graphviz                     0.20.1
  grpcio                       1.60.0
  h5py                         3.10.0
  hypernets                    0.3.1
  idna                         3.6
  ipython                      8.19.0
  jedi                         0.19.1
  Jinja2                       3.1.2
  joblib                       1.3.2
  keras                        2.15.0
  libclang                     16.0.6
  lightgbm                     4.2.0
  Markdown                     3.5.1
  MarkupSafe                   2.1.3
  matplotlib-inline            0.1.6
  ml-dtypes                    0.2.0
  numpy                        1.26.3
  oauthlib                     3.2.2
  opt-einsum                   3.3.0
  packaging                    23.2
  pandas                       2.1.4
  paramiko                     3.4.0
  parso                        0.8.3
  patsy                        0.5.6
  pexpect                      4.9.0
  pip                          23.3.2
  prettytable                  3.9.0
  prompt-toolkit               3.0.43
  protobuf                     4.23.4
  psutil                       5.9.7
  ptyprocess                   0.7.0
  pure-eval                    0.2.2
  pyasn1                       0.5.1
  pyasn1-modules               0.3.0
  pycparser                    2.21
  Pygments                     2.17.2
  PyNaCl                       1.5.0
  python-dateutil              2.8.2
  pytz                         2023.3.post1
  PyYAML                       6.0.1
  requests                     2.31.0
  requests-oauthlib            1.3.1
  rsa                          4.9
  scikit-learn                 1.3.2
  scipy                        1.11.4
  setuptools                   69.0.3
  six                          1.16.0
  stack-data                   0.6.3
  statsmodels                  0.14.1
  tabulate                     0.9.0
  tensorboard                  2.15.1
  tensorboard-data-server      0.7.2
  tensorflow                   2.15.0
  tensorflow-estimator         2.15.0
  tensorflow-io-gcs-filesystem 0.34.0
  tensorflow-macos             2.15.0
  termcolor                    2.4.0
  threadpoolctl                3.2.0
  tornado                      6.4
  tqdm                         4.66.1
  traitlets                    5.14.1
  typing_extensions            4.9.0
  tzdata                       2023.4
  urllib3                      2.1.0
  wcwidth                      0.2.12
  Werkzeug                     3.0.1
  wheel                        0.42.0
  wrapt                        1.14.1
  XlsxWriter                   3.1.9

Describe the current behavior

I run sample classification code from the documentation:

During execution, I see the following message in the logs:

01-04 14:44:02 I deeptables.m.deeptable.py 369 - Training finished.
./miniforge3/envs/sample_deeptable/lib/python3.11/site-packages/deeptables/models/deepmodel.py:188: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  save_model(self.model, h, save_format='h5')
01-04 14:44:02 I deeptables.m.deeptable.py 704 - Model has been saved to:dt_output/dt_20240104144359_linear_dnn_nets/linear+dnn_nets.h5

Describe the expected behavior

I do not see any output in the dt_output subdirectory where my source file was located. Furthermore, I couldn't find linear+dnn_nets.h5 anywhere on the hard drive.

Standalone code to reproduce the issue

# sample code from https://deeptables.readthedocs.io/en/latest/examples.html
from deeptables.models.deeptable import DeepTable, ModelConfig
from deeptables.models.deepnets import WideDeep
from deeptables.datasets import dsutils
from sklearn.model_selection import train_test_split

# Adult Data Set from UCI Machine Learning Repository: https://archive.ics.uci.edu/ml/datasets/Adult
df_train = dsutils.load_adult()
y = df_train.pop(14)
X = df_train

conf = ModelConfig(nets=WideDeep, metrics=["AUC", "accuracy"], auto_discrete=True)
dt = DeepTable(config=conf)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model, history = dt.fit(X_train, y_train, epochs=100)

[oaksharks]

Hi @miranska ! The model file dt_output/dt_20240104144359_linear_dnn_nets/linear+dnn_nets.h5 that recorded in your log is exactly a path that relative to DeepTables's workdir. You can check the workdir by code:

from hypernets.utils import fs
print(f"workdir: {fs.remote_root_}")

In my OS (centos 7.9), it's /tmp/workdir, so I should check the files generated by deeptables in this directory.

For the warning below:

You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.

It does not prevent model persistence, if you want to eliminate this warning, just downgrade tensorflow<=2.11.0

For further infomation of model persistence, you can refer to this example:

• https://github.com/DataCanvasIO/DeepTables/blob/master/deeptables/examples/quickstart.ipynb

[miranska]

@oaksharks, great thank you! So in order to persist a model in my own directory, I should first save it to fs.remote_root_ directory (which I can't change) and then move the model to my own directory using something like shutil.move(), correct?

[oaksharks]

You can do that shutil.move() or change the workdir, see more https://github.com/DataCanvasIO/DeepTables/issues/85

[miranska]

Thank you!