search

Lightning-AI / pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.
https://lightning.ai
Apache License 2.0
28.31k stars 3.38k forks source link

ModelCheckpoint, CSVLogger, TensorBoardLogger DO NOT have same saving paths. #14321

Closed dinhanhx closed 2 years ago

dinhanhx commented 2 years ago

🐛 Bug

ModelCheckpoint DID not save to same version folder when there are some loggers.

To Reproduce

import os

import torch
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.loggers import CSVLogger, TensorBoardLogger
from pytorch_lightning.callbacks import RichProgressBar, ModelCheckpoint

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

def run():
    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    val_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    test_data = DataLoader(RandomDataset(32, 64), batch_size=2)

    model = BoringModel()
    LOG_DIR = os.getcwd()
    trainer = Trainer(
        default_root_dir=LOG_DIR,
        num_sanity_val_steps=0,
        max_epochs=1,
        logger=[CSVLogger(LOG_DIR), TensorBoardLogger(LOG_DIR)], 
        callbacks=[RichProgressBar(), ModelCheckpoint(LOG_DIR, every_n_train_steps=1)],
        log_every_n_steps=1
    )
    trainer.fit(model, train_dataloaders=train_data)

if __name__ == "__main__":
    run()
.
├── __notebook_source__.ipynb
├── epoch=0-step=32.ckpt
└── lightning_logs
    ├── version_0
    │   ├── hparams.yaml
    │   └── metrics.csv
    └── version_1
        ├── events.out.tfevents.1660967780.f67d5d958ef0.17.0
        └── hparams.yaml

Expected behavior

Model checkpoints file should be under same path as other loggers.

.
├── __notebook_source__.ipynb
└── lightning_logs
    ├── version_0
    │   ├── hparams.yaml
    │   ├── epoch=0-step=32.ckpt
    │   └── metrics.csv
    └── version_1
        ├── events.out.tfevents.1660967780.f67d5d958ef0.17.0
        ├── epoch=0-step=32.ckpt
        └── hparams.yaml

Environment

* CUDA:
    - GPU:
        - Tesla P100-PCIE-16GB
    - available:         True
    - version:           11.0
* Lightning:
    - pytorch-ignite:    0.4.9
    - pytorch-lightning: 1.6.5
    - torch:             1.11.0
    - torchaudio:        0.11.0
    - torchmetrics:      0.9.2
    - torchtext:         0.12.0
    - torchvision:       0.12.0
* Packages:
    - absl-py:           1.1.0
    - accelerate:        0.11.0
    - access:            1.1.8
    - affine:            2.3.1
    - aiobotocore:       2.3.4
    - aiohttp:           3.8.1
    - aioitertools:      0.10.0
    - aiosignal:         1.2.0
    - albumentations:    1.2.1
    - alembic:           1.8.1
    - allennlp:          2.10.0
    - altair:            4.2.0
    - annoy:             1.17.0
    - ansiwrap:          0.8.4
    - anyio:             3.6.1
    - apache-beam:       2.40.0
    - aplus:             0.11.0
    - appdirs:           1.4.4
    - argon2-cffi:       21.3.0
    - argon2-cffi-bindings: 21.2.0
    - arrow:             1.2.2
    - arviz:             0.12.1
    - asn1crypto:        1.5.1
    - astroid:           2.11.7
    - astropy:           4.3.1
    - astunparse:        1.6.3
    - async-timeout:     4.0.2
    - asynctest:         0.13.0
    - atpublic:          2.3
    - attrs:             21.4.0
    - audioread:         2.1.9
    - autocfg:           0.0.8
    - autopage:          0.5.1
    - autopep8:          1.6.0
    - aws-requests-auth: 0.4.3
    - babel:             2.10.3
    - backcall:          0.2.0
    - backports.functools-lru-cache: 1.6.4
    - backports.zoneinfo: 0.2.1
    - base58:            2.1.1
    - bayesian-optimization: 1.2.0
    - bayespy:           0.5.22
    - beatrix-jupyterlab: 3.1.7
    - beautifulsoup4:    4.11.1
    - bidict:            0.22.0
    - binaryornot:       0.4.4
    - biopython:         1.79
    - black:             22.6.0
    - blake3:            0.2.1
    - bleach:            5.0.1
    - blinker:           1.4
    - blis:              0.7.8
    - bokeh:             2.4.3
    - boruta:            0.3
    - boto3:             1.24.33
    - botocore:          1.27.33
    - bq-helper:         0.4.1
    - bqplot:            0.12.33
    - branca:            0.5.0
    - brewer2mpl:        1.4.1
    - brotlipy:          0.7.0
    - cached-path:       1.1.5
    - cached-property:   1.5.2
    - cachetools:        4.2.4
    - cartopy:           0.19.0.post1
    - catalogue:         2.0.7
    - catalyst:          22.4
    - catboost:          1.0.6
    - category-encoders: 2.5.0
    - certifi:           2022.6.15
    - cesium:            0.10.1
    - cffi:              1.15.0
    - cftime:            1.6.1
    - chardet:           5.0.0
    - charset-normalizer: 2.1.0
    - chex:              0.1.3
    - clang:             5.0
    - cleverhans:        4.0.0
    - click:             8.0.4
    - click-plugins:     1.1.1
    - cliff:             3.10.1
    - cligj:             0.7.2
    - cloudpickle:       2.1.0
    - cmaes:             0.8.2
    - cmake:             3.22.5
    - cmd2:              2.4.2
    - cmdstanpy:         1.0.4
    - cmudict:           1.0.2
    - colorama:          0.4.5
    - colorcet:          3.0.0
    - colorlog:          6.6.0
    - colorlover:        0.3.0
    - commonmark:        0.9.1
    - conda:             4.13.0
    - conda-package-handling: 1.8.1
    - configparser:      5.2.0
    - contextily:        1.2.0
    - contextlib2:       21.6.0
    - convertdate:       2.4.0
    - cookiecutter:      2.1.1
    - crcmod:            1.7
    - cryptography:      37.0.2
    - cudf:              21.10.1
    - cufflinks:         0.17.3
    - cuml:              21.10.2
    - cupy:              9.6.0
    - cupy-cuda110:      10.6.0
    - cvxcanon:          0.1.2
    - cycler:            0.11.0
    - cymem:             2.0.6
    - cysignals:         1.11.2
    - cython:            0.29.30
    - cytoolz:           0.12.0
    - daal:              2021.6.0
    - daal4py:           2021.6.3
    - dask:              2022.2.0
    - dask-cudf:         21.10.1
    - dataclasses:       0.8
    - datasets:          2.1.0
    - datashader:        0.14.1
    - datashape:         0.5.2
    - datatable:         1.0.0
    - datatile:          1.0.1
    - deap:              1.3.1
    - debugpy:           1.6.0
    - decorator:         5.1.1
    - defusedxml:        0.7.1
    - delorean:          1.0.0
    - deprecat:          2.1.1
    - deprecation:       2.1.0
    - descartes:         1.1.0
    - dill:              0.3.5.1
    - dipy:              1.5.0
    - distlib:           0.3.5
    - distributed:       2022.2.0
    - dlib:              19.24.0
    - dm-tree:           0.1.7
    - docker:            5.0.3
    - docker-pycreds:    0.4.0
    - docopt:            0.6.2
    - docutils:          0.19
    - earthengine-api:   0.1.317
    - easydev:           0.12.0
    - easydict:          1.9
    - easyocr:           1.5.0
    - ecos:              2.0.10
    - eli5:              0.13.0
    - emoji:             2.0.0
    - en-core-web-lg:    3.3.0
    - en-core-web-sm:    3.3.0
    - entrypoints:       0.4
    - ephem:             4.1.3
    - esda:              2.4.1
    - essentia:          2.1b6.dev778
    - etils:             0.6.0
    - explainable-ai-sdk: 1.3.3
    - explainers:        0.1
    - fairscale:         0.4.6
    - fastai:            2.7.7
    - fastapi:           0.79.0
    - fastavro:          1.5.3
    - fastcore:          1.5.6
    - fastdownload:      0.0.7
    - fasteners:         0.17.3
    - fastjsonschema:    2.15.3
    - fastprogress:      1.0.2
    - fastrlock:         0.8
    - fasttext:          0.9.2
    - fbpca:             1.0
    - feather-format:    0.4.1
    - featuretools:      1.11.1
    - filelock:          3.7.1
    - fiona:             1.8.21
    - fitter:            1.4.0
    - flake8:            4.0.1
    - flashtext:         2.7
    - flask:             2.1.3
    - flatbuffers:       1.12
    - flax:              0.5.2
    - flit-core:         3.7.1
    - folium:            0.12.1.post1
    - fonttools:         4.33.3
    - frozendict:        2.3.2
    - frozenlist:        1.3.0
    - fsspec:            2022.5.0
    - funcy:             1.17
    - fury:              0.8.0
    - future:            0.18.2
    - fuzzywuzzy:        0.18.0
    - gast:              0.4.0
    - gatspy:            0.3
    - gcsfs:             2022.5.0
    - gensim:            4.0.1
    - geographiclib:     1.52
    - geohash:           1.0
    - geojson:           2.5.0
    - geopandas:         0.10.2
    - geoplot:           0.5.1
    - geopy:             2.2.0
    - geoviews:          1.9.5
    - ggplot:            0.11.5
    - giddy:             2.3.3
    - gitdb:             4.0.9
    - gitpython:         3.1.27
    - gluoncv:           0.10.5.post0
    - gluonnlp:          0.10.0
    - google-api-core:   1.32.0
    - google-api-python-client: 1.12.11
    - google-apitools:   0.5.31
    - google-auth:       1.35.0
    - google-auth-httplib2: 0.1.0
    - google-auth-oauthlib: 0.4.6
    - google-cloud-aiplatform: 0.6.0a1
    - google-cloud-appengine-logging: 1.1.2
    - google-cloud-audit-log: 0.2.2
    - google-cloud-automl: 1.0.1
    - google-cloud-bigquery: 2.2.0
    - google-cloud-bigtable: 2.10.1
    - google-cloud-core: 1.7.3
    - google-cloud-dataproc: 4.0.3
    - google-cloud-datastore: 2.7.1
    - google-cloud-dlp:  3.7.1
    - google-cloud-firestore: 2.5.3
    - google-cloud-kms:  2.11.2
    - google-cloud-language: 2.4.3
    - google-cloud-logging: 3.1.2
    - google-cloud-monitoring: 2.9.2
    - google-cloud-pubsub: 2.13.0
    - google-cloud-pubsublite: 1.4.2
    - google-cloud-recommendations-ai: 0.2.0
    - google-cloud-resource-manager: 1.5.1
    - google-cloud-scheduler: 2.6.4
    - google-cloud-spanner: 3.15.1
    - google-cloud-speech: 2.14.1
    - google-cloud-storage: 1.44.0
    - google-cloud-tasks: 2.9.1
    - google-cloud-translate: 3.7.4
    - google-cloud-videointelligence: 2.7.1
    - google-cloud-vision: 2.7.3
    - google-crc32c:     1.1.2
    - google-pasta:      0.2.0
    - google-resumable-media: 1.3.3
    - googleapis-common-protos: 1.56.3
    - gplearn:           0.4.2
    - gpxpy:             1.5.0
    - graphviz:          0.8.4
    - greenlet:          1.1.2
    - grpc-google-iam-v1: 0.12.4
    - grpcio:            1.43.0
    - grpcio-gcp:        0.2.2
    - grpcio-status:     1.47.0
    - gym:               0.25.0
    - gym-notices:       0.0.7
    - h11:               0.13.0
    - h2o:               3.36.1.3
    - h5py:              3.7.0
    - haversine:         2.6.0
    - hdfs:              2.7.0
    - heapdict:          1.0.1
    - hep-ml:            0.7.1
    - hijri-converter:   2.2.4
    - hmmlearn:          0.2.7
    - holidays:          0.14.2
    - holoviews:         1.15.0
    - hpsklearn:         0.1.0
    - html5lib:          1.1
    - htmlmin:           0.1.12
    - httplib2:          0.20.4
    - httplib2shim:      0.0.3
    - httptools:         0.4.0
    - huggingface-hub:   0.8.1
    - humanize:          4.2.3
    - hunspell:          0.5.5
    - husl:              4.0.3
    - hydra-slayer:      0.4.0
    - hyperopt:          0.2.7
    - hypertools:        0.8.0
    - ibis-framework:    2.1.1
    - idna:              3.3
    - igraph:            0.9.11
    - imagecodecs:       2021.11.20
    - imagehash:         4.2.1
    - imageio:           2.19.3
    - imbalanced-learn:  0.9.0
    - imgaug:            0.4.0
    - implicit:          0.4.4
    - importlib-metadata: 4.12.0
    - importlib-resources: 5.8.0
    - inequality:        1.0.0
    - iniconfig:         1.1.1
    - ipydatawidgets:    4.3.1.post1
    - ipykernel:         6.15.0
    - ipyleaflet:        0.17.0
    - ipympl:            0.7.0
    - ipython:           7.33.0
    - ipython-genutils:  0.2.0
    - ipython-sql:       0.3.9
    - ipyvolume:         0.5.2
    - ipyvue:            1.7.0
    - ipyvuetify:        1.8.2
    - ipywebrtc:         0.6.0
    - ipywidgets:        7.7.1
    - iso3166:           2.1.1
    - isort:             5.10.1
    - isoweek:           1.3.3
    - itsdangerous:      2.1.2
    - janome:            0.4.2
    - jax:               0.3.14
    - jaxlib:            0.3.14+cuda11.cudnn805
    - jedi:              0.18.1
    - jeepney:           0.8.0
    - jieba:             0.42.1
    - jinja2:            3.1.2
    - jinja2-time:       0.2.0
    - jmespath:          1.0.1
    - joblib:            1.0.1
    - json5:             0.9.5
    - jsonlines:         1.2.0
    - jsonnet:           0.18.0
    - jsonschema:        4.6.1
    - jupyter:           1.0.0
    - jupyter-client:    7.3.4
    - jupyter-console:   6.4.4
    - jupyter-core:      4.10.0
    - jupyter-http-over-ws: 0.0.8
    - jupyter-lsp:       1.5.1
    - jupyter-server:    1.18.0
    - jupyter-server-mathjax: 0.2.5
    - jupyter-server-proxy: 3.2.1
    - jupyterlab:        3.2.9
    - jupyterlab-git:    0.37.1
    - jupyterlab-lsp:    3.10.1
    - jupyterlab-pygments: 0.2.2
    - jupyterlab-server: 2.10.3
    - jupyterlab-widgets: 1.1.1
    - jupytext:          1.13.8
    - kaggle:            1.5.12
    - kaggle-environments: 1.9.11
    - keras:             2.6.0
    - keras-preprocessing: 1.1.2
    - keras-tuner:       1.1.2
    - keyring:           23.6.0
    - keyrings.google-artifactregistry-auth: 1.0.0
    - kiwisolver:        1.4.3
    - kmapper:           2.0.1
    - kmodes:            0.12.1
    - korean-lunar-calendar: 0.2.1
    - kornia:            0.5.8
    - kt-legacy:         1.0.4
    - kubernetes:        24.2.0
    - langcodes:         3.3.0
    - langid:            1.1.6
    - lazy-object-proxy: 1.7.1
    - learntools:        0.3.4
    - leven:             1.0.4
    - libpysal:          4.6.2
    - librosa:           0.9.2
    - lightfm:           1.16
    - lightgbm:          3.3.2
    - lime:              0.2.0.1
    - line-profiler:     3.5.1
    - llvmlite:          0.38.1
    - lmdb:              1.3.0
    - lml:               0.1.0
    - locket:            1.0.0
    - lunarcalendar:     0.0.9
    - lxml:              4.9.1
    - mako:              1.2.1
    - mapclassify:       2.4.3
    - marisa-trie:       0.7.7
    - markdown:          3.3.7
    - markdown-it-py:    2.1.0
    - markovify:         0.9.4
    - markupsafe:        2.0.1
    - matplotlib:        3.5.2
    - matplotlib-inline: 0.1.3
    - matplotlib-venn:   0.11.7
    - matrixprofile:     1.1.10
    - mccabe:            0.6.1
    - mdit-py-plugins:   0.3.0
    - mdurl:             0.1.0
    - memory-profiler:   0.60.0
    - mercantile:        1.2.1
    - mgwr:              2.1.2
    - missingno:         0.4.2
    - mistune:           0.8.4
    - mizani:            0.7.3
    - mlcrate:           0.2.0
    - mlens:             0.2.3
    - mlxtend:           0.20.0
    - mmh3:              3.0.0
    - mne:               1.0.3
    - mnist:             0.2.2
    - mock:              4.0.3
    - momepy:            0.5.3
    - more-itertools:    8.13.0
    - mpld3:             0.5.8
    - mpmath:            1.2.1
    - msgpack:           1.0.4
    - msgpack-numpy:     0.4.8
    - multidict:         6.0.2
    - multimethod:       1.4
    - multipledispatch:  0.6.0
    - multiprocess:      0.70.13
    - munch:             2.5.0
    - munkres:           1.1.4
    - murmurhash:        1.0.7
    - mxnet-cu110:       1.9.1
    - mypy-extensions:   0.4.3
    - nb-conda:          2.2.1
    - nb-conda-kernels:  2.3.1
    - nbclassic:         0.3.7
    - nbclient:          0.5.13
    - nbconvert:         6.4.5
    - nbdime:            3.1.1
    - nbformat:          5.4.0
    - nest-asyncio:      1.5.5
    - netcdf4:           1.6.0
    - networkx:          2.5
    - nibabel:           4.0.1
    - nilearn:           0.9.1
    - nltk:              3.7
    - nnabla:            1.29.0
    - nnabla-ext-cuda110: 1.29.0
    - nose:              1.3.7
    - notebook:          6.4.12
    - notebook-executor: 0.2
    - notebook-shim:     0.1.0
    - numba:             0.55.2
    - numexpr:           2.8.3
    - numpy:             1.21.6
    - nvtx:              0.2.3
    - oauth2client:      4.1.3
    - oauthlib:          3.2.0
    - odfpy:             1.4.1
    - olefile:           0.46
    - onnx:              1.12.0
    - opencv-contrib-python: 4.5.4.60
    - opencv-python:     4.5.4.60
    - opencv-python-headless: 4.5.4.60
    - openslide-python:  1.2.0
    - opt-einsum:        3.3.0
    - optax:             0.1.3
    - optuna:            2.10.1
    - orderedmultidict:  1.0.1
    - orjson:            3.7.5
    - ortools:           9.3.10497
    - osmnx:             1.1.1
    - overrides:         6.1.0
    - packaging:         21.3
    - palettable:        3.3.0
    - pandarallel:       1.6.1
    - pandas:            1.3.5
    - pandas-datareader: 0.10.0
    - pandas-profiling:  3.1.0
    - pandas-summary:    0.2.0
    - pandasql:          0.7.3
    - pandocfilters:     1.5.0
    - panel:             0.13.1
    - papermill:         2.3.4
    - param:             1.12.2
    - parso:             0.8.3
    - parsy:             1.4.0
    - partd:             1.2.0
    - path:              16.4.0
    - path.py:           12.5.0
    - pathos:            0.2.9
    - pathspec:          0.9.0
    - pathtools:         0.1.2
    - pathy:             0.6.2
    - patsy:             0.5.2
    - pbr:               5.9.0
    - pdf2image:         1.16.0
    - pdpbox:            0.2.1
    - pexpect:           4.8.0
    - phik:              0.12.2
    - pickleshare:       0.7.5
    - pillow:            9.1.1
    - pip:               22.1.2
    - platformdirs:      2.5.1
    - plotly:            5.9.0
    - plotly-express:    0.4.1
    - plotnine:          0.8.0
    - pluggy:            1.0.0
    - pointpats:         2.2.0
    - polyglot:          16.7.4
    - pooch:             1.6.0
    - portalocker:       2.5.1
    - pox:               0.3.1
    - ppca:              0.0.4
    - ppft:              1.7.6.5
    - preprocessing:     0.1.13
    - preshed:           3.0.6
    - prettytable:       3.3.0
    - progressbar2:      4.0.0
    - prometheus-client: 0.14.1
    - promise:           2.3
    - prompt-toolkit:    3.0.30
    - pronouncing:       0.2.0
    - prophet:           1.1
    - proto-plus:        1.20.6
    - protobuf:          3.19.4
    - psutil:            5.9.1
    - ptyprocess:        0.7.0
    - pudb:              2022.1.2
    - pulp:              2.6.0
    - py:                1.11.0
    - py-lz4framed:      0.14.0
    - py-stringmatching: 0.4.2
    - py-stringsimjoin:  0.3.2
    - py4j:              0.10.9.5
    - pyaml:             21.10.1
    - pyarabic:          0.6.15
    - pyarrow:           5.0.0
    - pyasn1:            0.4.8
    - pyasn1-modules:    0.2.7
    - pyastronomy:       0.17.1
    - pybind11:          2.10.0
    - pycodestyle:       2.8.0
    - pycosat:           0.6.3
    - pycountry:         22.3.5
    - pycparser:         2.21
    - pycrypto:          2.6.1
    - pyct:              0.4.8
    - pycuda:            2022.1
    - pydantic:          1.8.2
    - pydash:            5.1.0
    - pydegensac:        0.1.2
    - pydeprecate:       0.3.2
    - pydicom:           2.3.0
    - pydocstyle:        6.1.1
    - pydot:             1.4.2
    - pydub:             0.25.1
    - pyemd:             0.5.1
    - pyerfa:            2.0.0.1
    - pyexcel-io:        0.6.6
    - pyexcel-ods:       0.6.0
    - pyfasttext:        0.4.6
    - pyflakes:          2.4.0
    - pygeos:            0.12.0
    - pygments:          2.12.0
    - pyjwt:             2.4.0
    - pykalman:          0.9.5
    - pyldavis:          3.2.2
    - pylint:            2.14.5
    - pymc3:             3.11.5
    - pymeeus:           0.5.11
    - pymongo:           3.12.3
    - pympler:           1.0.1
    - pynndescent:       0.5.7
    - pynvml:            11.4.1
    - pynvrtc:           9.2
    - pyocr:             0.8.3
    - pyopenssl:         22.0.0
    - pyparsing:         3.0.9
    - pypdf:             1.13
    - pyperclip:         1.8.2
    - pyprind:           2.11.3
    - pyproj:            3.1.0
    - pyrsistent:        0.18.1
    - pysal:             2.6.0
    - pyshp:             2.3.0
    - pysocks:           1.7.1
    - pytesseract:       0.3.9
    - pytest:            7.1.2
    - python-bidi:       0.4.2
    - python-dateutil:   2.8.2
    - python-dotenv:     0.20.0
    - python-igraph:     0.9.11
    - python-levenshtein: 0.12.2
    - python-louvain:    0.16
    - python-lsp-jsonrpc: 1.0.0
    - python-lsp-server: 1.5.0
    - python-slugify:    6.1.2
    - python-utils:      3.3.3
    - pythreejs:         2.3.0
    - pytoolconfig:      1.2.1
    - pytools:           2022.1.12
    - pytorch-ignite:    0.4.9
    - pytorch-lightning: 1.6.5
    - pytz:              2022.1
    - pytz-deprecation-shim: 0.1.0.post0
    - pyu2f:             0.1.5
    - pyupset:           0.1.1.post7
    - pyviz-comms:       2.2.0
    - pywavelets:        1.3.0
    - pyyaml:            6.0
    - pyzmq:             23.2.0
    - qgrid:             1.3.1
    - qtconsole:         5.3.1
    - qtpy:              2.1.0
    - quantecon:         0.5.3
    - quantities:        0.13.0
    - qudida:            0.0.4
    - quilt3:            5.0.0
    - randomgen:         1.23.1
    - rasterio:          1.2.10
    - rasterstats:       0.16.0
    - ray:               1.13.0
    - regex:             2021.11.10
    - requests:          2.28.1
    - requests-futures:  1.0.0
    - requests-oauthlib: 1.3.1
    - resampy:           0.3.1
    - responses:         0.18.0
    - retrying:          1.3.3
    - rgf-python:        3.12.0
    - rich:              12.1.0
    - rmm:               21.10.1
    - rope:              1.2.0
    - rsa:               4.8
    - rtree:             1.0.0
    - ruamel-yaml-conda: 0.15.100
    - rvlib:             0.0.6
    - s2sphere:          0.2.5
    - s3fs:              2022.5.0
    - s3transfer:        0.6.0
    - sacremoses:        0.0.53
    - scattertext:       0.1.6
    - scikit-image:      0.19.3
    - scikit-learn:      1.0.2
    - scikit-learn-intelex: 2021.6.3
    - scikit-multilearn: 0.2.0
    - scikit-optimize:   0.9.0
    - scikit-plot:       0.3.7
    - scikit-surprise:   1.1.1
    - scipy:             1.7.3
    - seaborn:           0.11.2
    - secretstorage:     3.3.2
    - segregation:       2.3.1
    - semver:            2.13.0
    - send2trash:        1.8.0
    - sentencepiece:     0.1.96
    - sentry-sdk:        1.7.2
    - setproctitle:      1.2.3
    - setuptools:        59.8.0
    - setuptools-git:    1.2
    - shap:              0.41.0
    - shapely:           1.8.0
    - shortuuid:         1.0.9
    - simpervisor:       0.4
    - simpleitk:         2.1.1.2
    - simplejson:        3.17.6
    - six:               1.16.0
    - sklearn:           0.0
    - sklearn-contrib-py-earth: 0.1.0+1.gdde5f89
    - sklearn-pandas:    2.2.0
    - slicer:            0.0.7
    - smart-open:        5.2.1
    - smhasher:          0.150.1
    - smmap:             3.0.5
    - sniffio:           1.2.0
    - snowballstemmer:   2.2.0
    - snuggs:            1.4.7
    - sortedcontainers:  2.4.0
    - soundfile:         0.10.3.post1
    - soupsieve:         2.3.1
    - spacy:             3.3.1
    - spacy-legacy:      3.0.9
    - spacy-loggers:     1.0.3
    - spaghetti:         1.6.5
    - spectral:          0.22.4
    - spglm:             1.0.8
    - sphinx-rtd-theme:  0.2.4
    - spint:             1.0.7
    - splot:             1.1.5.post1
    - spopt:             0.4.1
    - spreg:             1.2.4
    - spvcm:             0.3.0
    - sqlalchemy:        1.4.39
    - sqlparse:          0.4.2
    - squarify:          0.4.3
    - srsly:             2.4.3
    - starlette:         0.19.1
    - statsmodels:       0.13.2
    - stemming:          1.0.1
    - stevedore:         3.5.0
    - stop-words:        2018.7.23
    - stopit:            1.1.2
    - stumpy:            1.11.1
    - sympy:             1.10.1
    - tabulate:          0.8.10
    - tangled-up-in-unicode: 0.1.0
    - tbb:               2021.6.0
    - tblib:             1.7.0
    - tenacity:          8.0.1
    - tensorboard:       2.6.0
    - tensorboard-data-server: 0.6.1
    - tensorboard-plugin-wit: 1.8.1
    - tensorboardx:      2.5.1
    - tensorflow:        2.6.4
    - tensorflow-addons: 0.14.0
    - tensorflow-cloud:  0.1.14
    - tensorflow-datasets: 4.3.0
    - tensorflow-estimator: 2.6.0
    - tensorflow-gcs-config: 2.6.0
    - tensorflow-hub:    0.12.0
    - tensorflow-io:     0.21.0
    - tensorflow-metadata: 1.9.0
    - tensorflow-probability: 0.14.1
    - tensorflow-serving-api: 2.9.0
    - tensorflow-transform: 1.9.0
    - tensorpack:        0.11
    - termcolor:         1.1.0
    - terminado:         0.15.0
    - testpath:          0.6.0
    - text-unidecode:    1.3
    - textblob:          0.17.1
    - texttable:         1.6.4
    - textwrap3:         0.9.2
    - tfx-bsl:           1.9.0
    - theano:            1.0.5
    - theano-pymc:       1.1.2
    - thinc:             8.0.17
    - threadpoolctl:     3.1.0
    - tifffile:          2021.11.2
    - tinycss2:          1.1.1
    - tobler:            0.9.0
    - tokenizers:        0.12.1
    - toml:              0.10.2
    - tomli:             2.0.1
    - tomlkit:           0.11.1
    - toolz:             0.11.2
    - torch:             1.11.0
    - torchaudio:        0.11.0
    - torchmetrics:      0.9.2
    - torchtext:         0.12.0
    - torchvision:       0.12.0
    - tornado:           6.1
    - tpot:              0.11.7
    - tqdm:              4.64.0
    - traceml:           1.0.1
    - traitlets:         5.3.0
    - traittypes:        0.2.1
    - transformers:      4.20.1
    - treelite:          2.1.0
    - treelite-runtime:  2.1.0
    - trueskill:         0.4.5
    - tsfresh:           0.19.0
    - typed-ast:         1.5.4
    - typeguard:         2.13.3
    - typer:             0.4.2
    - typing-extensions: 4.1.1
    - typing-utils:      0.1.0
    - tzdata:            2022.1
    - tzlocal:           4.2
    - ucx-py:            0.22.1+33.g9971bd1
    - ujson:             5.3.0
    - umap-learn:        0.5.3
    - unicodedata2:      14.0.0
    - unidecode:         1.3.4
    - update-checker:    0.18.0
    - uritemplate:       3.0.1
    - urllib3:           1.26.9
    - urwid:             2.1.2
    - urwid-readline:    0.13
    - uvicorn:           0.18.2
    - uvloop:            0.16.0
    - vaex:              4.9.2
    - vaex-astro:        0.9.1
    - vaex-core:         4.9.2
    - vaex-hdf5:         0.12.2
    - vaex-jupyter:      0.8.0
    - vaex-ml:           0.17.0
    - vaex-server:       0.8.1
    - vaex-viz:          0.5.2
    - vecstack:          0.4.0
    - virtualenv:        20.15.1
    - visions:           0.7.4
    - vowpalwabbit:      9.2.0
    - vtk:               9.1.0
    - wand:              0.6.8
    - wandb:             0.12.21
    - wasabi:            0.9.1
    - watchfiles:        0.15.0
    - wavio:             0.0.4
    - wcwidth:           0.2.5
    - webencodings:      0.5.1
    - websocket-client:  1.3.3
    - websockets:        10.3
    - werkzeug:          2.1.2
    - wfdb:              4.0.0
    - whatthepatch:      1.0.2
    - wheel:             0.37.1
    - widgetsnbextension: 3.6.1
    - witwidget:         1.8.0
    - woodwork:          0.16.4
    - wordbatch:         1.4.9
    - wordcloud:         1.8.2.2
    - wordsegment:       1.3.1
    - wrapt:             1.14.1
    - wslink:            1.6.6
    - xarray:            0.20.2
    - xarray-einstats:   0.2.2
    - xgboost:           1.6.1
    - xvfbwrapper:       0.2.9
    - xxhash:            3.0.0
    - xyzservices:       2022.6.0
    - yacs:              0.1.8
    - yapf:              0.32.0
    - yarl:              1.7.2
    - yellowbrick:       1.4
    - zict:              2.2.0
    - zipp:              3.8.0
* System:
    - OS:                Linux
    - architecture:
        - 64bit
        - 
    - processor:         x86_64
    - python:            3.7.12
    - version:           #1 SMP Tue Aug 9 15:29:43 UTC 2022

Additional context

No more information

cc @borda @awaelchli @edward-io @ananthsub @rohitgr7 @kamil-kaczmarek @Raalsky @Blaizzy

dinhanhx commented 2 years ago

Kaggle notebook to reproduce bugs

tshu-w commented 2 years ago

Hi, this is due to the fact that the checkpoint is currently stored in default_root_dir when using multiple loggers. I will submit a PR these two days to ensure write to the first logger in the list of loggers as discussed in https://github.com/Lightning-AI/lightning/issues/12177.

dinhanhx commented 2 years ago

is there any workaround?

tshu-w commented 2 years ago

You can try my PR util merged https://github.com/Lightning-AI/lightning/pull/14325