Closed dinhanhx closed 1 year ago
With larger scale (bigger model, bigger dataset, bigger batch size), the differences are noticeable.
In the following pictures, the purple line is the resumed version of the blue line. And the green line is the training without interruption. As we can see, the blue line and the green line are the same until the interruption. After the interruption, the purple line is very different from the green line.
@dinhanhx This is expected, because after resuming, the random state of the program (e.g. in torch) is different than it was when it stopped. This is very much expected. What matters is that the training loss converges to the same result in the end, this seems to be the case in your experiments (the curves are not identical, but their average value is the same). Please let me know if I should explain it in more detail.
Restoring the random state to exactly the way it was when stopped is highly non-trivial. We investigated this in the past but found it too complex, while at the same time it is rarely needed in practice.
What matters is that the training loss converges to the same result in the end, this seems to be the case in your experiments (the curves are not identical, but their average value is the same).
I have noticed the similar pattern with my other training experiments other than this issue. I guess the difference would not affect the final accuracy. Thanks @awaelchli for explaining.
I have noticed the similar pattern with my other training experiments other than this issue
Just to clarify, I'm not sure what you are saying. Is it that in general you agree with me and that training experiments from the past have shown this behavior, but you are saying this particular experiment you showed in this issue is not following that? However, the loss curves you posted here follow the same trend and the only difference is the variation is the loss bumps.
Hey again @dinhanhx I just want to make sure I have all the information here. So is the explanation consistent with all your experiments and the concerns resolved? Or is there still something unclear? Please let me know so we can either close the ticket or look into it if something is still not working.
So is the explanation consistent with all your experiments and the concerns resolved?
Yes. Everything is clear, I suppose.
Bug description
As the title said, the loss values are different when resuming from ckpt vs training without interruption.
First, run the code below without interruption. Second, run the code again, wait till a certain step, kill it Finally, run the code again with the ckpt
What version are you seeing the problem on?
v2.0
How to reproduce the bug
Error messages and logs
Environment
Current environment
* CUDA: - GPU: - Tesla P100-PCIE-16GB - available: True - version: 11.8 * Lightning: - lightning: 2.0.5 - lightning-cloud: 0.5.37 - lightning-utilities: 0.9.0 - pytorch-ignite: 0.4.12 - pytorch-lightning: 2.0.4 - torch: 2.0.0 - torchaudio: 2.0.1 - torchdata: 0.6.0 - torchinfo: 1.8.0 - torchmetrics: 1.0.0 - torchtext: 0.15.1 - torchvision: 0.15.1 * Packages: - absl-py: 1.4.0 - accelerate: 0.20.3 - access: 1.1.9 - affine: 2.4.0 - aiobotocore: 2.5.2 - aiofiles: 22.1.0 - aiohttp: 3.8.4 - aiohttp-cors: 0.7.0 - aioitertools: 0.11.0 - aiorwlock: 1.3.0 - aiosignal: 1.3.1 - aiosqlite: 0.19.0 - albumentations: 1.3.1 - alembic: 1.11.1 - altair: 5.0.1 - annoy: 1.17.3 - ansiwrap: 0.8.4 - anyio: 3.7.0 - apache-beam: 2.46.0 - aplus: 0.11.0 - appdirs: 1.4.4 - argon2-cffi: 21.3.0 - argon2-cffi-bindings: 21.2.0 - array-record: 0.4.0 - arrow: 1.2.3 - arviz: 0.12.1 - astroid: 2.15.6 - astropy: 5.3.1 - asttokens: 2.2.1 - astunparse: 1.6.3 - async-timeout: 4.0.2 - atpublic: 4.0 - attrs: 23.1.0 - audioread: 3.0.0 - autopep8: 2.0.2 - babel: 2.12.1 - backcall: 0.2.0 - backoff: 2.2.1 - backports.functools-lru-cache: 1.6.4 - bayesian-optimization: 1.4.3 - bayespy: 0.5.26 - beatrix-jupyterlab: 2023.621.222118 - beautifulsoup4: 4.12.2 - bidict: 0.22.1 - biopython: 1.81 - blake3: 0.2.1 - bleach: 6.0.0 - blessed: 1.20.0 - blinker: 1.6.2 - blis: 0.7.9 - blosc2: 2.0.0 - bokeh: 3.1.1 - boltons: 23.0.0 - boruta: 0.3 - boto3: 1.26.100 - botocore: 1.29.161 - bq-helper: 0.4.1 - bqplot: 0.12.39 - branca: 0.6.0 - brewer2mpl: 1.4.1 - brotlipy: 0.7.0 - cachetools: 5.3.0 - cartopy: 0.21.1 - catalogue: 2.0.8 - catalyst: 22.4 - catboost: 1.2 - category-encoders: 2.6.1 - certifi: 2023.5.7 - cesium: 0.12.1 - cffi: 1.15.1 - cftime: 1.6.2 - charset-normalizer: 3.1.0 - chex: 0.1.81 - cleverhans: 4.0.0 - click: 8.1.4 - click-plugins: 1.1.1 - cligj: 0.7.2 - cloud-tpu-client: 0.10 - cloud-tpu-profiler: 2.4.0 - cloudpickle: 2.2.1 - cmaes: 0.9.1 - cmdstanpy: 1.1.0 - cmudict: 1.0.13 - colorama: 0.4.6 - colorcet: 3.0.1 - colorful: 0.5.5 - colorlog: 6.7.0 - colorlover: 0.3.0 - comm: 0.1.3 - conda: 23.5.0 - conda-content-trust: 0+unknown - conda-package-handling: 2.0.2 - conda-package-streaming: 0.8.0 - confection: 0.1.0 - contextily: 1.3.0 - contourpy: 1.1.0 - convertdate: 2.4.0 - crcmod: 1.7 - croniter: 1.4.1 - cryptography: 41.0.1 - cubinlinker: 0.3.0 - cuda-python: 11.8.2 - cudf: 23.6.1 - cufflinks: 0.17.3 - cuml: 23.6.0 - cupy: 12.1.0 - cvxcanon: 0.1.2 - cycler: 0.11.0 - cymem: 2.0.7 - cysignals: 1.11.2 - cython: 0.29.35 - cytoolz: 0.12.0 - daal: 2023.1.1 - daal4py: 2023.1.1 - dacite: 1.8.1 - dask: 2023.7.0 - dask-cuda: 23.6.0 - dask-cudf: 23.6.1 - dataclasses-json: 0.5.9 - datasets: 2.1.0 - datashader: 0.15.1 - datashape: 0.5.2 - datatile: 1.0.3 - dateutils: 0.6.12 - db-dtypes: 1.1.1 - deap: 1.3.3 - debugpy: 1.6.7 - decorator: 5.1.1 - deepdiff: 6.3.1 - defusedxml: 0.7.1 - delorean: 1.0.0 - deprecat: 2.1.1 - deprecated: 1.2.14 - deprecation: 2.1.0 - descartes: 1.1.0 - dill: 0.3.6 - dipy: 1.7.0 - distlib: 0.3.6 - distributed: 2023.3.2.1 - dm-tree: 0.1.8 - docker: 6.1.3 - docker-pycreds: 0.4.0 - docopt: 0.6.2 - docstring-parser: 0.15 - docstring-to-markdown: 0.12 - docutils: 0.20.1 - earthengine-api: 0.1.358 - easydict: 1.10 - easyocr: 1.7.0 - ecos: 2.0.12 - eli5: 0.13.0 - emoji: 2.6.0 - en-core-web-lg: 3.5.0 - en-core-web-sm: 3.5.0 - entrypoints: 0.4 - ephem: 4.1.4 - esda: 2.4.3 - essentia: 2.1b6.dev1034 - et-xmlfile: 1.1.0 - etils: 1.3.0 - exceptiongroup: 1.1.1 - executing: 1.2.0 - explainable-ai-sdk: 1.3.3 - fastai: 2.7.12 - fastapi: 0.98.0 - fastavro: 1.7.4 - fastcore: 1.5.29 - fastdownload: 0.0.7 - fasteners: 0.18 - fastjsonschema: 2.17.1 - fastprogress: 1.0.3 - fastrlock: 0.8 - fasttext: 0.9.2 - fbpca: 1.0 - feather-format: 0.4.1 - featuretools: 1.26.0 - filelock: 3.12.2 - fiona: 1.9.4.post1 - fitter: 1.5.2 - flake8: 6.0.0 - flashtext: 2.7 - flask: 2.3.2 - flatbuffers: 23.5.26 - flax: 0.7.0 - flit-core: 3.9.0 - folium: 0.14.0 - fonttools: 4.40.0 - fqdn: 1.5.1 - frozendict: 2.3.8 - frozenlist: 1.3.3 - fsspec: 2023.6.0 - funcy: 2.0 - fury: 0.9.0 - future: 0.18.3 - fuzzywuzzy: 0.18.0 - gast: 0.4.0 - gatspy: 0.3 - gcsfs: 2023.6.0 - gensim: 4.3.1 - geographiclib: 2.0 - geohash: 1.0 - geojson: 3.0.1 - geopandas: 0.13.2 - geoplot: 0.5.1 - geopy: 2.3.0 - geoviews: 1.10.0 - ggplot: 0.11.5 - giddy: 2.3.4 - gitdb: 4.0.10 - gitpython: 3.1.31 - google-api-core: 2.11.1 - google-api-python-client: 2.92.0 - google-apitools: 0.5.31 - google-auth: 2.20.0 - google-auth-httplib2: 0.1.0 - google-auth-oauthlib: 1.0.0 - google-cloud-aiplatform: 0.6.0a1 - google-cloud-artifact-registry: 1.8.1 - google-cloud-automl: 1.0.1 - google-cloud-bigquery: 2.34.4 - google-cloud-bigtable: 1.7.3 - google-cloud-core: 2.3.2 - google-cloud-datastore: 2.16.1 - google-cloud-dlp: 3.12.1 - google-cloud-language: 2.10.1 - google-cloud-monitoring: 2.15.0 - google-cloud-pubsub: 2.17.1 - google-cloud-pubsublite: 1.8.2 - google-cloud-recommendations-ai: 0.7.1 - google-cloud-resource-manager: 1.10.1 - google-cloud-spanner: 3.36.0 - google-cloud-storage: 1.44.0 - google-cloud-translate: 3.11.2 - google-cloud-videointelligence: 2.11.3 - google-cloud-vision: 2.8.0 - google-crc32c: 1.5.0 - google-pasta: 0.2.0 - google-resumable-media: 2.5.0 - googleapis-common-protos: 1.59.1 - gplearn: 0.4.2 - gpustat: 1.0.0 - gpxpy: 1.5.0 - graphviz: 0.20.1 - greenlet: 2.0.2 - grpc-google-iam-v1: 0.12.6 - grpcio: 1.51.3 - grpcio-status: 1.48.2 - gviz-api: 1.10.0 - gym: 0.26.2 - gym-notices: 0.0.8 - gymnasium: 0.26.3 - gymnasium-notices: 0.0.1 - h11: 0.14.0 - h2o: 3.42.0.1 - h5py: 3.9.0 - haversine: 2.8.0 - hdfs: 2.7.0 - hep-ml: 0.7.2 - hijri-converter: 2.3.1 - hmmlearn: 0.3.0 - holidays: 0.24 - holoviews: 1.16.2 - hpsklearn: 0.1.0 - html5lib: 1.1 - htmlmin: 0.1.12 - httplib2: 0.21.0 - httptools: 0.6.0 - huggingface-hub: 0.16.4 - humanize: 4.7.0 - hunspell: 0.5.5 - husl: 4.0.3 - hydra-slayer: 0.4.1 - hyperopt: 0.2.7 - hypertools: 0.8.0 - ibis-framework: 6.0.0 - idna: 3.4 - igraph: 0.10.5 - imagecodecs: 2023.7.4 - imagehash: 4.3.1 - imageio: 2.31.1 - imbalanced-learn: 0.10.1 - imgaug: 0.4.0 - implicit: 0.5.2 - importlib-metadata: 6.8.0 - importlib-resources: 5.12.0 - inequality: 1.0.0 - iniconfig: 2.0.0 - inquirer: 3.1.3 - ipydatawidgets: 4.3.5 - ipykernel: 6.23.3 - ipyleaflet: 0.17.3 - ipympl: 0.7.0 - ipython: 8.14.0 - ipython-genutils: 0.2.0 - ipython-sql: 0.5.0 - ipyvolume: 0.6.3 - ipyvue: 1.9.2 - ipyvuetify: 1.8.10 - ipywebrtc: 0.6.0 - ipywidgets: 7.7.1 - isoduration: 20.11.0 - isort: 5.12.0 - isoweek: 1.3.3 - itsdangerous: 2.1.2 - janome: 0.5.0 - jaraco.classes: 3.2.3 - jax: 0.4.13 - jaxlib: 0.4.13+cuda11.cudnn86 - jedi: 0.18.2 - jeepney: 0.8.0 - jieba: 0.42.1 - jinja2: 3.1.2 - jmespath: 1.0.1 - joblib: 1.3.0 - json5: 0.9.14 - jsonpatch: 1.32 - jsonpointer: 2.0 - jsonschema: 4.17.3 - jupyter-client: 8.3.0 - jupyter-console: 6.6.3 - jupyter-core: 5.3.1 - jupyter-events: 0.6.3 - jupyter-http-over-ws: 0.0.8 - jupyter-lsp: 1.5.1 - jupyter-server: 2.7.0 - jupyter-server-fileid: 0.9.0 - jupyter-server-mathjax: 0.2.6 - jupyter-server-proxy: 4.0.0 - jupyter-server-terminals: 0.4.4 - jupyter-server-ydoc: 0.8.0 - jupyter-ydoc: 0.2.4 - jupyterlab: 3.6.5 - jupyterlab-git: 0.41.0 - jupyterlab-lsp: 4.2.0 - jupyterlab-pygments: 0.2.2 - jupyterlab-server: 2.23.0 - jupyterlab-widgets: 3.0.7 - jupytext: 1.14.6 - kaggle: 1.5.15 - kaggle-environments: 1.12.0 - keras: 2.12.0 - keras-cv: 0.5.0 - keras-tuner: 1.3.5 - keyring: 24.2.0 - keyrings.google-artifactregistry-auth: 1.1.2 - kfp: 2.0.1 - kfp-pipeline-spec: 0.2.2 - kfp-server-api: 2.0.0 - kiwisolver: 1.4.4 - kmapper: 2.0.1 - kmodes: 0.12.2 - korean-lunar-calendar: 0.3.1 - kornia: 0.6.12 - kt-legacy: 1.0.5 - kubernetes: 26.1.0 - langcodes: 3.3.0 - langid: 1.1.6 - lazy-loader: 0.2 - lazy-object-proxy: 1.9.0 - learntools: 0.3.4 - leven: 1.0.4 - levenshtein: 0.21.1 - libclang: 16.0.0 - libmambapy: 1.4.7 - libpysal: 4.7.0 - librosa: 0.10.0.post2 - lightgbm: 3.3.2 - lightning: 2.0.5 - lightning-cloud: 0.5.37 - lightning-utilities: 0.9.0 - lime: 0.2.0.1 - line-profiler: 4.0.3 - linkify-it-py: 2.0.2 - llvmlite: 0.40.1 - lml: 0.1.0 - locket: 1.0.0 - lunarcalendar: 0.0.9 - lxml: 4.9.3 - lz4: 4.3.2 - mako: 1.2.4 - mamba: 1.4.7 - mapclassify: 2.5.0 - marisa-trie: 0.8.0 - markdown: 3.4.3 - markdown-it-py: 3.0.0 - markovify: 0.9.4 - markupsafe: 2.1.3 - marshmallow: 3.19.0 - marshmallow-enum: 1.5.1 - matplotlib: 3.7.2 - matplotlib-inline: 0.1.6 - matplotlib-venn: 0.11.9 - mccabe: 0.7.0 - mdit-py-plugins: 0.4.0 - mdurl: 0.1.2 - memory-profiler: 0.61.0 - mercantile: 1.2.1 - mgwr: 2.1.2 - missingno: 0.5.2 - mistune: 0.8.4 - mizani: 0.9.2 - ml-dtypes: 0.2.0 - mlcrate: 0.2.0 - mlens: 0.2.3 - mlxtend: 0.22.0 - mmh3: 4.0.0 - mne: 1.4.2 - mnist: 0.2.2 - mock: 5.0.2 - momepy: 0.6.0 - more-itertools: 9.1.0 - mpld3: 0.5.9 - mpmath: 1.3.0 - msgpack: 1.0.5 - msgpack-numpy: 0.4.8 - multidict: 6.0.4 - multimethod: 1.9.1 - multipledispatch: 1.0.0 - multiprocess: 0.70.14 - munkres: 1.1.4 - murmurhash: 1.0.9 - mypy-extensions: 1.0.0 - nb-conda: 2.2.1 - nb-conda-kernels: 2.3.1 - nbclassic: 1.0.0 - nbclient: 0.5.13 - nbconvert: 6.4.5 - nbdime: 3.2.0 - nbformat: 5.9.0 - nest-asyncio: 1.5.6 - netcdf4: 1.6.4 - networkx: 3.1 - nibabel: 5.1.0 - nilearn: 0.10.1 - ninja: 1.11.1 - nltk: 3.2.4 - nose: 1.3.7 - notebook: 6.5.4 - notebook-executor: 0.2 - notebook-shim: 0.2.3 - numba: 0.57.1 - numexpr: 2.8.4 - numpy: 1.25.0 - nvidia-ml-py: 11.495.46 - nvtx: 0.2.5 - oauth2client: 4.1.3 - oauthlib: 3.2.2 - objsize: 0.6.1 - odfpy: 1.4.1 - olefile: 0.46 - onnx: 1.14.0 - opencensus: 0.11.2 - opencensus-context: 0.1.3 - opencv-contrib-python: 4.8.0.74 - opencv-python: 4.8.0.74 - opencv-python-headless: 4.8.0.74 - openpyxl: 3.1.2 - openslide-python: 1.2.0 - opentelemetry-api: 1.18.0 - opentelemetry-exporter-otlp: 1.18.0 - opentelemetry-exporter-otlp-proto-common: 1.18.0 - opentelemetry-exporter-otlp-proto-grpc: 1.18.0 - opentelemetry-exporter-otlp-proto-http: 1.18.0 - opentelemetry-proto: 1.18.0 - opentelemetry-sdk: 1.18.0 - opentelemetry-semantic-conventions: 0.39b0 - opt-einsum: 3.3.0 - optax: 0.1.5 - optuna: 3.2.0 - orbax-checkpoint: 0.2.7 - ordered-set: 4.1.0 - orderedmultidict: 1.0.1 - orjson: 3.9.1 - ortools: 9.4.1874 - osmnx: 1.1.1 - overrides: 7.3.1 - packaging: 21.3 - pandas: 2.0.2 - pandas-datareader: 0.10.0 - pandas-profiling: 3.6.6 - pandas-summary: 0.2.0 - pandasql: 0.7.3 - pandocfilters: 1.5.0 - panel: 1.2.0 - papermill: 2.4.0 - param: 1.13.0 - parso: 0.8.3 - parsy: 2.1 - partd: 1.4.0 - path: 16.7.1 - path.py: 12.5.0 - pathos: 0.3.0 - pathtools: 0.1.2 - pathy: 0.10.1 - patsy: 0.5.3 - pdf2image: 1.16.3 - pexpect: 4.8.0 - phik: 0.12.3 - pickleshare: 0.7.5 - pillow: 9.5.0 - pip: 23.1.2 - pkgutil-resolve-name: 1.3.10 - platformdirs: 3.8.1 - plotly: 5.15.0 - plotly-express: 0.4.1 - plotnine: 0.10.1 - pluggy: 1.0.0 - pointpats: 2.3.0 - polars: 0.18.6 - polyglot: 16.7.4 - pooch: 1.6.0 - pox: 0.3.2 - ppca: 0.0.4 - ppft: 1.7.6.6 - preprocessing: 0.1.13 - preshed: 3.0.8 - prettytable: 3.8.0 - progressbar2: 4.2.0 - prometheus-client: 0.17.0 - promise: 2.3 - prompt-toolkit: 3.0.38 - pronouncing: 0.2.0 - prophet: 1.1.1 - proto-plus: 1.22.3 - protobuf: 4.21.12 - psutil: 5.9.5 - ptxcompiler: 0.8.1 - ptyprocess: 0.7.0 - pudb: 2022.1.3 - pulp: 2.7.0 - pure-eval: 0.2.2 - py-cpuinfo: 9.0.0 - py-lz4framed: 0.14.0 - py-spy: 0.3.14 - py4j: 0.10.9.7 - pyaml: 23.7.0 - pyarabic: 0.6.15 - pyarrow: 11.0.0 - pyasn1: 0.5.0 - pyasn1-modules: 0.3.0 - pyastronomy: 0.19.0 - pybind11: 2.10.4 - pyclipper: 1.3.0.post4 - pycodestyle: 2.10.0 - pycolmap: 0.4.0 - pycosat: 0.6.4 - pycparser: 2.21 - pycryptodome: 3.18.0 - pyct: 0.5.0 - pycuda: 2022.2.2 - pydantic: 1.10.10 - pydegensac: 0.1.2 - pydicom: 2.4.1 - pydocstyle: 6.3.0 - pydot: 1.4.2 - pydub: 0.25.1 - pyemd: 1.0.0 - pyerfa: 2.0.0.3 - pyexcel-io: 0.6.6 - pyexcel-ods: 0.6.0 - pyfasttext: 0.4.6 - pyflakes: 3.0.1 - pygltflib: 1.15.6 - pygments: 2.15.1 - pyjwt: 2.7.0 - pykalman: 0.9.5 - pyldavis: 3.2.2 - pylibraft: 23.6.2 - pylint: 2.17.4 - pymc3: 3.11.5 - pymeeus: 0.5.12 - pymongo: 3.13.0 - pympler: 1.0.1 - pynndescent: 0.5.10 - pynvml: 11.4.1 - pynvrtc: 9.2 - pyocr: 0.8.3 - pyopenssl: 23.2.0 - pyparsing: 3.1.0 - pypdf: 3.12.0 - pyproj: 3.6.0 - pyrsistent: 0.19.3 - pysal: 23.1 - pyshp: 2.3.1 - pysocks: 1.7.1 - pytesseract: 0.3.10 - pytest: 7.4.0 - python-bidi: 0.4.2 - python-dateutil: 2.8.2 - python-dotenv: 1.0.0 - python-editor: 1.0.4 - python-igraph: 0.10.5 - python-json-logger: 2.0.7 - python-levenshtein: 0.21.1 - python-louvain: 0.16 - python-lsp-jsonrpc: 1.0.0 - python-lsp-server: 1.7.4 - python-multipart: 0.0.6 - python-slugify: 8.0.1 - python-utils: 3.7.0 - pythreejs: 2.4.2 - pytoolconfig: 1.2.5 - pytools: 2023.1 - pytorch-ignite: 0.4.12 - pytorch-lightning: 2.0.4 - pytz: 2023.3 - pyu2f: 0.1.5 - pyupset: 0.1.1.post7 - pyviz-comms: 2.3.2 - pywavelets: 1.4.1 - pyyaml: 6.0 - pyzmq: 25.1.0 - qgrid: 1.3.1 - qtconsole: 5.4.3 - qtpy: 2.3.1 - quantecon: 0.7.1 - quantities: 0.14.1 - qudida: 0.0.4 - raft-dask: 23.6.2 - randomgen: 1.23.1 - rapidfuzz: 3.1.1 - rasterio: 1.3.8 - rasterstats: 0.19.0 - ray: 2.5.1 - ray-cpp: 2.5.1 - readchar: 4.0.5 - regex: 2023.6.3 - requests: 2.31.0 - requests-oauthlib: 1.3.1 - requests-toolbelt: 0.10.1 - responses: 0.18.0 - retrying: 1.3.4 - rfc3339-validator: 0.1.4 - rfc3986-validator: 0.1.1 - rgf-python: 3.12.0 - rich: 13.4.2 - rmm: 23.6.0 - rope: 1.9.0 - rsa: 4.9 - rtree: 1.0.1 - ruamel-yaml-conda: 0.15.100 - ruamel.yaml: 0.17.32 - ruamel.yaml.clib: 0.2.7 - s2sphere: 0.2.5 - s3fs: 2023.6.0 - s3transfer: 0.6.1 - safetensors: 0.3.1 - scattertext: 0.1.19 - scikit-image: 0.21.0 - scikit-learn: 1.2.2 - scikit-learn-intelex: 2023.1.1 - scikit-multilearn: 0.2.0 - scikit-optimize: 0.9.0 - scikit-plot: 0.3.7 - scikit-surprise: 1.1.3 - scipy: 1.11.1 - seaborn: 0.12.2 - secretstorage: 3.3.3 - segment-anything: 1.0 - segregation: 2.4.2 - semver: 3.0.1 - send2trash: 1.8.2 - sentencepiece: 0.1.99 - sentry-sdk: 1.27.1 - setproctitle: 1.3.2 - setuptools: 59.8.0 - setuptools-git: 1.2 - setuptools-scm: 7.1.0 - shap: 0.42.0 - shapely: 1.8.5.post1 - shellingham: 1.5.1 - simpervisor: 1.0.0 - simpleitk: 2.2.1 - simplejson: 3.19.1 - six: 1.16.0 - sklearn-pandas: 2.2.0 - slicer: 0.0.7 - smart-open: 6.3.0 - smhasher: 0.150.1 - smmap: 5.0.0 - sniffio: 1.3.0 - snowballstemmer: 2.2.0 - snuggs: 1.4.7 - sortedcontainers: 2.4.0 - soundfile: 0.12.1 - soupsieve: 2.4.1 - soxr: 0.3.5 - spacy: 3.5.4 - spacy-legacy: 3.0.12 - spacy-loggers: 1.0.4 - spaghetti: 1.7.4 - spectral: 0.23.1 - spglm: 1.0.8 - sphinx-rtd-theme: 0.2.4 - spint: 1.0.7 - splot: 1.1.5.post1 - spopt: 0.5.0 - spreg: 1.3.2 - spvcm: 0.3.0 - sqlalchemy: 2.0.17 - sqlglot: 17.3.0 - sqlparse: 0.4.4 - squarify: 0.4.3 - srsly: 2.4.6 - stack-data: 0.6.2 - starlette: 0.27.0 - starsessions: 1.3.0 - statsmodels: 0.14.0 - stemming: 1.0.1 - stop-words: 2018.7.23 - stopit: 1.1.2 - stumpy: 1.11.1 - sympy: 1.12 - tables: 3.8.0 - tabulate: 0.9.0 - tangled-up-in-unicode: 0.2.0 - tbb: 2021.9.0 - tblib: 1.7.0 - tenacity: 8.2.2 - tensorboard: 2.12.3 - tensorboard-data-server: 0.7.1 - tensorboard-plugin-profile: 2.13.0 - tensorboardx: 2.6 - tensorflow: 2.12.0 - tensorflow-addons: 0.20.0 - tensorflow-cloud: 0.1.16 - tensorflow-datasets: 4.9.2 - tensorflow-decision-forests: 1.4.0 - tensorflow-estimator: 2.12.0 - tensorflow-gcs-config: 2.12.0 - tensorflow-hub: 0.12.0 - tensorflow-io: 0.31.0 - tensorflow-io-gcs-filesystem: 0.31.0 - tensorflow-metadata: 0.14.0 - tensorflow-probability: 0.20.1 - tensorflow-serving-api: 2.12.1 - tensorflow-text: 2.12.1 - tensorflow-transform: 0.14.0 - tensorflowjs: 3.15.0 - tensorpack: 0.11 - tensorstore: 0.1.40 - termcolor: 2.3.0 - terminado: 0.17.1 - testpath: 0.6.0 - text-unidecode: 1.3 - textblob: 0.17.1 - texttable: 1.6.7 - textwrap3: 0.9.2 - theano: 1.0.5 - theano-pymc: 1.1.2 - thinc: 8.1.10 - threadpoolctl: 3.1.0 - tifffile: 2023.4.12 - timm: 0.9.2 - tinycss2: 1.2.1 - tobler: 0.10 - tokenizers: 0.13.3 - toml: 0.10.2 - tomli: 2.0.1 - tomlkit: 0.11.8 - toolz: 0.12.0 - torch: 2.0.0 - torchaudio: 2.0.1 - torchdata: 0.6.0 - torchinfo: 1.8.0 - torchmetrics: 1.0.0 - torchtext: 0.15.1 - torchvision: 0.15.1 - tornado: 6.3.2 - tpot: 0.12.0 - tqdm: 4.65.0 - traceml: 1.0.8 - traitlets: 5.9.0 - traittypes: 0.2.1 - transformers: 4.30.2 - treelite: 3.2.0 - treelite-runtime: 3.2.0 - trueskill: 0.4.5 - tsfresh: 0.20.1 - typeguard: 2.13.3 - typer: 0.9.0 - typing-extensions: 4.6.3 - typing-inspect: 0.9.0 - typing-utils: 0.1.0 - tzdata: 2023.3 - tzlocal: 5.0.1 - uc-micro-py: 1.0.2 - ucx-py: 0.32.0 - ujson: 5.8.0 - umap-learn: 0.5.3 - unicodedata2: 15.0.0 - unidecode: 1.3.6 - update-checker: 0.18.0 - uri-template: 1.3.0 - uritemplate: 3.0.1 - urllib3: 1.26.16 - urwid: 2.1.2 - urwid-readline: 0.13 - uvicorn: 0.22.0 - uvloop: 0.17.0 - vaex: 4.16.0 - vaex-astro: 0.9.3 - vaex-core: 4.16.1 - vaex-hdf5: 0.14.1 - vaex-jupyter: 0.8.1 - vaex-ml: 0.18.1 - vaex-server: 0.8.1 - vaex-viz: 0.5.4 - vecstack: 0.4.0 - virtualenv: 20.21.0 - visions: 0.7.5 - vowpalwabbit: 9.8.0 - vtk: 9.2.6 - wand: 0.6.11 - wandb: 0.15.5 - wasabi: 1.1.2 - watchfiles: 0.19.0 - wavio: 0.0.7 - wcwidth: 0.2.6 - webcolors: 1.13 - webencodings: 0.5.1 - websocket-client: 1.6.1 - websockets: 11.0.3 - werkzeug: 2.3.6 - wfdb: 4.1.2 - whatthepatch: 1.0.5 - wheel: 0.40.0 - widgetsnbextension: 3.6.4 - witwidget: 1.8.1 - woodwork: 0.24.0 - wordbatch: 1.4.9 - wordcloud: 1.9.2 - wordsegment: 1.3.1 - wrapt: 1.14.1 - wurlitzer: 3.0.3 - xarray: 2023.6.0 - xarray-einstats: 0.5.1 - xgboost: 1.7.6 - xvfbwrapper: 0.2.9 - xxhash: 3.2.0 - xyzservices: 2023.5.0 - y-py: 0.5.9 - yapf: 0.40.1 - yarl: 1.9.2 - ydata-profiling: 4.3.1 - yellowbrick: 1.5 - ypy-websocket: 0.8.2 - zict: 3.0.0 - zipp: 3.15.0 - zstandard: 0.19.0 * System: - OS: Linux - architecture: - 64bit - - processor: x86_64 - python: 3.10.12 - release: 5.15.109+ - version: #1 SMP Thu Jul 13 10:52:19 UTC 2023More info
I provide the kaggle notebook to produce quickly
https://www.kaggle.com/inhanhv/resume-not-same-with-from-scratch-training
I also google the behaviour and find related post:
https://lightning.ai/forums/t/resuming-training-gives-different-model-result-weights/2677