모델 저장이 되지 않습니다.

nsml.save시 minio.error.IncompleteBody: IncompleteBody: message: You did not provide the number of bytes specified by the Content-Length HTTP header 와 함께 모델 저장이 되지 않습니다.

두번째로 모델을 저장했을 때 다음과 같은 오류가 나며, 저장하는 주기를 50step으로 변경시 오류가 발생하지 않습니다.

Informations

CLI

OS (Mac, Windows, Linux, etc) and version:
client version(please show nsml --version):
Full command line:

WEB

Brower (chrome, firfox, etc):
URL:

What is your NSML login ID?

codertimo

What is name of session in problem? (bug message or screenshot) team_89/16_tcls_movie/418

Steps to reproduce the problem nsml run -a "--task=movie" -d 16_tcls_movie -g 1

What is the expected behavior?

Traceback (most recent call last):
  File "main.py", line 65, in <module>
    main(trainer, local=locals())
  File "main.py", line 39, in main
    trainer.nsml_train(train_text_path, train_label_path, val_text_path, val_label_path)
  File "/app/tasks/movie/trainer.py", line 350, in nsml_train
    self.train(train_data_loader)
  File "/app/pnlp/train/trainer/trainer.py", line 52, in train
    train_output = self.train_one_epoch(train_data_loader, epoch)
  File "/app/pnlp/train/trainer/trainer.py", line 123, in train_one_epoch
    return self._run_one_epoch(self.train_step, data_loader, epoch, True)
  File "/app/tasks/movie/trainer.py", line 59, in _run_one_epoch
    return super()._run_one_epoch(step_function, data_loader, epoch, is_train)
  File "/app/pnlp/train/trainer/trainer.py", line 90, in _run_one_epoch
    step_output = step_function(model_input)
  File "/app/tasks/movie/trainer.py", line 72, in train_step
    model_output["loss"]
  File "/app/tasks/movie/trainer.py", line 123, in get_step_metric
    nsml.save()
  File "/app/nsml/client.py", line 434, in save
    minio_client.fput_object(MINIO_SESSION, minio_path, file)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 534, in fput_object
    content_type, metadata)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 750, in put_object
    metadata=metadata)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 1424, in _do_put_object
    content_sha256=sha256_hex
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 1765, in _url_open
    object_name).get_exception()
minio.error.IncompleteBody: IncompleteBody: message: You did not provide the number of bytes specified by the Content-Length HTTP header

Suggested solutions

ai-starthon / AI_Starthon2019

모델 저장이 되지 않습니다. #242

CLI

WEB