search

mapbox / robosat

Semantic segmentation on aerial and satellite imagery. Extracts features such as: buildings, parking lots, roads, water, clouds
MIT License
2.02k stars 383 forks source link

OSError: image file is truncated while running training #196

Closed himanshu-sikaria closed 4 years ago

himanshu-sikaria commented 4 years ago

Get the following error while running training in GPU. 

Batch Size:      16
Image Size:      256
Learning Rate:   0.0001
Loss function:   mIoU
Weights :        [1.574982, 7.019227]
 File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/src/app/robosat/tools/__main__.py", line 58, in <module>
    args.func(args)
  File "/usr/src/app/robosat/tools/train.py", line 129, in main
    train_hist = train(train_loader, num_classes, device, net, optimizer, criterion)
  File "/usr/src/app/robosat/tools/train.py", line 171, in train
    for images, masks, tiles in tqdm(loader, desc="Train", unit="batch", ascii=True):
  File "/opt/venv/lib/python3.6/site-packages/tqdm/_tqdm.py", line 1005, in __iter__
    for obj in iterable:
  File "/opt/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/opt/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
OSError: Traceback (most recent call last):
  File "/opt/venv/lib/python3.6/site-packages/PIL/ImageFile.py", line 224, in load
    s = read(self.decodermaxblock)
  File "/opt/venv/lib/python3.6/site-packages/PIL/PngImagePlugin.py", line 652, in load_read
    cid, pos, length = self.png.read()
  File "/opt/venv/lib/python3.6/site-packages/PIL/PngImagePlugin.py", line 122, in read
    length = i32(s)
  File "/opt/venv/lib/python3.6/site-packages/PIL/_binary.py", line 77, in i32be
    return unpack_from(">I", c, o)[0]
struct.error: unpack_from requires a buffer of at least 4 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/venv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/src/app/robosat/datasets.py", line 76, in __getitem__
    images, mask = self.joint_transform(images, mask)
  File "/usr/src/app/robosat/transforms.py", line 82, in __call__
    images, mask = transform(images, mask)
  File "/usr/src/app/robosat/transforms.py", line 119, in __call__
    images = [self.image_transform(v) for v in images]
  File "/usr/src/app/robosat/transforms.py", line 119, in <listcomp>
    images = [self.image_transform(v) for v in images]
  File "/usr/src/app/robosat/transforms.py", line 54, in __call__
    return image.convert(self.mode)
  File "/opt/venv/lib/python3.6/site-packages/PIL/Image.py", line 912, in convert
    self.load()
  File "/opt/venv/lib/python3.6/site-packages/PIL/ImageFile.py", line 230, in load
    raise IOError("image file is truncated")
OSError: image file is truncated
prakharcode commented 4 years ago

Hey @himanshu-sikaria, this is more of a problem with HTTP library of the os, HTTP servers put a limit on the size of the entity-body that is accepted.

Have a look here.