search

cgohlke / imagecodecs

Image transformation, compression, and decompression codecs
https://pypi.org/project/imagecodecs
BSD 3-Clause "New" or "Revised" License
111 stars 21 forks source link

Error in importing imagecodecs into colab to run program:imcd_lzw_decode returned IMCD_LZW_CORRUPT #13

Closed Mnsy-Syl closed 3 years ago

Mnsy-Syl commented 3 years ago

I import imagecodecs package into colab to read all TIF images in a folder. When I read the 231st TIF image, I will report an error: IMCD lzw decode returned IMCD LZW .However, when reading a small number of images, no error will be reported. What is the reason for this?

At the same time, I modified the source code of detectron2 to process the data in TIF format, in which I also used the imagecodecs package. However, the training process can only be iterated about 150 times, and then an error is reported: imagecodecs/ imcd.pyx in imagecodecs. imcd.ImcdError . init () TypeError: init () takes exactly 3 positional arguments (2 given) Are there any reliable suggestions?

Czaki commented 3 years ago

Which version of imagecodecs did you use?

cgohlke commented 3 years ago

Can you please post the full tracebacks and a short, self contained, correct, example.

Re IMCD_LZW_CORRUPT: Corrupted LZW strips are common.

Re TypeError: I don't see any place in the code where ImcdError is initialized with arguments missing.

Mnsy-Syl commented 3 years ago

Can you please post the full tracebacks and a short, self contained, correct, example.

Re IMCD_LZW_CORRUPT: Corrupted LZW strips are common.

Re TypeError: I don't see any place in the code where ImcdError is initialized with arguments missing.

Of course! Thank you for your attention.

[02/03 13:10:14 d2.engine.train_loop]: Starting training from iteration 0
[02/03 13:10:29 d2.utils.events]:  eta: 0:14:55  iter: 19  total_loss: 2.428  loss_cls: 0.8926  loss_box_reg: 0.001231  loss_mask: 0.6916  loss_rpn_cls: 0.7705  loss_rpn_loc: 0.08812  time: 0.6259  data_time: 0.1283  lr: 1.9981e-05  max_mem: 1541M
[02/03 13:10:41 d2.utils.events]:  eta: 0:14:43  iter: 39  total_loss: 1.668  loss_cls: 0.316  loss_box_reg: 0.0006488  loss_mask: 0.6697  loss_rpn_cls: 0.5859  loss_rpn_loc: 0.08157  time: 0.6262  data_time: 0.0064  lr: 3.9961e-05  max_mem: 1541M
[02/03 13:10:53 d2.utils.events]:  eta: 0:14:31  iter: 59  total_loss: 1.03  loss_cls: 0.1443  loss_box_reg: 0.0002654  loss_mask: 0.6423  loss_rpn_cls: 0.2127  loss_rpn_loc: 0.03738  time: 0.6163  data_time: 0.0061  lr: 5.9941e-05  max_mem: 1541M
[02/03 13:11:06 d2.utils.events]:  eta: 0:14:24  iter: 79  total_loss: 0.9292  loss_cls: 0.1536  loss_box_reg: 0.0007024  loss_mask: 0.5927  loss_rpn_cls: 0.1811  loss_rpn_loc: 0.03039  time: 0.6177  data_time: 0.0055  lr: 7.9921e-05  max_mem: 1541M
[02/03 13:11:18 d2.utils.events]:  eta: 0:14:25  iter: 99  total_loss: 0.9578  loss_cls: 0.1495  loss_box_reg: 0.0009513  loss_mask: 0.5321  loss_rpn_cls: 0.1712  loss_rpn_loc: 0.02306  time: 0.6182  data_time: 0.0081  lr: 9.9901e-05  max_mem: 1541M
[02/03 13:11:31 d2.utils.events]:  eta: 0:14:23  iter: 119  total_loss: 0.9316  loss_cls: 0.1887  loss_box_reg: 0.001114  loss_mask: 0.5312  loss_rpn_cls: 0.1917  loss_rpn_loc: 0.03098  time: 0.6233  data_time: 0.0124  lr: 0.00011988  max_mem: 1541M
[02/03 13:11:45 d2.utils.events]:  eta: 0:14:10  iter: 139  total_loss: 0.8167  loss_cls: 0.1636  loss_box_reg: 0.001619  loss_mask: 0.4657  loss_rpn_cls: 0.1481  loss_rpn_loc: 0.01185  time: 0.6281  data_time: 0.0077  lr: 0.00013986  max_mem: 1541M
[02/03 13:11:58 d2.utils.events]:  eta: 0:14:09  iter: 159  total_loss: 0.8165  loss_cls: 0.1998  loss_box_reg: 0.002127  loss_mask: 0.4666  loss_rpn_cls: 0.1286  loss_rpn_loc: 0.01807  time: 0.6343  data_time: 0.0074  lr: 0.00015984  max_mem: 1541M
ERROR [02/03 13:12:12 d2.engine.train_loop]: Exception during training:

TypeError                                 Traceback (most recent call last)
<ipython-input-6-186ec9fb598d> in <module>()
     18 trainer = CocoTrainer(cfg)
     19 trainer.resume_or_load(resume=False)
---> 20 trainer.train()

8 frames
/content/drive/MyDrive/detectron2_repo/detectron2/engine/defaults.py in train(self)
    417             OrderedDict of results, if evaluation is enabled. Otherwise None.
    418         """
--> 419         super().train(self.start_iter, self.max_iter)
    420         if len(self.cfg.TEST.EXPECTED_RESULTS) and comm.is_main_process():
    421             assert hasattr(

/content/drive/MyDrive/detectron2_repo/detectron2/engine/train_loop.py in train(self, start_iter, max_iter)
    132                 for self.iter in range(start_iter, max_iter):
    133                     self.before_step()
--> 134                     self.run_step()
    135                     self.after_step()
    136                 # self.iter == max_iter can be used by `after_train` to

/content/drive/MyDrive/detectron2_repo/detectron2/engine/defaults.py in run_step(self)
    427     def run_step(self):
    428         self._trainer.iter = self.iter
--> 429         self._trainer.run_step()
    430 
    431     @classmethod

/content/drive/MyDrive/detectron2_repo/detectron2/engine/train_loop.py in run_step(self)
    220         If you want to do something with the data, you can wrap the dataloader.
    221         """
--> 222         data = next(self._data_loader_iter)
    223         data_time = time.perf_counter() - start
    224 

/content/drive/MyDrive/detectron2_repo/detectron2/data/common.py in __iter__(self)
    177 
    178     def __iter__(self):
--> 179         for d in self.dataset:
    180             w, h = d["width"], d["height"]
    181             bucket_id = 0 if w > h else 1

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
   1083             else:
   1084                 del self._task_info[idx]
-> 1085                 return self._process_data(data)
   1086 
   1087     def _try_put_index(self):

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _process_data(self, data)
   1109         self._try_put_index()
   1110         if isinstance(data, ExceptionWrapper):
-> 1111             data.reraise()
   1112         return data
   1113 

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    426             # have message field
    427             raise self.exc_type(message=msg)
--> 428         raise self.exc_type(msg)
    429 
    430 

imagecodecs/_imcd.pyx in imagecodecs._imcd.ImcdError.__init__()

TypeError: __init__() takes exactly 3 positional arguments (2 given)

Is the information I provided sufficient? The error occurs in the middle of the iteration process, which should indicate that the error report of imagecodecs occurs after being called for a period of time.

Mnsy-Syl commented 3 years ago

Which version of imagecodecs did you use?

Pip automatically installs imagecodecs-2020.5.30.

Czaki commented 3 years ago

maybe upgrade its to never version? (pip install -U imagecodecs)

Mnsy-Syl commented 3 years ago

maybe upgrade its to never version? (pip install -U imagecodecs)

Just tried, the highest version that can be installed in colab is 2020.5.30 o(╥﹏╥)o It's too bad

Mnsy-Syl commented 3 years ago

maybe upgrade its to never version? (pip install -U imagecodecs)

Just tried, the highest version that can be installed in colab is 2020.5.30 o(╥﹏╥)o It's too bad

Czaki commented 3 years ago

Which version of python did You use?

Mnsy-Syl commented 3 years ago

Which version of python did You use?

Python 3.6.9

Czaki commented 3 years ago

@cgohlke drop support for python 3.6. Could You jump to a more recent version of python?

Mnsy-Syl commented 3 years ago

@cgohlke drop support for python 3.6. Could You jump to a more recent version of python?

It seems impossible to upgrade the Python version in colab. I'll try to connect to another GPU to run my code. Thank you very much!

cgohlke commented 3 years ago

Unfortunately torch re-raises the exception and the original traceback is lost.

Also in v2020.5.30 I don't see where ImcdError is called with the wrong number of arguments.

Are you maybe running low on memory?

Mnsy-Syl commented 3 years ago

Unfortunately torch re-raises the exception and the original traceback is lost.

Also in v2020.5.30 I don't see where ImcdError is called with the wrong number of arguments.

Are you maybe running low on memory?

Thanks for reminding! Also, the Python version of coLab is 3.6.9. Should I use a higher version?

cgohlke commented 3 years ago

Not sure what can be done without being able to reproduce the issue. Maybe you could share the 231st TIF image?

Mnsy-Syl commented 3 years ago

Not sure what can be done without being able to reproduce the issue. Maybe you could share the 231st TIF image?

I tried to replace the 231st TIF image but the same error was reported. I am referring to your suggestions and will feed back the results. Thank you!

Mnsy-Syl commented 3 years ago

@cgohlke drop support for python 3.6. Could You jump to a more recent version of python?

Thank you for your suggestion! The error will no longer occur under Python3.7 @cgohlke Thank you very much for your attention and kindly advice!

cgohlke commented 3 years ago

Thanks for following up. It would be good to know why this was happening on Python 3.6...