search

tensorlayer / SRGAN

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
https://github.com/tensorlayer/tensorlayerx
3.32k stars 813 forks source link

OutOfMemory With huge datasets. #186

Open mcDandy opened 5 years ago

mcDandy commented 5 years ago

I have huge dataset (about 30k images with size 1920x1080). It is possible to load only parts of dataset to avoid this?

2019-10-19 17:05:57.408663: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 Exception in thread threading_and_return: Traceback (most recent call last): File "C:\Program Files\Python37\lib\threading.py", line 926, in _bootstrap_inner self.run() File "C:\Program Files\Python37\lib\threading.py", line 870, in run self._target(*self._args, self._kwargs) File "C:\Program Files\Python37\lib\site-packages\tensorlayer\prepro.py", line 195, in apply_fn results[i] = fn(data, kwargs) File "C:\Program Files\Python37\lib\site-packages\tensorlayer\visualize.py", line 50, in read_image return imageio.imread(os.path.join(path, image)) File "C:\Program Files\Python37\lib\site-packages\imageio\core\functions.py", line 221, in imread reader = read(uri, format, "i", kwargs) File "C:\Program Files\Python37\lib\site-packages\imageio\core\functions.py", line 143, in get_reader return format.get_reader(request) File "C:\Program Files\Python37\lib\site-packages\imageio\core\format.py", line 164, in get_reader return self.Reader(self, request) File "C:\Program Files\Python37\lib\site-packages\imageio\core\format.py", line 214, in init self._open(self.request.kwargs.copy()) File "C:\Program Files\Python37\lib\site-packages\imageio\plugins\pillow.py", line 293, in _open return PillowFormat.Reader._open(self, pilmode=pilmode, as_gray=as_gray) File "C:\Program Files\Python37\lib\site-packages\imageio\plugins\pillow.py", line 137, in _open pil_try_read(self._im) File "C:\Program Files\Python37\lib\site-packages\imageio\plugins\pillow.py", line 518, in pil_try_read im.getdata()[0] File "C:\Program Files\Python37\lib\site-packages\PIL\Image.py", line 1304, in getdata self.load() File "C:\Program Files\Python37\lib\site-packages\PIL\ImageFile.py", line 209, in load self.load_prepare() File "C:\Program Files\Python37\lib\site-packages\PIL\PngImagePlugin.py", line 649, in load_prepare ImageFile.ImageFile.load_prepare(self) File "C:\Program Files\Python37\lib\site-packages\PIL\ImageFile.py", line 279, in load_prepare self.im = Image.core.new(self.mode, self.size) MemoryError

zsdonghao commented 5 years ago

load the images batch by batch then..

mcDandy commented 5 years ago

What I need to do in the code? Sorry I do not understand python and the libray rmuch.

mcDandy commented 4 years ago

Can anyone please show me how to do it? I have only 16GB Ram (4GB vRAM). Dataset is 1.6GB png (RAW: 9.08GB).