Error when converting DIV2K dataset to t7 files

[subzerofun]

I've placed the DIV2K dataset in a custom location, changed the cmd to convert png -> t7 accordingly but before any conversion happens i get this error:

th png_to_t7.lua -apath /mycustomfolder/div2k -dataset DIV2K

Converting /mycustomfolder/div2k/DIV2K_train_HR
/Users/me/torch/install/bin/luajit: png_to_t7.lua:113: bad argument #1 to 'sort' (table expected, got nil)
stack traceback:
    [C]: in function 'sort'
    png_to_t7.lua:113: in main chunk
    [C]: in function 'dofile'
    ...me/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
    [C]: at 0x0100b54360

I've got the following folders of the DIV2K dataset:

DIV2K_train_HR
DIV2K_train_LR_bicubic
DIV2K_train_LR_unknown
DIV2K_valid_HR
DIV2K_valid_LR_bicubic

PS: The link to your server isn't working anymore: http://cv.snu.ac.kr/research/EDSR/DIV2K.tar Could you please explain how to download the images from https://competitions.codalab.org/competitions/16303? There is no direct download link or data site.

[limbee]

The -apath setting seems to be wrong. You should set it to be the parent of DIV2K directory. Also, now I've fixed the DIV2K link, it will work :)

[subzerofun]

Thanks for the quick reply! I'm downloading the 7 GB now and will try again once i have extracted them. I think i was also missing a lof of files, my current DIV2K-folder only has around 4 GB of data.

BTW, i'm currently testing a SRGAN super resolution project (training will take another two days...): https://github.com/zsdonghao/SRGAN

Have you also compared your results against a similar approach? I have seen the comparison methods in the paper, but SRGAN apparently was not chosen (maybe not released when the paper was written?).

[limbee]

Since we originally developed the code to participate in the SR challenge, GAN approach was not considered.

However, I've just pushed my quick commit to the branch 'feature/srgan' so that you can refer to it. With -vgg, -adv option, you can apply VGG loss (Johnson 2016, Ledig 2017) and GAN loss (Ledig 2017), respectively. I checked it running, but some features e.g. multi-GPU or logging were not fully tested. I'll merge this branch after some tests.

I brought these codes from my previous SRGAN implementation. The repo is now private and deprecated, but I found someone has forked it when it was once public... So you can find it here: https://github.com/anthonyyuan/SRGAN.

[saeed-anwar]

Hi,

I am unable to download the dataset DIV2K. It stops downloading after 700mb.

Also is it possible for you to share the lmdb files for training so that we don't have to waste our resources for creating lmdb files again.

Waiting for your kind response.

[limbee]

I've just tested 2 methods: chrome download by clicking the link, and 'wget' command in Linux. Both of them worked fine. Could you tell me what kind of method did you use? Also, please check the address (http://cv.snu.ac.kr/research/EDSR/DIV2K.tar).

We didn't utilize any DBs. Clearly using LMDB would be a much better idea than loading the whole torch7 formatted dataset into memory every time. We would be grateful if you could share your code someday.

[saeed-anwar]

I tried downloading with chrome as well as firefox. but in both cases the download fails around 700MB.

I am trying to download again from the above link, if it works i will let you know.

[saeed-anwar]

Download works with wget, however fails with chrome or firefox.

[limbee]

Thank you for sharing this. Sometimes the browsers seem to fail at downloading large files.