search

lmb-freiburg / flownet2-docker

Dockerfile and runscripts for FlowNet 2.0 (estimation of optical flow)
https://lmb.informatik.uni-freiburg.de/Publications/2017/IMKDB17/
GNU General Public License v3.0
158 stars 52 forks source link

path issue #2

Closed ZikeYan closed 7 years ago

ZikeYan commented 7 years ago

I ran flownet2 for entire lists of images on the sample images successful. However, when I wrote three new .txt file for Sintel dataset, the error occurs as "img0 does not exist: /input-output/data/SINTEL/test/clean/alley_1/frame_0001.png".

I checked the .py file and found that the dockerize_filepath joins '/input-output' with the dirname I wrote. I wonder how the sample images can be run successful without join the /input-output to the path. And how can I run with my own image lists.

nikolausmayer commented 7 years ago

Hi,

/input-output is the mountpoint inside the container where the current working directory will be mounted, i.e. if data/SINTEL/test/clean/alley_1/frame_0001.png is a valid file, it is visible as /input-output/data/SINTEL/test/clean/alley_1/frame_0001.png inside the container.

Note that the data must be within the directory from which you call the run-network.sh script. I am not sure whether symlinks work.

My best guess is that your list files do not contain the correct paths. Note that the paths must be relative to $PWD, not relative to the list file.

Best, Nikolaus

ZikeYan commented 7 years ago

I checked the path and didn't find the mistake. Maybe I didn't know how docker works. I'll present the my problem as follows and wish that you could find something wrong:

The $PWD is ~/develop/flownet2-docker the Sintel dataset is copied in the ./data folder as ~/develop/flownet2-docker/data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png

Error occurs as "BaseException: img0 does not exist: /input-output/data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png" when I run:

$ ./run-network.sh -n FlowNet2-s -g 1 -vv ./run-network.sh -n FlowNet2-s -g 1 -vv data/Sintel1_list_test.txt data/Sintel2_list_test.txt data/Sintel_flo_list_test.txt or $./run-network.sh -n FlowNet2-s -g 1 -vv data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png data/SINTEL/test/clean/PERTURBED_market_3/frame_0002.png flow.flo

However, it succeed when running:

$ ./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo or $ ./run-network.sh -n FlowNet2-s -g 1 -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt as the readme.md instructed

The file do exist because it gives the image information when running:

$ file data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png

nikolausmayer commented 7 years ago

Hm, nothing bad so far. Can you please post the first lines of all three .txt files?

ZikeYan commented 7 years ago

1.data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png 2.data/SINTEL/test/clean/PERTURBED_market_3/frame_0002.png 3.Result/SINTEL/PERTURBED_market_3/frame_0001.flo

nikolausmayer commented 7 years ago

Strange. Can you start an interactive container and manually look what's in /input-output?

nvidia-docker run --rm --volume "${PWD}:/input-output:rw" -it "flownet2" /bin/bash

ZikeYan commented 7 years ago

it says "root@8997ac58b23c:/flownet2#". seemingly it's empty

nikolausmayer commented 7 years ago

Hmm, in flownet2 there should be at least these entries FN2_Makefile.config FN2_run-flownet-docker.py flownet2/ and /input-output should contain whatever was in the folder from which you executed that command...

Can you please post some info about your system? What's your OS and Docker version?

ZikeYan commented 7 years ago

Strange as it is, the three entries

FN2_Makefile.config, FN2_run-flownet-docker.py & flownet2 appears.

However, the error still occurs

nikolausmayer commented 7 years ago

That's really weird. If /input-output is empty, then the list files that ship with this repo should not work either, and neither should running the networks on single images. Did you do all of your tests in the same folder? (I remember I had problems in an NFS-mounted location)

ZikeYan commented 7 years ago

Finally I get it. Yes, it's the symbol link problem. I forgot about it. I copy the dataset to the folder and now it works.

Really appreciate your kindness and sorry for wasting your time :p

nikolausmayer commented 7 years ago

Great, no worries :+1: I should probably put that into the README.

uthynauta commented 6 years ago

I'm sorry I'm having the same issue, what is that "Symbol link problem" that ZikeYan is talking about?

nikolausmayer commented 6 years ago

@uthynauta If you are accessing the data in /input-output via some symbolic link, this can cause problems. I think this can happen when the symlink points into another filesystem.

Example: In this repo there is a data folder with images:

$ ls -l data
drwxr-xr-x  2 user group 4,0K Jun  1  2017 ./
drwxr-xr-x 13 user group 4,0K Jun  6 15:51 ../
-rw-r--r--  1 user group 686K Apr 26  2017 0000000-imgL.png
-rw-r--r--  1 user group 680K Apr 26  2017 0000001-imgL.png
...

but you might have links there, too:

$ ls -l data
drwxr-xr-x  2 user group 4,0K Jun  1  2017 ./
drwxr-xr-x 13 user group 4,0K Jun  6 15:51 ../
-rw-r--r--  1 user group 686K Apr 26  2017 0000000-imgL.png
-rw-r--r--  1 user group 680K Apr 26  2017 0000001-imgL.png
lrwxrwxrwx  1 user group   14 Jun  7 18:59 a_symbolic_link -> /media/user/0000002-imgL.png
...

These links can cause problems (I think it's because within the Docker container, you are not your own user and you don't have your usual access permissions).

uthynauta commented 6 years ago

I'm such a n00b I'm sorry, apparently running

sudo ./run-network.sh -n FlowNet2-s -g 0 -v data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt

does work, but when running my own list:

sudo ./run-network.sh -n FlowNet2-s -g 0 -v data/names1.txt data/names2.txt data/output.txt

I get this message again:

BaseException: img1 does not exist: /input-output/data/optflow/000039.png

I've already logged into the docker and navigated to /input-output/data and everything seems on its place. I'm really lost in this.

nikolausmayer commented 6 years ago

We are all noobs and we can only hope to continue learning from slightly lesser noobs :smirk:

Can you post the outputs of some commands?

Are you on a normal PC? Or in a cloud instance?

uthynauta commented 6 years ago

I've already ran an ls -l and apparently I don't have any symlinks. Could it be that the format of the TXT files is wrong?

nikolausmayer commented 6 years ago

That's also possible, but I don't think it's the case here. Your container throws this error: BaseException: img1 does not exist: /input-output/data/optflow/000039.png So it has already correctly parsed the txt file.

uthynauta commented 6 years ago

From outside the container for data

ls -l data
total 38988
-rwxrwxrwx 1 uthy uthy  702321 may 23 13:58 0000000-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000001-imgL.flo
-rwxrwxrwx 1 uthy uthy  696171 may 23 13:58 0000001-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000002-imgL.flo
-rwxrwxrwx 1 uthy uthy  728680 may 23 13:58 0000002-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000003-imgL.flo
-rwxrwxrwx 1 uthy uthy  705242 may 23 13:58 0000003-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000004-imgL.flo
-rwxrwxrwx 1 uthy uthy  668696 may 23 13:58 0000004-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000005-imgL.flo
-rwxrwxrwx 1 uthy uthy  649812 may 23 13:58 0000005-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000006-imgL.flo
-rwxrwxrwx 1 uthy uthy  650301 may 23 13:58 0000006-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000007-imgL.flo
-rwxrwxrwx 1 uthy uthy  666499 may 23 13:58 0000007-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000008-imgL.flo
-rwxrwxrwx 1 uthy uthy  686211 may 23 13:58 0000008-imgL.png
drwxrwxrwx 5 uthy uthy    4096 may 24 12:29 cabina
drwxrwxrwx 2 uthy uthy    4096 may 23 20:50 cflow
drwxrwxrwx 2 uthy uthy    4096 may 23 18:57 flow
-rwxrwxrwx 1 uthy uthy     176 jun  7 12:09 flow-first-images.txt
-rwxrwxrwx 1 uthy uthy     176 jun  7 12:10 flow-outputs.txt
-rwxrwxrwx 1 uthy uthy     176 jun  7 12:10 flow-second-images.txt
drwxrwxrwx 2 uthy uthy    4096 may 23 20:50 flow_vis
-rwxrwxrwx 1 uthy uthy    6984 jun  7 12:15 names1.txt
-rwxrwxrwx 1 uthy uthy    6987 jun  7 11:31 names2.txt
drwxrwxrwx 2 uthy uthy   12288 jun  7 12:14 optflow
-rwxrwxrwx 1 uthy uthy    8733 jun  7 11:28 output.txt
drwxrwxrwx 5 uthy uthy  180224 may 23 20:49 PNG
-rwxrwxrwx 1 uthy uthy  302049 may 23 13:58 teaser.png
drwxrwxrwx 2 uthy uthy    4096 may 23 19:01 test
-rwxrwxrwx 1 uthy uthy     136 jun  7 11:56 text.txt

From within the container:

root@2565204a334b:/flownet2# ls -l /input-output
total 4132
-rw-rw-r-- 1 1000 1000    1837 May 23 18:58 Dockerfile
-rw-rw-r-- 1 1000 1000    1746 May 23 18:58 FN2_Makefile.config
-rw-rw-r-- 1 1000 1000    4913 Jun  7 17:03 FN2_run-flownet-docker.py
-rw-rw-r-- 1 1000 1000   35141 May 23 18:58 LICENSE
-rw-rw-r-- 1 1000 1000      91 May 23 18:58 Makefile
-rw-rw-r-- 1 1000 1000    2997 May 23 18:58 README.md
drwxrwxrwx 9 1000 1000    4096 Jun  7 17:15 data
-rw-r--r-- 1 root root 4147212 Jun  7 16:54 flow.flo
-rw------- 1 root root      62 May 23 19:07 nohup.out
-rwxrwxr-x 1 1000 1000    5624 May 23 18:58 run-network.sh
-rw-rw-r-- 1 1000 1000     147 May 30 22:01 runme.sh

From outside the container:

ls -l data/optflow/000039.png
-rwxrwxrwx 1 uthy uthy 358295 jun  7 11:12 data/optflow/000039.png

stat data/optflow/000039.png
stat: cannot stat 'data/optflow/000039.png': No such file or directory

Inside the container:

root@6e528d8c6492:/flownet2# stat /input-output/data/optflow/000039.png
  File: '/input-output/data/optflow/000039.png'
  Size: 358295      Blocks: 704        IO Block: 4096   regular file
Device: 807h/2055d  Inode: 2931184     Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1000/ UNKNOWN)
Access: 2018-06-07 16:42:58.025642979 +0000
Modify: 2018-06-07 16:12:18.106159381 +0000
Change: 2018-06-07 17:52:23.862836387 +0000
 Birth: -
uthynauta commented 6 years ago

Ok, here's what I did:

In my TXT file I copied (from the command line window) the name it says is not finding, in my case data/optflow/000039.png and pasted it into the TXT, ran it again and now I'm getting this new error:

./run-network.sh -n FlowNet2-s -g 0 -v data/names1.txt data/names2.txt data/output.txt
Using GPU:       0
Running network: FlowNet2-s
Working dir:     /flownet2/flownet2/scripts
First input:     data/names1.txt
Second input:    data/names2.txt
Output:          data/output.txt
/flownet2/flownet2/scripts
Network forward pass using /flownet2/flownet2/models/FlowNet2-s/FlowNet2-s_weights.caffemodel.
Succeeded.
Traceback (most recent call last):
  File "run-flownet-docker.py", line 146, in <module>
    writeFlow(out, blob)
  File "run-flownet-docker.py", line 38, in writeFlow
    f = open(name, 'wb')
IOError: [Errno 2] No such file or directory: '/input-output/\xef\xbb\xbfdata/optflow/flow038.flo'
uthynauta commented 6 years ago

Ok, here's what happened, I do not know how I solved it but I solved it.

I opened the TXT files from the original GIT download, deleted the info contained there and added my list, then I saved it with another name. Now everything is working fine. I guess it had something to do with the format of the TXT file, I leave it here in case anyone else faces the same problem.

I really appreciate the time and effort you dedicated to these @nikolausmayer

nikolausmayer commented 6 years ago

Ok.. this IOError: [Errno 2] No such file or directory: '/input-output/\xef\xbb\xbfdata/optflow/flow038.flo' definitely looks like a formatting problem (\xef\xbb\xbf are bad characters in the path).

It's so strange that you can stat the file from within the container, but not outside... anyway, I'm glad that it works now :wink: