Issue accessing dataset mounted from s3 bucket

[virajprakash]

Hi, I am getting the following error when I use 'make run' ERROR:root:Invalid config: No dataset folder found at location 'data/dataset' However, I already made sure that the S3 bucket is setup correctly using s3fs and verified on my local file system s3fs idemo /root/opentopodata/data -o passwd_file=${HOME}/.passwd-s3fs

Also attached are relevant portions of my config.yaml and makefile.

Makefile portion

run: docker run --rm -it --volume /root/opentopodata/data:/app/data:ro -p 5000:5000 opentopodata:$(VERSION)

config.yaml:

• name: ned10m path: data/dataset

Thanks in advance !!!

Yes I know it is not recommended to use root and I will be sure to change that but I am almost positive this is a separate issue.

[ajnisbet]

Hmmm, thanks for the thorough explanation! I haven't encountered this before but I can suggest some things to try:

One would be mounting one directory deeper, so you're passing a "normal" directory to docker rather than a mount point. So mounting s3fs idemo /root/opentopodata/data/s3mount/ then the config file would point to data/s3mount/dataset.

The next step would be execing into the running docker container and having a look around.

• Run the container with the docker run command above
• In a new terminal run docker ps to get the container ID for opentopodata
• Run docker exec -it XXXXX bash where XXXX is the container ID
• cd /app/
• ls -lh and make sure the data folder is there with normal permissions
• ls -lh /app/data/ and make sure the dataset folder is there with normal permissions
• ls -lgh /app/data/dataset/ and make sure the tifs are there.

[ajnisbet]

I've never used s3fs, but have used rclone mount to mount cloud buckets then have those as docker volumes for opentopodata and gpxz: you could look into that if you're having lots of issues.

[virajprakash]

Hey I am still stuck unfortunately! We have switched to rclone since you seem to be reporting success with using it to mount a cloud bucket specifically on opentopodata. Here are my exact steps:

start rclone to mount from our s3 bucket into /data/s3mount Leaving this process running, I start a new terminal and use the docker run command from above

Below are the permissions for all the folders and also all the rest of the commands you mentioned: ls -lh

ls -lh /app/data/

and lastly ls -lgh /app/data/dataset returns all of the tiff files

config.yaml:

name: ned10m path: data/s3mount/dataset

Thank you so much for your help

[virajprakash]

I have also experimented with multiple config paths including the one you suggested and I also did make build before testing any of them using make run. However I am still seeing 'no dataset folder' when i try to run

[virajprakash]

We also tried using s3fs and no luck with that unfortunately

[ajnisbet]

It looks like the files are at data/dataset rather than data/s3mount/dataset, could you try updating config.yaml to match?

You could also try an absolute path: /app/data/dataset

[ajnisbet]

It looks like the files are at data/dataset rather than data/s3mount/dataset, could you try updating config.yaml to match?

You could also try an absolute path: /app/data/dataset

Closing assuming this was the issue.