Adding Data / Mapping Directories

[brian-t-allen]

I'm having a problem adding datasets or mapping data directories. This is a fresh install (on linux) using: git clone https://github.com/ajnisbet/opentopodata.git

After installing there is a "data" directory in the root directory, so I created a new directory inside of it and added some GeoTIFF's (N32W092.tiff, N32W093.tiff, etc).

After configuring config.yaml and doing "make build", when I do "make run" I get this: ERROR:root:Invalid config: Dataset folder 'data/test/' seems to be empty.

If config.yaml isn't pointed correctly I get: ERROR:root:Invalid config: No dataset folder found at location 'test/'

So I know it's finding the directory, but isn't seeing the files for some reason.

If I add the same folder to the "tests/data/datasets/" it works, but then "make build" takes MUCH longer, presumably because the data is getting built into the docker image (which won't scale nicely with the 3 TB of data I have).

As an alternate test I moved the data outside of the opentopodata altogether (at /test/test) and manually ran docker: docker run --rm -it --volume /test/test:/app/data:ro -p 5000:5000 opentopodata:cat VERSION`

Again I get the "seems to be empty" error, so it's seeing the folder but not the files in the folder (or something like that).

How can I map to existing directories to add large amounts of data without making the docker image massive?

What am I missing? Why is the "tests/data/datasets/" directory structure working when "data/" and external directories don't?

Thanks!

[ajnisbet]

Hmm, I couldn't replicate this, it doesn't seem like you're missing anything! It might be related to how docker volume mounts are defined: I have to look it up every time!

Here's what I did:

• Make a new folder in the repo folder opentopodata/data/test
• Add files like opentopodata/data/test/N32W093.tiff
• Add the dataset in the config with path: data/test/
• make build and make run

Is that what you did?

---

Also that last docker command is a little off:

Say your files are located like /test/test/N32W092.tiff. Then you want to mount them with --volume /test/test:/app/data/test:ro or --volume /test:/app/data:ro instead so that the files are located inside the docker image as /app/data/test/N32W092.tiff and in config.yaml the dataset path would be data/test/

---

Let me know how it goes: if it works I'd like to update the documentation or error messages to make it more clear. And if not, I'll try again to track down the bug!

[brian-t-allen]

Thanks for the quick reply.

It turns out the problem was being caused by selinux. Temporarily turning it off with setenforce 0 allowed it to work in the way that I wanted: having a .virt file at data/dataset_name/whatever.virt that points to a virtual path mapped using docker, which points to an external directory.

That way "make build" runs quickly and the docker image is small and lean, with the huge (3 TB) dataset being stored outside of the opentopodata directory and docker image (due to COPY . /app/ copying everything inside the opentopodata directory into the docker image).

Now I just need to find a permanent fix for selinux and create a massive .virt file with the entire dataset (4 resolutions layered using "multiple datasets").

Thanks!