Missing files in service containers

[steinbro]

Trying to build the Docker images, I'm running into problems.

In svcs/data/Dockerfile.ingest

COPY requirements.txt requirements_kubernetes.txt ingest.py kubescape.py tileperf.py queryperf.py extracts.json postgis-vt-util.sql tilefunc.sql $INGEST/

None of kubescape.py, tileperf.py, queryperf.py, or postgis-vt-util.sql are present in this repository. (At least I can find postgis-vt-util.sql elsewhere on Github.) I understand some bits were removed for the open source release, but were these particular files intentionally removed? In particular, kubescape.py is used extensively in ingest.py.

Separately, in svcs/data/misc/Dockerfile.ingest:

COPY Cargo.toml Cargo.lock /build/
COPY osmingester/Cargo.toml /build/osmingester
COPY osmingester/src/main.rs /build/osmingester/src

This doesn't match the file structure in the repository. There is no Cargo.lock file to copy (that should just be generated by cargo build during the image build process, no?) nor is there an osmingester folder.

[AdamGlass]

I think we were clear that the service was not entirely turnkey.

I as a Microsoft employee at tihe time (not currently) explicitly made the decisions about what files to include/not include in this area:

• tileperf.py - antique performance exploration code
• queryperf - similarly antique performance exploration code
• postgis-vt-util - available elsewhere and we didn't want to re-label it with our copyright
• kubescape.py, - this code reflected the entire history of the code base and was best lost. The rust-based ingester iwas superior in robustness and quality. The kubescape interactions in soundscape_provision are best replaced by inline creation of the postgres database.

the svcs/data/misc code is from a branch featuring a rust-based ingester that used osm2pgsql and was released as a supplement to the original source distribution on my request as imposm appears unmaintained. the docker file would've been in svcs/data and the rust code in svcs/data/osmingester

best of luck, Adam

[steinbro]

Adam, thanks for the context -- this is very helpful. And best of luck on your post-Microsoft endeavors.

It turns out there wasn't too much required functionality from kubescape.py. I managed to infer what was being used and write a minimal version that provides what ingest.py needed: https://github.com/steinbro/soundscape/blob/docker-compose/svcs/data/kubescape.py

I think it's a useful exercise to get the current implementation working as a frame of reference, even if there's little expectation it will continue to be used. I now have a docker-compose setup that populates a small Soundscape-compatible PostGIS database using the provided import.py and Dockerfile.ingst with minimal changes. In case it's useful to others, the configuration files are in the branch linked above.