Closed jata1 closed 5 months ago
That's strange. Can you try to start the container and exec in it:
ls -l /app/bin/*sh
ls -ld /app
ls -ld /app/bin
Thanks for helping me. I have made progress after editing the dockerfile and adding /app/bin/update_overpass_loop.sh so the same permissions are applied.
RUN chmod a+rx /app/docker-entrypoint.sh /app/bin/update_overpass.sh /app/bin/update_overpass_loop.sh /app/bin/rules_loop.sh /app/bin/dispatcher_start.sh \
/app/bin/oauth_cookie_client.py /app/bin/start_fcgiwarp.sh
Now running but queries take a few seconds to respond. when I cloned the entire planet it was faster so I need to look into this.
I think it must be an indexing issue (and I am running this on a raspberry pi) haha
The only data I actually want/need is speed limit data - this is a bit of an overkill but it is working...
http://localhost:8084/api/interpreter?data=[out:json];way[maxspeed](around:30.0,-33.75933,151.29756);out%20tags;
{
"version": 0.6,
"generator": "Overpass API 0.7.61.8 b1080abd",
"osm3s": {
"timestamp_osm_base": "2024-03-10T09:49:02Z",
"copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
},
"elements": [
{
"type": "way",
"id": 12542639,
"tags": {
"highway": "residential",
"maxspeed": "50",
"name": "Undercliffe Street",
"source:geometry": "LPI NSW Imagery",
"source:name": "NSW LPI Base Map",
"surface": "paved"
}
}
]
}
You can use OVERPASS_PLANET_PREPROCESS
to get only data you're interested using osmium (it should be already in the container).
Filtering DIFF's is still not implemented (see #67) - but you may also do the same approach on the diffs, and then your database should be significantly smaller (and hopefully faster).
Thanks. I am using the init method and only downloading / processing the Australia data - docker compose below. But it is much slower than the global clone. I am surprised about this so I think there is something wrong. It is a much smaller database though :-)
I had to use an older file that was in bz2 format https://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2 as I couldn't get the pre process command to work (file does not exist error).
The setup below does work but it took 3 days to process all the diffs as the base db was so old (i think)
I will try to do the same approach on the diffs.
version: '3.3'
services:
overpass:
image: overpass-api:latest
container_name: overpass-init
hostname: overpass_db
build: .
ports:
- 8084:80
volumes:
- /symlinks/omv-system/appdata/overpass-init:/db
environment:
- OVERPASS_META=yes
- OVERPASS_MODE=init
- OVERPASS_PLANET_URL=https://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2
## - OVERPASS_PLANET_URL=https://download.geofabrik.de/australia-oceania/australia-latest.osm.pbf
## - OVERPASS_COMPRESSION=gz
## - OVERPASS_PLANET_PREPROCESS='mv /planet.osm.bz2 /planet.osm.pbf && osmium cat -o /planet.osm.bz2 /planet.osm.pbf && rm /planet.osm.pbf'
- OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/
- OVERPASS_RULES_LOAD=10
- OVERPASS_UPDATE_SLEEP=3600
- OVERPASS_USE_AREAS=false
reading post 67 - it seems that that is what I have done. Taken the Oz base map and then applied all the diff files.
I'm way over my head with this stuff. Sorry to waste your time but I really appreciate the help.
Does it make sense that the australia only database would perform worse than the global database on the same hardware?
I'd try uncommenting - OVERPASS_COMPRESSION=gz
- this controls the compression of the Overpass database. IIRC Overpass documentation mentions, that with compression it is actually faster due to lower I/O.
Regarding the OVERPASS_PLANET_PREPROCESS
, if you download *bz2
file, start with:
OVERPASS_PLANET_PREPROCESS='mv /planet.osm.bz2 /planet2.osm.bz2 && osmium cat -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.pbf'
And just replace osmium cat
with osmium tags-filter
per your needs.
Oh wow. I am getting it. So if I use tags-filter it will just process tags rather than all the other stuff I don't need. I will try that for sure... Thank you
At the end of the PREPROCESS command we remove the file planet2.osm.pbf but where does this come from?
So there are following steps:
/planet-osm.bz2
(handled before OVERPASS_PLANET_PREPROCESS
kicks in)OVERPASS_PLANET_PREPROCESS
- move /planet-osm.bz2
to /planet-osm2.bz2
(osmium recognizes files by extension)osmium [...] -o /planet-osm.bz2 [...]
generates /planet-osm.bz2
file based on /planet-osm2.bz2
. Osmium can't handle conversion in-place as this would require a lot of RAM./planet-osm2.bz2
to save space. (created in step 2)planet-osm.bz2
generated by osmium
is used for further processingHope that helps.
love it. Thanks so much. I have learned a load over the last few days.
I'm not sure why I am getting this error - might be something to do with the docker image I made for arm64?
Running preprocessing command: 'mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.pbf'
/app/docker-entrypoint.sh: line 84: mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.pbf: No such file or directory
Failed to process planet file
You never created /planet2.osm.pbf
so that's why it is failing. You probably want to remove /planet2.osm.bz2
And you probably want to add something like w/maxspeed
to your osmium command. I getting this command right outside of Docker, ensure that you get expected results using osmium show
(or cat to uncompressed XML) and then incorporate that into container.
You never created
/planet2.osm.pbf
so that's why it is failing. You probably want to remove/planet2.osm.bz2
Yep - I spotted that error in the preprocess cmd earlier but I still get the error
Running preprocessing command: 'mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.bz2'
/app/docker-entrypoint.sh: line 84: mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.bz2: No such file or directory
Failed to process planet file
getting this command right outside of Docker
For this will I need another computer and install osmium directly? Or could I play around with osmium inside a docker container using docker exec?
You can play inside the docker, but just don't remove the source file, so you can easily retry.
@wiktorn - have made lots of progress but still one or two improvements to make.
I do think there is something wrong with OVERPASS_PLANET_PREPROCESS but could be the docker I built.
So please close this and I will open another thread to see if I can get OVERPASS_PLANET_PREPROCESS to work.
Thanks for all your help yesterday.
Hi @wiktorn - I am having fun with your docker but have an issue. I hope you can help. In summary:
My first setup, I build the image on my RPI (arm64) and managed to download the world in clone mode and got it working/running.
Now I am trying the init method and this is my progress/error. I hope you can help.
I think I am very close and have successfully downloaded the data - just need to start the container now I have init_done completed.
Anything I can try to fix this?