Starting container after init_done complete error with supervisor supervisor: couldn't exec /app/bin/update_overpass_loop.sh: EACCES

[jata1]

Hi @wiktorn - I am having fun with your docker but have an issue. I hope you can help. In summary:

My first setup, I build the image on my RPI (arm64) and managed to download the world in clone mode and got it working/running.

Now I am trying the init method and this is my progress/error. I hope you can help.

1. config to download the australia base map (done/working)
2. process the minutes diffs (done/completed but I think something goes wrong here)
3. container stops as expected with the message - Overpass container initialization complete. Exiting.
4. I note there is a file in the db dir called init_done
5. start the container and... see log below.

I think I am very close and have successfully downloaded the data - just need to start the container now I have init_done completed.

Anything I can try to fix this?

Starting supervisord process
2024-03-10 01:21:13,808 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-10 01:21:13,811 INFO supervisord started with pid 1
2024-03-10 01:21:14,818 INFO spawned: 'overpass_dispatch' with pid 8
2024-03-10 01:21:14,823 INFO spawned: 'nginx' with pid 9
2024-03-10 01:21:14,827 INFO spawned: 'fcgiwrap' with pid 10
2024-03-10 01:21:14,832 INFO spawned: 'update_overpass' with pid 13
supervisor: couldn't exec /app/bin/update_overpass_loop.sh: EACCES
supervisor: child process was not spawned
2024-03-10 01:21:14,838 WARN exited: update_overpass (exit status 127; not expected)
2024-03-10 01:21:15,840 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-10 01:21:15,840 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-10 01:21:15,840 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-10 01:21:15,842 INFO spawned: 'update_overpass' with pid 26
supervisor: couldn't exec /app/bin/update_overpass_loop.sh: EACCES
supervisor: child process was not spawned
2024-03-10 01:21:15,846 WARN exited: update_overpass (exit status 127; not expected)
2024-03-10 01:21:17,850 INFO spawned: 'update_overpass' with pid 27
supervisor: couldn't exec /app/bin/update_overpass_loop.sh: EACCES
supervisor: child process was not spawned
2024-03-10 01:21:17,856 WARN exited: update_overpass (exit status 127; not expected)
2024-03-10 01:21:20,861 INFO spawned: 'update_overpass' with pid 28
supervisor: couldn't exec /app/bin/update_overpass_loop.sh: EACCES
supervisor: child process was not spawned
2024-03-10 01:21:20,865 WARN exited: update_overpass (exit status 127; not expected)
2024-03-10 01:21:21,866 INFO gave up: update_overpass entered FATAL state, too many start retries too quickly

[wiktorn]

That's strange. Can you try to start the container and exec in it:

ls -l /app/bin/*sh
ls -ld /app
ls -ld /app/bin

[jata1]

Thanks for helping me. I have made progress after editing the dockerfile and adding /app/bin/update_overpass_loop.sh so the same permissions are applied.

RUN chmod a+rx /app/docker-entrypoint.sh /app/bin/update_overpass.sh /app/bin/update_overpass_loop.sh /app/bin/rules_loop.sh /app/bin/dispatcher_start.sh \
    /app/bin/oauth_cookie_client.py /app/bin/start_fcgiwarp.sh

[jata1]

Now running but queries take a few seconds to respond. when I cloned the entire planet it was faster so I need to look into this.

I think it must be an indexing issue (and I am running this on a raspberry pi) haha

[jata1]

The only data I actually want/need is speed limit data - this is a bit of an overkill but it is working...

http://localhost:8084/api/interpreter?data=[out:json];way[maxspeed](around:30.0,-33.75933,151.29756);out%20tags;

{
  "version": 0.6,
  "generator": "Overpass API 0.7.61.8 b1080abd",
  "osm3s": {
    "timestamp_osm_base": "2024-03-10T09:49:02Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "way",
  "id": 12542639,
  "tags": {
    "highway": "residential",
    "maxspeed": "50",
    "name": "Undercliffe Street",
    "source:geometry": "LPI NSW Imagery",
    "source:name": "NSW LPI Base Map",
    "surface": "paved"
  }
}

  ]
}

[wiktorn]

You can use OVERPASS_PLANET_PREPROCESS to get only data you're interested using osmium (it should be already in the container).

Filtering DIFF's is still not implemented (see #67) - but you may also do the same approach on the diffs, and then your database should be significantly smaller (and hopefully faster).

[jata1]

Thanks. I am using the init method and only downloading / processing the Australia data - docker compose below. But it is much slower than the global clone. I am surprised about this so I think there is something wrong. It is a much smaller database though :-)

I had to use an older file that was in bz2 format https://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2 as I couldn't get the pre process command to work (file does not exist error).

The setup below does work but it took 3 days to process all the diffs as the base db was so old (i think)

I will try to do the same approach on the diffs.

version: '3.3'

services:
  overpass:
    image: overpass-api:latest
    container_name: overpass-init
    hostname: overpass_db
    build: .
    ports:
      - 8084:80
    volumes:
      - /symlinks/omv-system/appdata/overpass-init:/db
    environment:
      - OVERPASS_META=yes
      - OVERPASS_MODE=init
      - OVERPASS_PLANET_URL=https://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2
 ##     - OVERPASS_PLANET_URL=https://download.geofabrik.de/australia-oceania/australia-latest.osm.pbf
 ##     - OVERPASS_COMPRESSION=gz
 ##     - OVERPASS_PLANET_PREPROCESS='mv /planet.osm.bz2 /planet.osm.pbf && osmium cat -o /planet.osm.bz2 /planet.osm.pbf && rm /planet.osm.pbf'
      - OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/
      - OVERPASS_RULES_LOAD=10
      - OVERPASS_UPDATE_SLEEP=3600
      - OVERPASS_USE_AREAS=false

[jata1]

reading post 67 - it seems that that is what I have done. Taken the Oz base map and then applied all the diff files.

I'm way over my head with this stuff. Sorry to waste your time but I really appreciate the help.

Does it make sense that the australia only database would perform worse than the global database on the same hardware?

[wiktorn]

I'd try uncommenting - OVERPASS_COMPRESSION=gz - this controls the compression of the Overpass database. IIRC Overpass documentation mentions, that with compression it is actually faster due to lower I/O.

Regarding the OVERPASS_PLANET_PREPROCESS, if you download *bz2 file, start with:

OVERPASS_PLANET_PREPROCESS='mv /planet.osm.bz2 /planet2.osm.bz2 && osmium cat -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.pbf'

And just replace osmium cat with osmium tags-filter per your needs.

[jata1]

Oh wow. I am getting it. So if I use tags-filter it will just process tags rather than all the other stuff I don't need. I will try that for sure... Thank you

[jata1]

At the end of the PREPROCESS command we remove the file planet2.osm.pbf but where does this come from?

[wiktorn]

So there are following steps:

1. download remote file to /planet-osm.bz2 (handled before OVERPASS_PLANET_PREPROCESS kicks in)
2. Fist command in OVERPASS_PLANET_PREPROCESS - move /planet-osm.bz2 to /planet-osm2.bz2 (osmium recognizes files by extension)
3. Second command osmium [...] -o /planet-osm.bz2 [...] generates /planet-osm.bz2 file based on /planet-osm2.bz2. Osmium can't handle conversion in-place as this would require a lot of RAM.
4. Third command removes /planet-osm2.bz2 to save space. (created in step 2)
5. planet-osm.bz2 generated by osmium is used for further processing

Hope that helps.

[jata1]

love it. Thanks so much. I have learned a load over the last few days.

[jata1]

I'm not sure why I am getting this error - might be something to do with the docker image I made for arm64?

Running preprocessing command: 'mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.pbf'
/app/docker-entrypoint.sh: line 84: mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.pbf: No such file or directory
Failed to process planet file

[wiktorn]

You never created /planet2.osm.pbf so that's why it is failing. You probably want to remove /planet2.osm.bz2

[wiktorn]

And you probably want to add something like w/maxspeed to your osmium command. I getting this command right outside of Docker, ensure that you get expected results using osmium show (or cat to uncompressed XML) and then incorporate that into container.

[jata1]

You never created /planet2.osm.pbf so that's why it is failing. You probably want to remove /planet2.osm.bz2

Yep - I spotted that error in the preprocess cmd earlier but I still get the error

Running preprocessing command: 'mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.bz2'
/app/docker-entrypoint.sh: line 84: mv /planet.osm.bz2 /planet2.osm.bz2 && osmium tags-filter -o /planet.osm.bz2 /planet2.osm.bz2 && rm /planet2.osm.bz2: No such file or directory
Failed to process planet file

[jata1]

getting this command right outside of Docker

For this will I need another computer and install osmium directly? Or could I play around with osmium inside a docker container using docker exec?

[wiktorn]

You can play inside the docker, but just don't remove the source file, so you can easily retry.

[jata1]

@wiktorn - have made lots of progress but still one or two improvements to make.

I do think there is something wrong with OVERPASS_PLANET_PREPROCESS but could be the docker I built.

So please close this and I will open another thread to see if I can get OVERPASS_PLANET_PREPROCESS to work.

Thanks for all your help yesterday.