[nominatim][feature] Support import continue in init job

[dekzz]

Hello @robjuz, thanks for the chart!

Here is an attempt to support import continue option in init job, which might help in some cases when nominatim import fails but can be re-run without repeating all steps from scratch (i.e. osm data is imported but indexing fails for some reason). Motivation for this is my own case of larger dataset (europe) import on low-end infra which took like 5 days and failed on wikimedia import.

Option continue is not really documented (or at least I didn't mange to find it) except for this section. But I've managed to dig out something from source code.

[robjuz]

Hi.

Thx for the PR.

Could you explain me how it use it?

When I remember correctly, the import job gets restarted of error. When do you set the Continue variable?

[dekzz]

Ah, I didn't see the restart policy. The idea was to set continue config property and re-run helm upgrade in case of failure. State/step tracking of nominatim import execution seems a bit complicated to achieve, which is a prerequisite to implement automated continuation.

Also, maybe it makes sense to actually disable restart because only init containers benefit from it, since as soon as the database is created each restart would just get stuck at first command, which is db creation, resulting in pod restart loop (unless you manully drop the nominatim db). This can be detected by checking logs:

Using project directory: /nominatim
Creating database
error createdb: error: database creation failed: ERROR:  database "nominatim" already exists
FATAL Creating new database failed.
Using project directory: /nominatim
Creating database
error createdb: error: database creation failed: ERROR:  database "nominatim" already exists
FATAL Creating new database failed.
Using project directory: /nominatim
Creating database
error createdb: error: database creation failed: ERROR:  database "nominatim" already exists
FATAL Creating new database failed.

[dekzz]

@robjuz any thoughts on this?

[robjuz]

Sorry, was on holidays. I will take a look on this tomorrow

[dbt-lucka]

This is a useful feature for the helm chart. I am running my Germany import since 12 hours now and it turns out 30 Gig of memory is still not enough and crashed in one of the last steps.

I think your ampersands (??) in the markdown is broken though:

[dekzz]

Thanks for noticing and pointing that out, fixed it.

[robjuz]

@dekzz I think I have an idea.

When I was working on the chart in the first place, I noticed that the redeployment of the chart will not not start the import job.

In the meantime I was working on another chart with a database migration. adding a helm hook annotation to the job metadata will start it on every upgrade:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "nominatim.fullname" . }}-init
  namespace: {{ include "common.names.namespace" . | quote }}
  labels: {{- include "common.labels.standard" . | nindent 4 }}
  annotations:
    "helm.sh/hook": post-install,post-upgrade

Now we can set the restartPolicy to Never and let the user decide what to do

[dekzz]

Hey @robjuz, sorry for late reply.

I've added the hook and disabled restart and I think everything that we discussed is now covered.