North America import fails

alterEgo123 commented 5 months ago

Hello, I'm trying to deploy nominatim on AWS but I keep getting this error

Defaulted container "nominatim" out of: nominatim, wait-for-db (init), download-pbf (init)
2024-01-10 15:42:28: Using project directory: /nominatim
2024-01-10 15:42:29: Creating database
2024-01-10 15:42:30: Setting up country tables
2024-01-10 15:42:32: Importing OSM data file
2024-01-10 15:42:32  osm2pgsql version 1.9.2
2024-01-10 15:42:32  Database version: 15.2
2024-01-10 15:42:32  PostGIS version: 3.3
2024-01-10 15:42:32  Storing properties to table '"public"."osm2pgsql_properties"'.
2024-01-10 17:32:40  Reading input files done in 6608s (1h 50m 8s).                       
2024-01-10 17:32:40    Processed 1821108675 nodes in 719s (11m 59s) - 2533k/s
2024-01-10 17:32:40    Processed 162164552 ways in 4778s (1h 19m 38s) - 34k/s
2024-01-10 17:32:40    Processed 1692584 relations in 1111s (18m 31s) - 2k/s
2024-01-10 17:32:40  No marked ways (Skipping stage 2).
2024-01-10 17:32:40  Clustering table 'place' by geometry...
2024-01-10 17:33:21  Done postprocessing on table 'planet_osm_nodes' in 0s
2024-01-10 17:33:21  Building index on table 'planet_osm_ways'
2024-01-10 17:51:12  Building index on table 'planet_osm_rels'
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 225, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 94, in run
    database_import.import_osm_data(files,
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 113, in import_osm_data
    run_osm2pgsql(options)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 76, in run_osm2pgsql
    subprocess.run(cmd, cwd=options.get('cwd', '.'),
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '0', '--style', '/usr/local/etc/nominatim/import-full.lua', '--output', 'flex', '--create', '--flat-nodes', '/nominatim/flatnode/flatnode.file', 'data.osm.pbf']' died with <Signals.SIGSEGV: 11>.

I'm reserving a 60GB RAM and 5 vCPU for the pod, and my database is in RDS with 2GB of RAM and 500GB of disk space. How can I fix this?

robjuz commented 5 months ago

Hi @alterEgo123

If you give me your values.yaml I can try it out. The error is not telling me anything.

alterEgo123 commented 5 months ago

Hey @robjuz, thank you for the fast reply. This is my file

  replicaCount: 1

  updateStrategy:
    type: Recreate

  image:
    repository: mediagis/nominatim
    pullPolicy: IfNotPresent
    tag: ""
  nominatimInitialize:
    enabled: true
    pbfUrl: https://download.geofabrik.de/north-america-latest.osm.pbf
    importWikipedia: false
    importGB_Postcode: false
    importUS_Postcode: false
    importStyle: full
    # customStyleUrl: https://raw.githubusercontent.com/david-mart/Nominatim/master/settings/import-street.style
    threads: 16
    freeze: false
    wikipediaUrl: https://nominatim.org/data/wikimedia-importance.sql.gz
    resources:  
      requests:
        cpu: 5000m
        memory: 60Gi
      limits:
        cpu: 5000m
        memory: 60Gi

  nominatimReplications:
    enabled: false

  nominatim:
    extraEnv: []

  nominatimUi:
    enabled: false
    version: 3.2.1

  serviceAccount:
    create: true
    annotations: {}
    name: ""

  podAnnotations: {}

  podSecurityContext:
    fsGroup: 101

  securityContext: {}

  service:
    type: ClusterIP
    port: 80
    clusterIP: ""
    loadBalancerIP: ""
    externalTrafficPolicy: Cluster
    loadBalancerSourceRanges: []
    extraPorts: []
    annotations: {}
    sessionAffinity: None
    sessionAffinityConfig: {}

  resources: 
     limits:
       cpu: 4000m
       memory: 29Gi
     requests:
       cpu: 4000m
       memory: 29Gi

  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 100
    targetCPUUtilizationPercentage: 80

  postgresql:
    enabled: false

  externalDatabase:
    host: xx
    port: 5432
    user: xx
    password: xx

  datapvc:
    enabled: true
    storageClass: gp2
    accessModes:
      - ReadWriteOnce
    size: 20Gi
    existingClaim: 

  flatnode:
    enabled: true
    storageClass: gp2
    accessModes:
      - ReadWriteOnce
    size: 100Gi
    existingClaim:

And this is my storage class

Name:            gp2
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            fsType=ext4,type=gp2
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

robjuz / helm-charts

North America import fails #74