Import full planet OSM : no space left on device

[ness31]

Hello, I'm doing OSM full planet import on a 8 CPU/16Go RAM machine, and database tablespace is on a 800Go HDD.

1. the import process is very slow : after 125h running I get only 100% relations and 3.9% ways written (0% nodes and 0% coords)
2. I finally get an error message "pg could not extend file ... No space left on device" whereas my disk with tablespace has is still 800Go of free space

[2020-01-04T20:36:02+01:00] 125:55:02 [progress] 125h1m0s C: 0/s ( 0.0%) N: 0/s ( 0.0%) W: 200/s ( 3.9%) R: 10/s (100.0%) [2020-01-04T20:36:25+01:00] 125:55:26 [fatal] bulk insert into "osm_roads": SQL Error: pq: n'a pas pu étendre le fichier « base/18535/5937630.6 » : Aucun espace disponible sur le périphérique in query COPY "import"."osm_roads" ("osm_id", "geometry", "type", "name", "tunnel", "bridge", "oneway", "ref", "z_order", "access", "service", "class") FROM STDIN

Did I miss something to speed up import process ? Where would I need more disk space ? on my system ? in pgsql data folder ? obviously tablespace location is not enough ...

Thanks

[ghost]

Hi

I had this situation, which was due to the temporary space being filled up.

Issue a df -h command and see if another disk is full.

I installed a full import of the planet, using the latest imposm3, and typically it took 25 hours. However I used SDD devices for the database, and an HDD device for the pbf file. In addition, I set the memory to 48 GB, this is on a virtual machine running on the xcp-ng hypervisor, with 8 cpu’s allocated. The vm is a server version of debian 10 (buster), with postgresql 12, the latest postgis and the database parameters set accordingly. Pre the SDD times, 60 hours was typical.

The table space for the planet used about 650GB (but of course if you add more mappings, then the space will increase)

GiS systems are very greedy!!

Bonne Chance

On 6 Jan 2020, at 11:27, ness31 notifications@github.com wrote:

Hello, I'm doing OSM full planet import on a 8 CPU/16Go RAM machine, and database tablespace is on a 800Go HDD.

the import process is very slow : after 125h running I get only 100% relations and 3.9% ways written (0% nodes and 0% coords) I finally get an error message "pg could not extend file ... No space left on device" whereas my disk with tablespace has is still 800Go of free space [2020-01-04T20:36:02+01:00] 125:55:02 [progress] 125h1m0s C: 0/s ( 0.0%) N: 0/s ( 0.0%) W: 200/s ( 3.9%) R: 10/s (100.0%) [2020-01-04T20:36:25+01:00] 125:55:26 [fatal] bulk insert into "osm_roads": SQL Error: pq: n'a pas pu étendre le fichier « base/18535/5937630.6 » : Aucun espace disponible sur le périphérique in query COPY "import"."osm_roads" ("osm_id", "geometry", "type", "name", "tunnel", "bridge", "oneway", "ref", "z_order", "access", "service", "class") FROM STDIN

Did I miss something to speed up import process ? Where would I need more disk space ? on my system ? in pgsql data folder ? obviously tablespace location is not enough ...

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JE4IA52SOPVWIBOLBTQ4MIS3A5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IEGNF3A, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7N7JHYIPHKTEJDYRJNVDTQ4MIS3ANCNFSM4KDDDRMQ.

[ness31]

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ? (note that my db tablespace is on /data)

[ghost]

Hi

Filesystem Size Used Avail Use% Mounted on /dev/xvda1 441G 155G 264G 37% / /dev/xvdf1 914G 274G 594G 32% /mnt/osm_db /dev/xvdc1 1007G 304G 652G 32% /mnt/tdisk

As you can see the database disk is large, this is because you can actually have three tablespaces in use, import where the initial load goes. public where the database you will use resides backup so you can revert to the previous one.

when you deploy, the import tablespace contents are moved to public, after public has been moved to backup, so you are then left with two populated partitions.

the planet file is

-rw-r--r-- 1 osm osm 51956685635 Dec 19 14:36 planet-latest.osm.pbf

around 52Gb

The tempory space is defined in the config.json file "cachedir": "/tmp/imposm_cache”

which if it is in /tmp, is usually on the / partition is used during the load, and is usually deleted after a reboot.

Is this your first ever attempt at building a GIS system?, if so I would suggest trying a smaller file, Andorra for example.

As I am using a VM, I load using a large amount of memory, then reset it to a lower amount.

You of course cannot do this on a dedicated server.

The amount of volatile memory used is dependant on the postgresql.conf file, if you choose values that are too large, it will use the swap file, on the HDD, not good. Typically I only use 2G of swap (on an SDD) on a load, most other times it is not used (because of the memory made available)

I use an HDD for loading because I don't load that often and its a SATA6 device.

Hope this helps

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ.

[ghost]

Hi

RE the size of tmp, not sure, never bothered to monitor it, but I assume its at least the size of the PBF?

This page is helpful

https://imposm.org/docs/imposm3/latest/tutorial.html#preparation https://imposm.org/docs/imposm3/latest/tutorial.html#preparation

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ.

[ghost]

Hi

Just started a planet import (52Go) and the temporary space used on the file load, ie prior to the database load was 92Gb (or 92Go)

I loaded europe (25Go) the other day, on a hypervised debian buster VM under xcp-ng, 12Gb allocated RAM, 2*ssd disks, load took 10 hours. This latest load is using same VM, but with 8G RAM, and I estimate 20 hours plus to complete. CPU is an Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

Did you succeed in loading?

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ.

[ness31]

hello, yes indeed temp space used for planet build is heavy, I already add a dedicated tablespace on big HDD for my database but I had to move postgresql data folder on the same HDD to allow for temp file load. This is working fine now. However import time is still very high, right now after 140h I only have 100% relations and 3% ways (0% nodes and 0% coords). I don't understand where it is coming from. From you above config I would assume HDD is the limiting hardware (versus SSD) because I use a Intel Core i7 CPU with 16G RAM. Anyway I'll be patient... Thanks.

On Sun, 19 Jan 2020 at 09:40, aimdev notifications@github.com wrote:

Hi

Just started a planet import (52Go) and the temporary space used on the file load, ie prior to the database load was 92Gb (or 92Go)

I loaded europe (25Go) the other day, on a hypervised debian buster VM under xcp-ng, 12Gb allocated RAM, 2*ssd disks, load took 10 hours. This latest load is using same VM, but with 8G RAM, and I estimate 20 hours plus to complete. CPU is an Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

Did you succeed in loading?

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AMOKLWQAN6Q24TQYN2SJD2LQ6QGYPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKMHSQ#issuecomment-575980490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMOKLWVETJGJ7PLXWW72MULQ6QGYPANCNFSM4KDDDRMQ .

[ghost]

Hi

Just thinking about this, I have had to increase my tmp partiton as my load crashed after 16 hours, insufficient space!! It would appear, and I did not realise this, imposm writes to the disk again, so I am rerunning it, with 200Gb available. Hopefully I will get a more definitive sizing if and when it completes, before I reboot to clear /tmp I gave up ages ago with HDD, this system has a 500G m2 SSD, and a iTB 2.5 inch SSD. Also I believe how the mapping is set up will have an impact on sizing and time to. load, I have about 20 tables mapped (admin, water,roads etc)

Also tuning the database for loads may assist, there are varying opinions on the internet on the settings, I am not convinced mine are the best.

On 20 Jan 2020, at 08:48, ness31 notifications@github.com wrote:

hello, yes indeed temp space used for planet build is heavy, I already add a dedicated tablespace on big HDD for my database but I had to move postgresql data folder on the same HDD to allow for temp file load. This is working fine now. However import time is still very high, right now after 140h I only have 100% relations and 3% ways (0% nodes and 0% coords). I don't understand where it is coming from. From you above config I would assume HDD is the limiting hardware (versus SSD) because I use a Intel Core i7 CPU with 16G RAM. Anyway I'll be patient... Thanks.

On Sun, 19 Jan 2020 at 09:40, aimdev notifications@github.com wrote:

Hi

Just started a planet import (52Go) and the temporary space used on the file load, ie prior to the database load was 92Gb (or 92Go)

I loaded europe (25Go) the other day, on a hypervised debian buster VM under xcp-ng, 12Gb allocated RAM, 2*ssd disks, load took 10 hours. This latest load is using same VM, but with 8G RAM, and I estimate 20 hours plus to complete. CPU is an Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

Did you succeed in loading?

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AMOKLWQAN6Q24TQYN2SJD2LQ6QGYPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKMHSQ#issuecomment-575980490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMOKLWVETJGJ7PLXWW72MULQ6QGYPANCNFSM4KDDDRMQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JA4B6GC6KESGPDQPSDQ6VQOPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJL2QLA#issuecomment-576170028, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7N7JCOS43PBNA3ISUUKH3Q6VQOPANCNFSM4KDDDRMQ.

[ghost]

Hi

Just finished the planet load, temp space around 101Go, database 322Go, time around 31 hours. Allocated memory 13Go, swap 4Go, used swap ~400Mo

On disk 1, the /tmp, on disk 2, planet file and database so there is never read/write on the same disk at the same time.

Here is some of the more interesting log output… [2020-01-20T08:03:37Z] 0:00:00 [step] Starting: Imposm [2020-01-20T08:03:37Z] 0:00:00 [step] Starting: Reading OSM data

[2020-01-20T08:46:07Z] 0:42:29 [step] Finished: Reading OSM data in 42m29.553158436s [2020-01-20T08:46:07Z] 0:42:29 [step] Starting: Importing OSM data [2020-01-20T08:46:07Z] 0:42:29 [step] Starting: Writing OSM data

[2020-01-21T09:09:48Z] 25:06:10 [step] Finished: Writing OSM data in 24h23m40.891573238s

[2020-01-21T09:09:48Z] 25:06:10 [step] Starting: Creating generalized tables

[2020-01-21T09:48:14Z] 25:44:37 [step] Finished: Creating generalized tables in 38m26.414674278s [2020-01-21T09:48:14Z] 25:44:37 [step] Starting: Clustering on geometry

[2020-01-21T14:54:12Z] 30:50:35 [step] Finished: Importing OSM data in 30h8m5.506278942s [2020-01-21T14:54:12Z] 30:50:35 [step] Finished: Imposm in 30h50m35.256818086s

On 20 Jan 2020, at 08:48, ness31 notifications@github.com wrote:

hello, yes indeed temp space used for planet build is heavy, I already add a dedicated tablespace on big HDD for my database but I had to move postgresql data folder on the same HDD to allow for temp file load. This is working fine now. However import time is still very high, right now after 140h I only have 100% relations and 3% ways (0% nodes and 0% coords). I don't understand where it is coming from. From you above config I would assume HDD is the limiting hardware (versus SSD) because I use a Intel Core i7 CPU with 16G RAM. Anyway I'll be patient... Thanks.

On Sun, 19 Jan 2020 at 09:40, aimdev notifications@github.com wrote:

Hi

Just started a planet import (52Go) and the temporary space used on the file load, ie prior to the database load was 92Gb (or 92Go)

I loaded europe (25Go) the other day, on a hypervised debian buster VM under xcp-ng, 12Gb allocated RAM, 2*ssd disks, load took 10 hours. This latest load is using same VM, but with 8G RAM, and I estimate 20 hours plus to complete. CPU is an Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

Did you succeed in loading?

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AMOKLWQAN6Q24TQYN2SJD2LQ6QGYPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKMHSQ#issuecomment-575980490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMOKLWVETJGJ7PLXWW72MULQ6QGYPANCNFSM4KDDDRMQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JA4B6GC6KESGPDQPSDQ6VQOPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJL2QLA#issuecomment-576170028, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC7N7JCOS43PBNA3ISUUKH3Q6VQOPANCNFSM4KDDDRMQ.

[ness31]

Thanks for the figures. Up to now writing OSM data is 218h ... still running (R:100% / W: 13.5% / N: 0% / C: 0%) ... temp space 45 Go / db tablespace 100 Go ! RAM 16 Go / swap 8 Go ; HDD writing is just a nightmare !

On Tue, 21 Jan 2020 at 16:10, aimdev notifications@github.com wrote:

Hi

Just finished the planet load, temp space around 101Go, database 322Go, time around 31 hours. Allocated memory 13Go, swap 4Go, used swap ~400Mo

On disk 1, the /tmp, on disk 2, planet file and database so there is never read/write on the same disk at the same time.

Here is some of the more interesting log output… [2020-01-20T08:03:37Z] 0:00:00 [step] Starting: Imposm [2020-01-20T08:03:37Z] 0:00:00 [step] Starting: Reading OSM data

[2020-01-20T08:46:07Z] 0:42:29 [step] Finished: Reading OSM data in 42m29.553158436s [2020-01-20T08:46:07Z] 0:42:29 [step] Starting: Importing OSM data [2020-01-20T08:46:07Z] 0:42:29 [step] Starting: Writing OSM data

[2020-01-21T09:09:48Z] 25:06:10 [step] Finished: Writing OSM data in 24h23m40.891573238s

[2020-01-21T09:09:48Z] 25:06:10 [step] Starting: Creating generalized tables

[2020-01-21T09:48:14Z] 25:44:37 [step] Finished: Creating generalized tables in 38m26.414674278s [2020-01-21T09:48:14Z] 25:44:37 [step] Starting: Clustering on geometry

[2020-01-21T14:54:12Z] 30:50:35 [step] Finished: Importing OSM data in 30h8m5.506278942s [2020-01-21T14:54:12Z] 30:50:35 [step] Finished: Imposm in 30h50m35.256818086s

On 20 Jan 2020, at 08:48, ness31 notifications@github.com wrote:

hello, yes indeed temp space used for planet build is heavy, I already add a dedicated tablespace on big HDD for my database but I had to move postgresql data folder on the same HDD to allow for temp file load. This is working fine now. However import time is still very high, right now after 140h I only have 100% relations and 3% ways (0% nodes and 0% coords). I don't understand where it is coming from. From you above config I would assume HDD is the limiting hardware (versus SSD) because I use a Intel Core i7 CPU with 16G RAM. Anyway I'll be patient... Thanks.

On Sun, 19 Jan 2020 at 09:40, aimdev notifications@github.com wrote:

Hi

Just started a planet import (52Go) and the temporary space used on the file load, ie prior to the database load was 92Gb (or 92Go)

I loaded europe (25Go) the other day, on a hypervised debian buster VM under xcp-ng, 12Gb allocated RAM, 2*ssd disks, load took 10 hours. This latest load is using same VM, but with 8G RAM, and I estimate 20 hours plus to complete. CPU is an Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz

Did you succeed in loading?

On 6 Jan 2020, at 13:51, ness31 notifications@github.com wrote:

df -h seems to be ok:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur devtmpfs 7,7G 0 7,7G 0% /dev tmpfs 7,7G 92M 7,6G 2% /dev/shm tmpfs 7,7G 26M 7,7G 1% /run tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 12G 39G 23% / /dev/sda1 1014M 235M 780M 24% /boot /dev/sdb1 917G 92G 779G 11% /data /dev/mapper/centos-home 180G 84G 96G 47% /home tmpfs 1,6G 40K 1,6G 1% /run/user/1000

temporary space seems to be stored within "/var/lib/pgsql/12/data/base/", i.e on / which has 39Go available ... how much do I need ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <

https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JAH3YNJRMGZFL3MQ23Q4MZPJA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFPY2I#issuecomment-571145321 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AC7N7JD4EYMXIVQUVM2T7WLQ4MZPJANCNFSM4KDDDRMQ

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AMOKLWQAN6Q24TQYN2SJD2LQ6QGYPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKMHSQ#issuecomment-575980490 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AMOKLWVETJGJ7PLXWW72MULQ6QGYPANCNFSM4KDDDRMQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AC7N7JA4B6GC6KESGPDQPSDQ6VQOPA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJL2QLA#issuecomment-576170028>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AC7N7JCOS43PBNA3ISUUKH3Q6VQOPANCNFSM4KDDDRMQ .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omniscale/imposm3/issues/224?email_source=notifications&email_token=AMOKLWR3O6G5KDN6MV4IPE3Q64F4NA5CNFSM4KDDDRM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQCDGY#issuecomment-576725403, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMOKLWVY5O7D4BT2JW36XT3Q64F4NANCNFSM4KDDDRMQ .

[francois2metz]

I believe this issue can be closed. It's an infrastructure problem.