export(?) possible regression

[emilburzo]

Hi!

There seems to be some regression in the latest osmium-tool (or libosmium?)

I have ran the exact same steps on the same data, hardware, etc, only the versions differ

Steps:

• download OSM planet file (from pbf)
• extract relevant data using tags-filter (to pbf)
• export to geojsonseq
  • I suspect the regression is here, because the tags-filter steps took the same amount of time in both cases, but it was double for the export

protozero HEAD libosmium HEAD osmium-tool HEAD

[ 0:00]     overwrite: no
[ 0:00]     fsync: no
[ 0:00]   other options:
[ 0:00]     add referenced objects: yes
[ 0:00]   looking for tags...
[ 0:00]     on nodes: yes
[ 0:00]     on ways: yes
[ 0:00]     on relations: yes
[ 0:00] Following references...
[ 0:00]   Reading input file to find relations in relations...
[ 8:32]   Reading input file to find nodes/ways in relations...
[16:43]   Reading input file to find nodes in ways...
[41:47] Done following references.
[41:47] Opening input file...
[41:47] Opening output file...
[41:47] Copying matching objects to output file...
[73:11] Closing output file...
[73:11] Closing input file...
[73:11] Peak memory used: 2746 MBytes
[73:11] Done.

------

[ 0:00]     changeset: (omitted)
[ 0:00]     timestamp: (omitted)
[ 0:00]     uid:       (omitted)
[ 0:00]     user:      (omitted)
[ 0:00]     way_nodes: (omitted)
[ 0:00]   linear tags: none
[ 0:00]   area tags:   none
[ 0:00]   other options:
[ 0:00]     index type: sparse_file_array
[ 0:00]     add unique IDs: type and id
[ 0:00]     keep untagged features: no
[ 0:00] First pass (of two) through input file (reading relations)...
[ 1:11] First pass done.
[ 1:11] Second pass (of two) through input file...
[23:49] About 12411 MBytes used for node location index (in main memory or on disk).
[23:50] Second pass done.
[23:50] Wrote 15511268 features.
[23:50] Encountered 1668 errors.
[23:50] Peak memory used: 16260 MBytes
[23:50] Done.

protozero v1.6.2 libosmium v2.14.0 osmium-tool v1.8.0

[ 0:00]     overwrite: no
[ 0:00]     fsync: no
[ 0:00]   other options:
[ 0:00]     add referenced objects: yes
[ 0:00]   looking for tags...
[ 0:00]     on nodes: yes
[ 0:00]     on ways: yes
[ 0:00]     on relations: yes
[ 0:00] Following references...
[ 0:00]   Reading input file to find relations in relations...
[ 8:34]   Reading input file to find nodes/ways in relations...
[16:44]   Reading input file to find nodes in ways...
[41:32] Done following references.
[41:32] Opening input file...
[41:32] Opening output file...
[41:32] Copying matching objects to output file...
[73:26] Closing output file...
[73:26] Closing input file...
[73:26] Peak memory used: 2755 MBytes
[73:26] Done.

------

[ 0:00]     changeset: (omitted)
[ 0:00]     timestamp: (omitted)
[ 0:00]     uid:       (omitted)
[ 0:00]     user:      (omitted)
[ 0:00]     way_nodes: (omitted)
[ 0:00]   linear tags:
[ 0:00]   area tags:
[ 0:00]   other options:
[ 0:00]     index type: sparse_file_array
[ 0:00]     add unique IDs: type and id
[ 0:00]     keep untagged features: no
[ 0:00] First pass (of two) through input file (reading relations)...
[ 1:11] First pass done.
[ 1:11] Second pass (of two) through input file...
[42:56] About 12438 MBytes used for node location index (in main memory or on disk).
[42:56] Second pass done.
[42:56] Wrote 62414842 features.
[42:56] Encountered 3059 errors.
[42:56] Peak memory used: 19516 MBytes
[42:56] Done.

You can clearly see the very big difference in features written.

Let me know if I can help with any more information.

[joto]

Thanks for the report. Can you show me the config file you have been using for the export?

[emilburzo]

I hope you mean the command line arguments :) (I don't have any other config)

osmium export -vi sparse_file_array -u type_id -r filtered-tags.pbf -o filtered-tags.geojsonseq

[joto]

You were right. The export command did the wrong thing if no config file was used. I have fixed this now and released a new version 1.9.1.

[emilburzo]

I can confirm it works with 1.9.1, thanks for the quick fix!