arki-query --postproc hangs (v1.14)

[edigiacomo]

As you can see in the example above, arki-query --postproc hangs.

Arkimet version: v1.14.

arki-query --data works fine:

[arkimet@arkimet5 ~]$ time arki-query --debug --data "reftime:=today 00:00; product:tp" /arkivio/arkimet/dataset/cosmo_5M_ita > out.grib
Processing /arkivio/arkimet/dataset/cosmo_5M_ita...
Running query SELECT m.id, m.format, m.file, m.offset, m.size, m.notes, m.reftime, m.uniq, m.other FROM md AS m INDEXED BY md_idx_reftime WHERE reftime BETWEEN '2020-02-06 00:00:00' AND '2020-02-06 00:00:00' AND (reftime>='2020-02-06 00:00:00' AND reftime<='2020-02-06 00:00:59') AND uniq IN (SELECT id FROM mduniq WHERE product =21) ORDER BY m.reftime

real    0m0.129s
user    0m0.078s
sys 0m0.049s

[arkimet@arkimet5 ~]$ grib_count out.grib 
73

arki-query --postproc hangs and I have to kill it.

[arkimet@arkimet5 ~]$ time arki-query --debug --postproc="subarea 9.1 12.9 43.6 45.2" "reftime:=today 00:00; product:tp" /arkivio/arkimet/dataset/cosmo_5M_ita > out.grib
Processing /arkivio/arkimet/dataset/cosmo_5M_ita...
Running query SELECT m.id, m.format, m.file, m.offset, m.size, m.notes, m.reftime, m.uniq, m.other FROM md AS m INDEXED BY md_idx_reftime WHERE reftime BETWEEN '2020-02-06 00:00:00' AND '2020-02-06 00:00:00' AND (reftime>='2020-02-06 00:00:00' AND reftime<='2020-02-06 00:00:59') AND uniq IN (SELECT id FROM mduniq WHERE product =21) ORDER BY m.reftime
^C
real    2m35.890s
user    0m0.080s
sys 0m0.030s

Surprisingly, arki-query --inline piped to postprocessor works fine.

[arkimet@arkimet5 ~]$ time arki-query --debug --inline "reftime:=today 00:00; product:tp" /arkivio/arkimet/dataset/cosmo_5M_ita | /usr/lib64/arkimet/subarea 9.1 12.9 43.6 45.2 > sub.grib
Processing /arkivio/arkimet/dataset/cosmo_5M_ita...
Running query SELECT m.id, m.format, m.file, m.offset, m.size, m.notes, m.reftime, m.uniq, m.other FROM md AS m INDEXED BY md_idx_reftime WHERE reftime BETWEEN '2020-02-06 00:00:00' AND '2020-02-06 00:00:00' AND (reftime>='2020-02-06 00:00:00' AND reftime<='2020-02-06 00:00:59') AND uniq IN (SELECT id FROM mduniq WHERE product =21) ORDER BY m.reftime

[arkimet@arkimet5 ~]$ grib_count sub.grib 
73

Related issue: #32

[edigiacomo]

I've tried with singlepoint and it works, the it could be a problem of that specific postprocess (which was installed long time ago).

[arkimet@arkimet5 ~]$ time arki-query -p "singlepoint -f JSON 12.0 43.0" "reftime:=today 00:00; product:tp" /arkivio/arkimet/dataset/cosmo_5M_ita | wc -c
27059

real    0m1.226s
user    0m0.732s
sys 0m0.241s

[edigiacomo]

Well, this is awkward: the subarea postprocessor uses arki-xargs --max-size=1MB. If I increase the size to 100M, the postprocessor works fine again.

[edigiacomo]

Same problem with the latest arkimet version (1.18)