Problems in exporting from dballe

[bchiavarini]

Hi, we are experiencing some issues in exporting from dballe. I attach a zip file with the script we are using and the bufr file coming from dballe where we are experiencing the exporting issues.

For the _mobilestations.bufr we get this error from dballe.exporter:

File "/scripts/dballe2arkimet_mine.py", line 139, in dballe2arkimet
    tmpfile.write(exporter.to_binary(row.message))
RuntimeError: contradicting height indication found (both 100 and 100)

While for _30_marzo_todballe.bufr example we have some problems in exporting metadata to json using dballe.explorer and we get an error like this:

File "/scripts/dballe2arkimet_mine.py", line 149, in dballe2arkimet
    updater.add_json(fd.read())
RuntimeError: unsupported key "id" for Station

We are using Dballe 8.12

Thanks for your help!

[bchiavarini]

While for 30_marzo_to_dballe.bufr example we have some problems in exporting metadata to json using dballe.explorer and we get an error like this:

File "/scripts/dballe2arkimet_mine.py", line 149, in dballe2arkimet updater.add_json(fd.read()) RuntimeError: unsupported key "id" for Station

We have found what was the error for this point. We were using a dballe.Explorer instead of a dballeDBExplorer

[edigiacomo]

I tried to reproduce the first issue with the following code (dballe2arkimet_mine.py is quite complex and contains some hardcoded paths).

$ dbadb import --dsn=sqlite:test.db --wipe-first mobile_stations.bufr
$ python3 <<EOF
import dballe

db = dballe.DB.connect("sqlite:test.db")
tmpdatafile = "out.bufr"
rec = {}

with db.transaction() as tr:
    exporter = dballe.Exporter("BUFR")
    with open(tmpdatafile, 'wb') as tmpfile:
        for row in tr.query_messages(rec):
            tmpfile.write(exporter.to_binary(row.message))
EOF
$ echo $?
0
$ ls -la out.bufr mobile_stations.bufr 
-rw-r--r--. 1 edg edg 240882 Sep 11 12:16 mobile_stations.bufr
-rw-rw-r--. 1 edg edg 240882 Sep 18 08:08 out.bufr

I am not getting any error. Maybe I'm not reproducing correctly the issue?

[bchiavarini]

I'm having this problem on the db for the day mobile stations data are saved and the bufr i gave you in attach is an export of the db data of that day. I have tried to import the file in a temp db as you have done and you're right, the issue does not appear.. maybe happened something wrong when i originally imported the data in the actual dballe? how can i check that?

[edigiacomo]

I'm sorry, but I don't know. @spanezz do you have any ideas?

[spanezz]

I have no ideas. I'd say, if it happens again, try also with Emanuele's simplified script, and see if it gets more easily reproducible

[edigiacomo]

@bchiavarini please reopen this issue (or a new one) if you experience again the problem.