error importing

[lucadelu]

Running the following command I got several errors...

v.in.sos.py url=http://istsos.geodati.fmach.it/istsos/edenext? offering=GRIDA observed_properties='urn:ogc:def:parameter:x-istsos:1.0:meteo:air:humidity:relative' response_format='text/xml;subtype="om/1.0.0"' output=grida

name=
mapset=
fullname=
file=
DBMI-SQLite driver error:
Error in sqlite3_prepare():
no such table: gridaoneday

DBMI-SQLite driver error:
Error in sqlite3_prepare():
no such table: gridaoneday

WARNING: Width for column x012_03_14T18_35_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
WARNING: Width for column x012_03_14T19_35_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
.....................
WARNING: Width for column x014_09_09T09_39_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
WARNING: Width for column x014_09_09T10_39_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
DBMI-SQLite driver error:
Error in sqlite3_prepare():
too many columns on grida

DBMI-SQLite driver error:
Error in sqlite3_prepare():
too many columns on grida

Traceback (most recent call last):
  File "v.in.sos.py", line 266, in <module>
    main()
  File "v.in.sos.py", line 161, in main
    create_maps(parsed_obs)
  File "v.in.sos.py", line 248, in create_maps
    quiet=True)
  File "/home/lucadelu/compilati/grass72/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py", line 410, in run_command
    return handle_errors(returncode, returncode, args, kwargs)
  File "/home/lucadelu/compilati/grass72/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py", line 329, in handle_errors
    returncode=returncode)
grass.exceptions.CalledModuleError: Module run None ['v.in.ogr', '--q', '-o', 'input=/grassdata/latlong/sos/.tmp/geowork/6788.0', 'output=grida'] ended with error
Process ended with non-zero return code -11. See errors in the (error) output.

No output is created

This is the input geojson file grida_geojson.txt

Probably the column name x014_09_09T10_39_01_01_00 could be like t20140909T1039010100

[lucadelu]

If I run with event_time I got only

name=
mapset=
fullname=
file=
DBMI-SQLite driver error:
Error in sqlite3_prepare():
no such table: gridaoneday

DBMI-SQLite driver error:
Error in sqlite3_prepare():
no such table: gridaoneday

WARNING: Width for column x012_03_14T18_35_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
WARNING: Width for column x012_03_14T19_35_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
..........
WARNING: Width for column x012_03_15T16_31_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!
WARNING: Width for column x012_03_15T17_31_01_01_00 set to 255 (was not
         specified by OGR), some strings may be truncated!

[pesekon2]

About warnings: Is there any way to suppress those warnings? They appears because values doesn't have specified length. But I don't know their length (they don't appear when I retype them to float, but they don't have to be floats) and I even don't know how to set their length.

About "too many columns". I think it's some problem of GRASS itself, because the same imports work for smaller time span.

About name= etc: It is raised from g.findfile which I am using to find out if there exists the output map.

But what am I really concerned about is that you have gridaoneday as a table name. Is there any reason for this table name? Because tables are named after observations...

PS: I have renamed the timestamp columns.

[lucadelu]

Why should the values be not floats?

About "too many columns" it is not related to GRASS but probably to SQLite https://sqlite.org/limits.html

gridaoneday is the name of output....

[pesekon2]

I thought that there can be any type in value. Most often it's float, but it can be also string, like "red", "yellow" or something like "bear", "hedgehog", "manatee". But anyways I can solve it for those with float types and the other ones will still raise warnings.

I will take a look about how to hoodwink SQLite limits (where to set SQLITE_MAX_COLUMN), but it seems to be setted during the compilation and it's a problem. And I'm afraid that it will be incredibly slow for many columns (as it is O(N2)).

[lucadelu]

Now the error is like this, but probably is the same problem

Traceback (most recent call last):
  File "v.in.sos.py", line 336, in <module>
    main()
  File "v.in.sos.py", line 179, in main
    create_maps(parsed_obs, off, layerscount, new)
  File "v.in.sos.py", line 281, in create_maps
    new.table.insert([insert], many=True)
  File "/home/lucadelu/compilati/grass72/dist.x86_64-pc-linux-gnu/etc/python/grass/pygrass/vector/table.py", line 1109, in insert
    return cur.executemany(self.columns.insert_str, values)
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 194, and there are 156869 supplied.

maybe you could try to set a maximum limit of columns (2000) and if it is more, you can suggest (with grass.warning) to set a event_time with a smaller range or to recompile SQLite linking to SQLite limits page.

[pesekon2]

Hi, I reopen this issue and propose a new approach.

Now we have a layer per offering and per observed property. Timestamps are columns, procedures rows.

The new idea is having a layer per offering and per procedure. Then observed properties are columns, timestamps rows (maybe with one more column for timestamps to allow user to use SQL queries). It will also end in one more column (called connection?) to connect more rows to one point.

@lucadelu @ninsbl What do you guys think about it?
The workflow: The insertion of the first obsprop will be the same with minor changes. For the next obsprops, I will do use table.execute() with SQL statement to update a row. Would you prefer a different approach?