v.in.sos modifications

[ninsbl]

Hi Ondrej,

I had several issues with v.in.sos. Below and in the PR some suggestions for changes... (did not check r.in.sos)

Requests threw an error because of observed_properties not being a list. Then no maps were produced because "property" seems to be a reserved key word in Python. At least I always got an AttributeError...

Now import is working, but for procedures that cover longer time periods, it is !extremely! slow (even for just two procedures with 5 measurements each) and eating a lot of ressources.

As a quick fix, I thought about introducing a flag for only retrieving procedure information (no observations) and started to work on it.

I would probably also revisit the discussion about how observations should be stored with v.in.sos. A "linked" table per observed property (and observations as rows) seems to be more appropriate to me than a column per observation.

[pesekon2]

Hi Stefan,

Requests threw an error because of observed_properties not being a list.

Mmm, that's strange, because observed_properties are gives as a list here. Can you send me a command causing the troubles please? I wasn't able to reproduce this error.

Then no maps were produced because "property" seems to be a reserved key word in Python. At least I always got an AttributeError...

property should be a built-in function in Python so it shouldn't raise an error. But probably it's still a good idea to rename these variables. However, I again didn't face any problem with this. Can you send me again an example command, please?

As a quick fix, I thought about introducing a flag for only retrieving procedure information (no observations) and started to work on it.

That's really great idea! It could be really useful/helpful feature. Thanks.

I would probably also revisit the discussion about how observations should be stored with v.in.sos. A "linked" table per observed property (and observations as rows) seems to be more appropriate to me than a column per observation.

Yes, I really don't like the current approach, but I think that there is some problem with linked tables in GRASS GIS. Matteo already suggested pivot tables and it didn't work in GRASS GIS. Unfortunately, I have currently much less time than expected and I'm trying to start works on my diploma thesis finally. I should be able to follow all contributions and code little, little bit, but I'm afraid that I'm not able to contribute in wide development until summer. Then, I would like to solve and improve some things (especially the problem with observations as columns, but I still don't have exact idea how) and move modules into the official repository.

[ninsbl]

Yes, sorry, you are right! I should have started with a proper bug report... (and some of my changes actually do not really help caus I changed the command too).

So here I try to replicate my issues with the istsos demo server (which was not always possible):

With username and password I get the following error message:

GRASS 7.4.0svn (ETRS_33N):/data/grassdata > v.in.sos --overwrite --verbose url="http://ninsrv16/istsos/tovvegetation?" output=sostest offering=temporary username=user1 password=justonavisit
WARNING: No data base element files found
Traceback (most recent call last):
  File "/usr/local/grass-7.4.0svn/scripts/v.in.sos", line 366, in <module>
    main()
  File "/usr/local/grass-7.4.0svn/scripts/v.in.sos", line 218, in main
    password=options['password'])
  File "/usr/local/lib/python2.7/dist-packages/OWSLib-0.15.0-py2.7.egg/owslib/swe/observation/sos100.py", line 203, in get_observation
    password=self.password, **url_kwargs).read()
  File "/usr/local/lib/python2.7/dist-packages/OWSLib-0.15.0-py2.7.egg/owslib/util.py", line 180, in openURL
    **rkwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='ninsrv16', port=80): Read timed out. (read timeout=30)

However, with the deom server (and without password) I get

GRASS 7.4.0svn (ETRS_33N):/data/grassdata > v.in.sos --overwrite --verbose url="http://istsos.org/istsos/demo?" output=sostest offering=BELLINZONA
WARNING: No data base element files found
Traceback (most recent call last):
  File "/usr/local/grass-7.4.0svn/scripts/v.in.sos", line 366, in <module>
    main()
  File "/usr/local/grass-7.4.0svn/scripts/v.in.sos", line 238, in main
    create_maps(parsed_obs, off, layerscount, new, secondsGranularity)
  File "/usr/local/grass-7.4.0svn/scripts/v.in.sos", line 259, in create_maps
    epochS = int(time.mktime(time.strptime(startTime, timestampPattern)))
  File "/usr/lib/python2.7/_strptime.py", line 467, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%S'

Which is due to the fact that create_maps() function uses the empty string from options nstead of the fixed time span from handle_not_given_options()

[pesekon2]

Everything seems fine to me. Thank you very much for this helpful patch.

As soon as possible, I will implement your features and patches also into the other modules.