Computer cannot contact the appropriate internet server to get the latest radar.dat information

[johncoxon]

Upon import, I get the following error message:

ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo
WARNING:root:Reading hdw.dat info from local files in None
ERROR:root:radarRead: cannot read None
Traceback (most recent call last):
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 67, in radarRead
    file_net = open(pathOpen, 'r')
TypeError: coercing to Unicode: need string or buffer, NoneType found
You may be getting this error because your computer cannot contact an appropriate internet server to get the latest radar.dat information.  You can use a local file instead by setting the SD_RADAR environment variable to the location of a local copy of radar.dat.

Example, you might add a similar line to your .bashrc:
export SD_RADAR=/home/username/tables/radar.dat

Also, make sure your SD_HDWPATH also points to the location of your hdw.dat files.
You can get the latest hdw.dat files from  https://github.com/vtsuperdarn/hdw.dat
Example, you might add a similar line to your .bashrc:
export SD_HDWPATH=/home/username/tables/hdw.dat/

ERROR:root:Could not update .radars.sqlite file with                               hdw.dat info

As far as I can tell, this error is coming from a failure to connect to a MongoDB database on line 315 of pydarn/radar/radInfoIo.py, so I tried running lines 315 – 332 to see whether I could replicate the error in the console and see what happened.

In [1]: from pymongo import MongoClient
In [2]: uri = "mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo"
In [3]: conn = MongoClient(uri)
In [4]: dba = conn['radarInfo']
In [5]: colSel = lambda colName: dba[colName].find()
In [6]: db_select = {'rad': colSel("radars"),
   ...:              'hdw': colSel("hdw"),
   ...:              'inf': colSel("metadata")}
In [7]: db_select
Out[7]: 
{'hdw': <pymongo.cursor.Cursor at 0x1108b7610>,
 'inf': <pymongo.cursor.Cursor at 0x1108b7310>,
 'rad': <pymongo.cursor.Cursor at 0x1108b7850>}
In [10]: conn.server_info()
---------------------------------------------------------------------------
ConfigurationError                        Traceback (most recent call last)
<ipython-input-10-d474f869388f> in <module>()
----> 1 conn.server_info()

/anaconda2/lib/python2.7/site-packages/pymongo/mongo_client.pyc in server_info(self, session)
   1452         return self.admin.command("buildinfo",
   1453                                   read_preference=ReadPreference.PRIMARY,
-> 1454                                   session=session)
   1455 
   1456     def list_databases(self, session=None, **kwargs):

/anaconda2/lib/python2.7/site-packages/pymongo/database.pyc in command(self, command, value, check, allowable_errors, read_preference, codec_options, session, **kwargs)
    527         """
    528         client = self.__client
--> 529         with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
    530             return self._command(sock_info, command, slave_ok, value,
    531                                  check, allowable_errors, read_preference,

/anaconda2/lib/python2.7/contextlib.pyc in __enter__(self)
     15     def __enter__(self):
     16         try:
---> 17             return self.gen.next()
     18         except StopIteration:
     19             raise RuntimeError("generator didn't yield")

/anaconda2/lib/python2.7/site-packages/pymongo/mongo_client.pyc in _socket_for_reads(self, read_preference)
    980         topology = self._get_topology()
    981         single = topology.description.topology_type == TOPOLOGY_TYPE.Single
--> 982         server = topology.select_server(read_preference)
    983         with self._get_socket(server) as sock_info:
    984             slave_ok = (single and not sock_info.is_mongos) or (

/anaconda2/lib/python2.7/site-packages/pymongo/topology.pyc in select_server(self, selector, server_selection_timeout, address)
    222         return random.choice(self.select_servers(selector,
    223                                                  server_selection_timeout,
--> 224                                                  address))
    225 
    226     def select_server_by_address(self, address,

/anaconda2/lib/python2.7/site-packages/pymongo/topology.pyc in select_servers(self, selector, server_selection_timeout, address)
    181         with self._lock:
    182             server_descriptions = self._select_servers_loop(
--> 183                 selector, server_timeout, address)
    184 
    185             return [self.get_server_by_address(sd.address)

/anaconda2/lib/python2.7/site-packages/pymongo/topology.pyc in _select_servers_loop(self, selector, timeout, address)
    212                 selector, address)
    213 
--> 214         self._description.check_compatible()
    215         return server_descriptions
    216 

/anaconda2/lib/python2.7/site-packages/pymongo/topology_description.pyc in check_compatible(self)
    119         """
    120         if self._incompatible_err:
--> 121             raise ConfigurationError(self._incompatible_err)
    122 
    123     def has_server(self, address):

ConfigurationError: Server at sd-work9.ece.vt.edu:27017 reports wire version 0, but this version of PyMongo requires at least 2 (MongoDB 2.6).

This seems to indicate that the computer can run everything except for conn.server_info() on line 317, but when I tried commenting that line out and importing davitpy in a new IPython session, I got the following, which seems to indicate that this isn't the whole story – but perhaps it's helpful?

ERROR:root:problem importing radar: Server at sd-work9.ece.vt.edu:27017 reports wire version 0, but this version of PyMongo requires at least 2 (MongoDB 2.6).
Traceback (most recent call last):
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/__init__.py", line 29, in <module>
    import radar
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/radar/__init__.py", line 81, in <module>
    _ = updateRadars()
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 283, in __init__
    isUp = self.sqlUpdate()
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 409, in sqlUpdate
    arr_rad = self.__makeInsDict(self.db_select['rad'], self.dtype_rad)
  File "/anaconda2/lib/python2.7/site-packages/davitpy-0.8-py2.7-macosx-10.6-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 460, in __makeInsDict
    for ir, row in enumerate(sel):
  File "/anaconda2/lib/python2.7/site-packages/pymongo/cursor.py", line 1176, in next
    if len(self.__data) or self._refresh():
  File "/anaconda2/lib/python2.7/site-packages/pymongo/cursor.py", line 1087, in _refresh
    self.__send_message(q)
  File "/anaconda2/lib/python2.7/site-packages/pymongo/cursor.py", line 924, in __send_message
    **kwargs)
  File "/anaconda2/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1009, in _send_message_with_response
    server = topology.select_server(selector)
  File "/anaconda2/lib/python2.7/site-packages/pymongo/topology.py", line 224, in select_server
    address))
  File "/anaconda2/lib/python2.7/site-packages/pymongo/topology.py", line 183, in select_servers
    selector, server_timeout, address)
  File "/anaconda2/lib/python2.7/site-packages/pymongo/topology.py", line 214, in _select_servers_loop
    self._description.check_compatible()
  File "/anaconda2/lib/python2.7/site-packages/pymongo/topology_description.py", line 121, in check_compatible
    raise ConfigurationError(self._incompatible_err)
ConfigurationError: Server at sd-work9.ece.vt.edu:27017 reports wire version 0, but this version of PyMongo requires at least 2 (MongoDB 2.6).

[e-271]

For anyone trying to work around this issue, here's what I did to get it working (on Ubuntu 16):

1. Download radar.dat from here: https://github.com/SuperDARN/rst/blob/master/tables/superdarn/radar.dat and put in ~/tables/radar.dat

2. Clone hdw.dat files from here: https://github.com/vtsuperdarn/hdw.dat and put in ~/tables/hwd.dat

3. Add these two lines to ~/.bashrc:

   export SD_HDWPATH=/home/username/tables/hdw.dat/ export SD_RADAR=/home/username/tables/radar.dat

4. Restart terminal. It still gives me this error:

   ERROR:root:Could not connect to remote DB: mongodb://sd_dbread:5d@sd-work9.ece.vt.edu:27017/radarInfo WARNING:root:Reading hdw.dat info from local files in /home/esther/tables/hdw.dat/

but it works fine.

[johncoxon]

It would be nice if there was a way to suppress the error, but it's nice to have it take less room!

[asreimer]

Hey @johncoxon, just so you know, the solution to your problem was in the output you posted in this comment.

That being said, we should do better to document this known issue in the README and installation instructions. Long term, we are going to remove all the mongodb stuff.

[johncoxon]

It’s true I could have gone and installed the .dat file, and I tried, but I didn’t manage to find it, I suspect because I’m not at all part of the SuperDARN community. I was also trying to help you all isolate whatever the MongoDB bug is, but if it’s being removed I’ll stop trying (I’ve been poking it in my downtime).

On a broader tack, i only use Tsyganenko tracing and mapObj, so it could be good to suppress the text above unless the user tries to actually use something that relies on hdw.dat?

[asreimer]

Even better is to pull Tsyganenko out of davitpy and into it's own repository. It's not a SuperDARN specific thing so it doesn't need to be in a SuperDARN specific python package.

The reason why the error text gets printed, that is the reason you see the text about hdw.dat files, is that when you run import davitpy the entire davitpy package is imported all at once.

Does every one of the modules in davitpy get imported all at once? Yes:

As you said, this isn't useful for those who only want to use a subset of davitpy. A potential fix is to remove all or some the import statements in all of the __init__.py files.

As for the pymongo problem, this looks like an issue with the VT server running an old version of mongodb. The error you have reported is clear:

ConfigurationError: Server at sd-work9.ece.vt.edu:27017 reports wire version 0, but this version of PyMongo requires at least 2 (MongoDB 2.6).

@ksterne, what version of mongodb is running on sd-work9? Can this be updated easily?

Thanks for poking away at the code @johncoxon!

[johncoxon]

I suppose the question of having models in Davitpy/in separate repos comes down to a question of philosophy! I can see why it might be better to split them off, but it's nice having a module that has useful stuff like that all in one place. I think I'd want to keep using mapObj (which is currently wowing colleagues with most excellent animations of AMPERE currents) though.

When I import the bits I use, I do from davitpy.models import tsyganenko and from davitpy.utils import plotUtils, so I suppose that's why I didn't expect the radar data to need to be there. To be honest though, once the mongodb bug is resolved (which looks promising!) it won't really matter either way.

Thanks for your help on this 😄 I am enjoying using Davitpy and I hope I'm more help than hindrance…