OperationalError: no such table: rad

[arnaud85]

Hi everyone,

I've installed the Fedora version of DaViTPy with the fedora's install script but when I want to test a simple code to display radar's features I have some troubles.

I my Ipython notebook, I write this few lines :

%pylab inline from davitpy.pydarn.radar import *

radars = network() print radars

And there is an error : "OperationalError: no such table: rad"

What's the problem ? Is there a trouble with some environment variables ? I don't understand how to solve this ...

Thank you for the help,

Arnaud

[asreimer]

Hi @arnaud85,

Which branch are you using, master?

I'm running Fedora 23 and I've tried your code snippet on both the master and develop branches without any trouble.

Can you check your /tmp directory for a sd directory? If /tmp/sd/ exists, search for a .radars.sqlite file.

[arnaud85]

Hi @asreimer,

Thank you very much for your answer.

The trouble is that I have no file ".radars.sqlite" file in my /tmp/sd/ directory. This is why the system tell me this kind of things ...

But why ? Why do I have no radar database file in this directory ?

Thank you very much again.

Arnaud

[asreimer]

I'm not sure why you don't have a .radars.sqlite file. If you don't, davitpy should complain when you do a from davitpy.pydarn.radar import *.

You must be getting an error message somewhere that you haven't included here. Also, the .radars.sqlite file is a hidden file so you need to look for it using ls -a /tmp/sd/.

[arnaud85]

OMG I was sure that there was no ".radars.sqlite" file in my "/tmp/sd/" directory ... But it is ! I looked for it too fast and I was wrong ! ...

This exists but it is empty. Rights on it are "-rw-r--r--." . Do you think it is not good ?

Thank you so much for your answers and your super fast reactivity.

Arnaud

[asreimer]

Excellent! Since the file is there and the error you are having seems to be related to a problem with the tables in the file, I suggest you delete the .radars.sqlite file because I think it might be corrupted. That way, the next time you fire up davitpy, a new .radars.sqlite file will download and everything should be working after that.

[arnaud85]

I have the .radars.sqlite file. I tried to remove it and launch a script containing these few lines of code :

from davitpy.pydarn.radar import * radars = network() print(radars)

The file seems to be re-created each time I do the operation delete-launching code but it still empty every time ... Is it a problem of rights on it ? Rights for this file is : -rw-r--r--

Thank you for your answers, again.

Arnaud

[asreimer]

The .radars.sqlite file shouldn't be empty. The file I have is 24K and has the same permissions as you have.

We just updated DaViTpy today. Could you try reinstalling it? In the new version of davitpy, we've added a verbosity key to the davitpyrc file in davitpy/davitpy. After you have reinstalled davitpy from the current master or develop branch, could you run the following code and give me the output from it?

import davitpy
from davitpy.pydarn.radar import *
davitpy.rcParams['verbosity'] = 'debug-annoying'
radars = network()
print(radars)

Also, are you able to ping sd-work9.ece.vt.edu?

It would also be nice to see what you get if you try to manually update the .radars.sqlite file with:

from davitpy.pydarn import radar
t = radar.updateRadars()

[arnaud85]

I tried to re-install DaViTPy with the new version got from this GitHub (Master Branch) on my Fedora os and I have still the same problem : the .radars.sqlite file is empty and the output of the code you suggest me is :

Traceback (most recent call last): File "test.py", line 4, in radars = network() File "/usr/lib64/python2.7/site-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radStruct.py", line 81, in init cur.execute("select count() from rad") sqlite3.OperationalError: no such table: rad

Also, the output of the second code you suggest me is :

Traceback (most recent call last): File "test.py", line 2, in t = radar.updateRadars() File "/usr/lib64/python2.7/site-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 282, in init isUp = self.sqlUpdate() File "/usr/lib64/python2.7/site-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 402, in sqlUpdate arr_rad = self.makeInsDict(self.db_select['rad'], self.dtype_rad) File "/usr/lib64/python2.7/site-packages/davitpy-0.5-py2.7-linux-x86_64.egg/davitpy/pydarn/radar/radInfoIo.py", line 453, in makeInsDict for ir, row in enumerate(sel): File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next if len(self.data) or self._refresh(): File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh self.__read_concern)) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in send_message **kwargs) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response server = topology.select_server(selector) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 142, in select_server address)) File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 118, in select_servers self._error_message(selector)) pymongo.errors.ServerSelectionTimeoutError: sd-work9.ece.vt.edu:27017: [Errno 111] Connection refused

Although, when I try to ping the address you give me, it seems to be good ... :

PING sd-work9.ece.vt.edu (128.173.144.41) 56(84) bytes of data. 64 bytes from sd-work9.ece.vt.edu (128.173.144.41): icmp_seq=1 ttl=47 time=112 ms 64 bytes from sd-work9.ece.vt.edu (128.173.144.41): icmp_seq=2 ttl=47 time=112 ms 64 bytes from sd-work9.ece.vt.edu (128.173.144.41): icmp_seq=3 ttl=47 time=112 ms ...

If I want to install DaViTPy on a virtual environment, what are the steps to do ? Cause, I tried it on a virtual environment based on Python 2.7.x and I failed also ... Is there other things to do apart go into this virtual environment and launch dependencies/install scripts ?

Thank you.

[asreimer]

In the error message you posted: "pymongo.errors.ServerSelectionTimeoutError: sd-work9.ece.vt.edu:27017: [Errno 111] Connection refused"

There's the problem right there I think. You can ping the database server, but it won't let you connect to download the file, hence the file being empty.

Have you changed anything in davitpyrc?

@ksterne, can you look at why sd-work9 might not be accepting @arnaud85's connection? Maybe VT is blocking some IP addresses? It might show in sd-work9's connection log scripts.

@arnaud85, it is doubtful in my mind that running this in a virtual machine/environment will fix this issue for you. It seems your computer isn't being allowed to connect to sd-work9 (the Virginia Tech hardware.dat database server) so neither will a virtual machine/environment running on the computer.

[arnaud85]

Ok. But can I have to subscribe to a list or anything else to have access to this file or is it totally free ? Because I found your contribution surfing on the internet and looking at some tools that could be well for me; I work as a IT engineer/data scientist at the French Institute for Scientific Research and now I'm interesting on computing feet of magnetic field lines starting from satellites flying on the magnetosphere and reaching the Earth's ground in order to look if this point on the surface of the Earth is in a SuperDarn radar's field of view or not.

So, since I have told you all this stuff, I take the opportunity to ask you a question : do you think is this possible to calculate a field of view of a SuperDarn radar with your toolkit and after that can we know if a point of coords x,y,z is in the field of view of a radar or not ? Because it is my prime goal looking at your toolkit which seems to be great !

Thank you so much

Arnaud

[asreimer]

There is not list or anything that you need to subscribe to. For some reason Virginia Tech's hardware.dat server isn't accepting your connection (I'm not at Virginia Tech, so I don't have access to the computers there).

You didn't modify the davitpyrc file did you?

DaViTpy is able to calculate a FOV of a SuperDARN radar, either from hardware.dat information or from custom parameters (see the code in davitpy/davitpy/pydarn/radar. We also have the AACGM and Tsyganenko models available.

I have some code that can say if a lat/lon (but not x,y,z, you would have to map the footprint of a satellite to the ground and use the lat/lon at the ground) is inside a range-cell of a SuperDARN FOV, but that code is not yet in davitpy. I can add the code to davitpy, or email it to you if you wish (written in python, vectorized and very fast).

[arnaud85]

No, no, I didn' modify this file. I give you the content of it (sorry for the big sample):

This run control sets global parameters for DaViTPy. If you need to modify

it, please re-run the 'setup.py install' script in order to copy the

modified file to the appropriate system location.

#

VERBOSITY

Valid levels are silent, helpful, debug, and debug-annoying. These

correspond to the logging module logging levels of 60, 30, 20, and

10. See https://docs.python.org/2/howto/logging.html for more info.

verbosity : helpful

ADDRESS SFTP DATABASE, USERNAME, PASSWORD, AND PORT

DB : sd-data.ece.vt.edu DBREADUSER : sd_dbread DBREADPASS : 5d DB_PORT : 22

RADAR HARDWARE.DAT SERVER ADDRESS and PORT

SDDB : sd-work9.ece.vt.edu:27017 SDBREADUSER : sd_dbread SDBREADPASS : 5d DBWRITEUSER : DBWRITEPASS :

TEMPORARY DIRECTORY

DAVIT_TMPDIR : /tmp/sd/

RADAR DATA FILE FETCHING

The evironment variables are python dictionary capable formatted strings

appended encode radar name, channel, and/or date information. Currently

supported dictionary keys which can be used are:

#

"date" : datetime.datetime.strftime("%Y%m%d")

"year" : 0 padded 4 digit year

"month" : 0 padded 2 digit month

"day" : 0 padded 2 digit day

"hour" : 0 padded 2 digit day

"ftype" : filetype string

"radar" : 3-chr radarcode

"channel" : single character string, ex) 'a'

#

So for time = datetime(2012,2,1,0), rad='sas', and fileType='fitacf'

DAVIT_REMOTE_DIRFORMAT='data/{year}/{ftype}/{radar}/' would become:

'data/2012/fitacf/sas/'

#

ONLY the *FNAMEFMT environment variables will be converted to lists

if the environment variable contain a comma (','). For example:

#

DAVIT_REMOTE_FNAMEFMT='{date}.{hour}......{radar}.{ftype},{date}.{hour}......{radar}.{channel}.{ftype}'

becomes remote_fnamefmt = ['{date}.{hour}......{radar}.{ftype}','{date}.{hour}......{radar}.{channel}.{ftype}'] in python.

DAVIT_REMOTE_DIRFORMAT : data/{year}/{ftype}/{radar}/ DAVIT_REMOTE_FNAMEFMT : {date}.{hour}......{radar}.{ftype},{date}.{hour}......{radar}.{channel}.{ftype} DAVIT_LOCAL_DIRFORMAT : /sd-data/{year}/{ftype}/{radar}/ DAVIT_LOCAL_FNAMEFMT : {date}.{hour}......{radar}.{ftype},{date}.{hour}......{radar}.{channel}.{ftype}

MAP DATA FILE FETCHING

DAVIT_SD_REMOTE_DIRFORMAT : data/{year}/{ftype}/{hemi}/ DAVIT_SD_REMOTE_FNAMEFMT : {date}.{hemi}.{ftype} DAVIT_SD_LOCAL_DIRFORMAT : /sd-data/{year}/{ftype}/{hemi}/ DAVIT_SD_LOCAL_FNAMEFMT : {date}.{hemi}.{ftype}

SET THE PATH TO THE AACGM COEFFICIENT FILES

By default, this is set to where the coefficients are installed with

DaViTpy after the line below this block is uncommented, ie)

'davitpy install directory'/tables/aacgm/aacgm_coeffs/

AACGM_DAVITPY_DAT_PREFIX : /davitpy_dir_path/tables/aacgm/aacgm_coeffs/

Oh yes, very cool. I'm very interested by your code. Can you add it to the toolkit or share me it by email at : abiegun@irap.omp.eu. Thank you !

[ksterne]

Hey @arnaud85,

What IP address are you trying to access from? I can see if it's been blocked for some odd reason.

[arnaud85]

Do you see this IP when I try to connect at the server : 194.199.146.250 ? I think it is the proxy address of our lab ... Can it does some trouble to be hidden before a proxy ?

[ksterne]

I'm not seeing that IP address blacklisted on the sd-work9 computer. I have added it so that it should allow a connection though. Can you try it again?

[arnaud85]

I have tried to re-install quicly the soft and my .radars.sqlite file is always empty ... I will try more stuff later. Thanks for your help.

[asreimer]

Ah that's annoying! It sounds like a networking issue.

You could try running davitpy on a home computer and that would tell you if it is network specific. Might be that the network you are on is blocking port 27017?

Sorry we can't be of more help.

[ksterne]

That is quite strange! Maybe onto @asreimer's point, are you familiar with Netcat? I'm not as familiar with Fedora, but on Ubuntu I might run something like:

nc -vz sd-work9.ece.vt.edu 27017

and from a computer at Virginia Tech I get:

Connection to sd-work9.ece.vt.edu 27017 port [tcp/*] succeeded!

Are you able to get something similar? This should tell you whether or not you're able to connect to the database port on 27017. For more on Netcat (or nc):

http://linux.die.net/man/1/nc

[asreimer]

I always forget about netcat.

On Fedora the command nc -v sd-work9.ece.vt.edu 27017 works for me.

[arnaud85]

I've just tried this command on my Fedora : nc -v sd-work9.ece.vt.edu 27017

and the answer is :

Ncat: Version 6.45 ( http://nmap.org/ncat ) Ncat: Connection refused.

Do you think my proxy can make unavailable the socket between me and your server ?

I will try to install the DaViTPy toolkit in my personal computer (Mac OS X) soon and I will see if I have the same problem ...

[ksterne]

Ok @arnaud85, it seems as though we've confirmed that from your computer, you are unable to see port 27017 on sd-work9.ece.vt.edu. Since I haven't heard of this before, you might check with the internet provider or your proxy to see if it is blocking that port.

[arnaud85]

It works !!

Yes, a firewall blocked this port 27017 ... It is now open.

I tested the simple code and it works :

import davitpy from davitpy.pydarn.radar import * davitpy.rcParams['verbosity'] = 'debug-annoying' radars = network() print(radars)

So, is it possible to look at the code which help to know if a point of coords lat,lon is inside the field of view of a SuperDarn radar ? It can help me a lot !

Thank you for all.

Arnaud

[asreimer]

Hi @arnaud85,

I'll be emailing you the code that I have that finds the beam and gate number that a lat,lon pair corresponds to.

Cheers, Ashton

[arnaud85]

Thank you very much for the code, this will help me a lot !

Now I can play with your great toolkit !

Thank you for your help again.

Cheers,

Arnaud